AI crawlers: who they are and how to block or charge them
The 29 AI crawlers and AI user agents that visit websites most, who runs each one, what it collects, and the exact robots.txt lines to block it. Or skip the choice between "free" and "blocked": charge them per page with pay-per-crawl.
How many sites block each one? See the state of AI crawler blocking across 60 well-known sites.
| Crawler | Operator | What it does |
|---|---|---|
GPTBot | OpenAI | Collects content for AI training |
OAI-SearchBot | OpenAI | Indexes pages for AI search answers |
ChatGPT-User | OpenAI | Fetches pages live when a user asks |
ClaudeBot | Anthropic | Collects content for AI training |
Claude-User | Anthropic | Fetches pages live when a user asks |
Claude-SearchBot | Anthropic | Indexes pages for AI search answers |
anthropic-ai | Anthropic | Collects content for AI training |
PerplexityBot | Perplexity | Indexes pages for AI search answers |
Perplexity-User | Perplexity | Fetches pages live when a user asks |
Google-Extended | Collects content for AI training | |
GoogleOther | Several uses, including AI | |
CCBot | Common Crawl | Collects content for AI training |
Bytespider | ByteDance | Collects content for AI training |
Amazonbot | Amazon | Several uses, including AI |
Applebot-Extended | Apple | Collects content for AI training |
meta-externalagent | Meta | Collects content for AI training |
FacebookBot | Meta | Collects content for AI training |
cohere-ai | Cohere | Several uses, including AI |
Diffbot | Diffbot | Several uses, including AI |
YouBot | You.com | Indexes pages for AI search answers |
MistralAI-User | Mistral AI | Fetches pages live when a user asks |
DuckAssistBot | DuckDuckGo | Fetches pages live when a user asks |
ElevenlabsCrawler | ElevenLabs | Collects content for AI training |
ElevenlabsBot | ElevenLabs | Fetches pages live when a user asks |
PetalBot | Huawei | Indexes pages for AI search answers |
Timpibot | Timpi | Indexes pages for AI search answers |
ImagesiftBot | ImageSift | Several uses, including AI |
omgili | Webz.io | Several uses, including AI |
AI2Bot | Allen Institute for AI | Collects content for AI training |
Block all of them
Or pick and choose with the robots.txt generator.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: YouBot
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: ElevenlabsCrawler
User-agent: ElevenlabsBot
User-agent: PetalBot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: omgili
User-agent: AI2Bot
Disallow: /Blocking is all or nothing: a blocked crawler also can't cite or link to you. Pay-per-crawl answers AI crawlers with HTTP 402 and a price instead, so the ones that pay get the page and you get paid. Search engines (Googlebot, Bingbot) are not on this list.