AI crawlers / ClaudeBot
ClaudeBot: what it is and how to block or charge it
ClaudeBot is run by Anthropic. Anthropic's crawler for collecting public web content that may be used to train Claude models.
| Operator | Anthropic |
|---|---|
| Purpose | Collects content for AI training |
| User-agent contains | ClaudeBot |
| Official IP list | None published, so it can only be recognized by name. Which crawlers publish one |
| Same operator | Claude-User, Claude-SearchBot, anthropic-ai |
Block ClaudeBot in robots.txt
User-agent: ClaudeBot
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match ClaudeBot in the User-Agent header.
Or charge ClaudeBot per page
Instead of blocking it, answer ClaudeBot with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is ClaudeBot reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.