AI crawlers / CCBot
CCBot: what it is and how to block or charge it
CCBot is run by Common Crawl. Common Crawl's crawler. Its open web archive is one of the most widely used sources of AI training data.
| Operator | Common Crawl |
|---|---|
| Purpose | Collects content for AI training |
| User-agent contains | CCBot |
| Official IP list | index.commoncrawl.org/ccbot.json (5 ranges). Verify an address |
Block CCBot in robots.txt
User-agent: CCBot
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match CCBot in the User-Agent header.
Or charge CCBot per page
Instead of blocking it, answer CCBot with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is CCBot reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.