AI crawlers / AI2Bot
AI2Bot: what it is and how to block or charge it
AI2Bot is run by Allen Institute for AI. The Allen Institute for AI's crawler for collecting web content for open research models.
| Operator | Allen Institute for AI |
|---|---|
| Purpose | Collects content for AI training |
| User-agent contains | AI2Bot |
| Official IP list | None published, so it can only be recognized by name. Which crawlers publish one |
Block AI2Bot in robots.txt
User-agent: AI2Bot
Disallow: /robots.txt is a request, not a lock: well-behaved crawlers follow it. To block by user agent at the server, match AI2Bot in the User-Agent header.
Or charge AI2Bot per page
Instead of blocking it, answer AI2Bot with HTTP 402 Payment Required and a price. Crawlers and agents that support x402 pay a fraction of a cent in USDC straight to your wallet and get the page; people and search engines browse free. Set it up free on pay-per-crawl in about five minutes.
Is AI2Bot reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.