AI crawlers / live on this site
AI crawlers on aipaygen.com, live
We run our own pay-per-crawl here. These are the AI crawler requests this site got in the last 30 days, each checked against the IP addresses its company publishes. Updated every minute, nothing edited out.
Counting since 2026-11-16. Last 30 days. Every number is a live database count.
Verified 972 requests by Web Bot Auth signature and 8,394 by the company's published IP ranges. Check what we see for your own request.
By day
Which crawlers
| Crawler | Company | Requests | Asked to pay | IP check |
|---|---|---|---|---|
meta-externalagent | Meta | 28,998 | 533 | Name only: the company publishes no IP list |
GoogleOther | 9,658 | 2,807 | Real: IP matches the published list | |
Amazonbot | Amazon | 4,239 | 2,683 | Name only: the company publishes no IP list |
GPTBot | OpenAI | 730 | 35 | Real: IP matches the published list |
ClaudeBot | Anthropic | 487 | 1 | IP checked since Nov 17 (the company's list is new) |
ElevenlabsCrawler | ElevenLabs | 468 | 80 | Name only: the company publishes no IP list |
ShapBot | 353 | 0 | Name only: the company publishes no IP list | |
Signed agent (achivx.com) | 141 | 0 | Name only: the company publishes no IP list | |
Signed agent (prober.tersign.ai) | 116 | 0 | Name only: the company publishes no IP list | |
Signed agent (brick.blue) | 102 | 0 | Name only: the company publishes no IP list | |
Signed agent (crawler.exa.ai) | 85 | 0 | Name only: the company publishes no IP list | |
Signed agent (agenstry.com) | 72 | 0 | Name only: the company publishes no IP list | |
ChatGPT-User | OpenAI | 65 | 0 | Real: IP matches the published list |
Signed agent (agentconcierge.biz) | 56 | 0 | Name only: the company publishes no IP list | |
Claude-User (fake) | Anthropic | 35 | 0 | Fake: name doesn't match the company's addresses |
OAI-SearchBot | OpenAI | 24 | 1 | Real: IP matches the published list |
Signed agent (agent.agentdrives.com) | 21 | 0 | Name only: the company publishes no IP list | |
GPTBot (fake) | OpenAI | 20 | 35 | Fake: name doesn't match the company's addresses |
PerplexityBot | Perplexity | 19 | 6 | Real: IP matches the published list |
Signed agent (exauser.crawler.exa.ai) | 10 | 0 | Name only: the company publishes no IP list | |
ChatGPT-User (fake) | OpenAI | 10 | 0 | Fake: name doesn't match the company's addresses |
OAI-SearchBot (fake) | OpenAI | 9 | 1 | Fake: name doesn't match the company's addresses |
Claude-User | Anthropic | 9 | 0 | IP checked since Nov 17 (the company's list is new) |
Bytespider | ByteDance | 9 | 0 | Name only: the company publishes no IP list |
Signed agent (www.meta.com) | 7 | 0 | Name only: the company publishes no IP list | |
Signed agent (unfragile.ai) | 7 | 0 | Name only: the company publishes no IP list | |
Perplexity-User (fake) | Perplexity | 7 | 0 | Fake: name doesn't match the company's addresses |
DuckAssistBot (fake) | DuckDuckGo | 7 | 0 | Fake: name doesn't match the company's addresses |
CCBot (fake) | Common Crawl | 7 | 0 | Fake: name doesn't match the company's addresses |
PerplexityBot (fake) | Perplexity | 6 | 6 | Fake: name doesn't match the company's addresses |
YouBot | You.com | 5 | 0 | Name only: the company publishes no IP list |
cohere-ai | Cohere | 4 | 0 | Name only: the company publishes no IP list |
DuckAssistBot (disguised) | DuckDuckGo | 4 | 0 | Disguised: AI company address, browser name |
ClaudeBot (fake) | Anthropic | 4 | 1 | Fake: name doesn't match the company's addresses |
Claude-SearchBot (fake) | Anthropic | 4 | 0 | Fake: name doesn't match the company's addresses |
CCBot | Common Crawl | 4 | 0 | Real: IP matches the published list |
Signed agent (bot.duckduckgo.com) (fake) | 2 | 0 | Fake: name doesn't match the company's addresses | |
Signed agent (shapbot.parallel.ai) | 1 | 0 | Name only: the company publishes no IP list | |
Signed agent (api.lyrenth.com) | 1 | 0 | Name only: the company publishes no IP list |
What they read most
/browse/api | 15,235 |
/browse/skill | 6,427 |
/browse/mcp | 6,336 |
/browse/agent | 2,913 |
/browse/service | 2,041 |
/market | 218 |
/browse/listing | 97 |
/ | 70 |
Which crawlers read the rules
Reads of the files where this site states its rules for crawlers, by AI crawlers whose address matches their company's published list (last 30 days).
| Crawler | /robots.txt | /llms.txt | /.well-known/ai-terms |
|---|---|---|---|
Signed agent (agent.agentdrives.com) | 67 | 4 | 0 |
OAI-SearchBot | 38 | 0 | 0 |
Signed agent (brick.blue) | 12 | 12 | 0 |
Signed agent (agenstry.com) | 19 | 0 | 0 |
ClaudeBot | 9 | 0 | 1 |
Signed agent (exauser.crawler.exa.ai) | 3 | 7 | 0 |
Signed agent (crawler.exa.ai) | 4 | 6 | 0 |
PerplexityBot | 8 | 0 | 0 |
GoogleOther | 0 | 4 | 1 |
ShapBot | 3 | 0 | 1 |
Signed agent (prober.tersign.ai) | 2 | 0 | 0 |
CCBot | 1 | 1 | 0 |
Amazonbot | 0 | 0 | 1 |
ChatGPT-User | 0 | 1 | 0 |
Signed agent (achivx.com) | 1 | 0 | 0 |
Signed agent (unfragile.ai) | 1 | 0 | 0 |
How to read this
Only our marketplace listings (third-party APIs, MCP servers and agents) are priced: AI crawlers asking for one get a $0.001 price instead of the page. Our own pages stay free to AI on purpose, so assistants can read and cite them; that's why a crawler can have many requests and none asked to pay. Most don't pay yet; some stop, like the one in our case study. "Fake" means a request used a crawler's name from an address its company doesn't publish. "Name only" means the company (ByteDance, Meta, Amazon and others) publishes no address list, so the name can't be checked. Anthropic and Mistral started publishing their lists recently; their visits are checked by IP from Nov 17, 2026, and earlier ones were counted by name only.
See the same thing for your own site: check your server log in your browser, see whether someone copied your pages, or start the free report, no wallet needed.