AI crawlers / live on this site

AI crawlers on aipaygen.com, live

We run our own pay-per-crawl here. These are the AI crawler requests this site got in the last 30 days, each checked against the IP addresses its company publishes. Updated every minute, nothing edited out.

45,806AI crawler requests
6,146asked to pay (402)
115fake or disguised
0paid by a crawler

Counting since 2026-11-16. Last 30 days. Every number is a live database count.

Verified 972 requests by Web Bot Auth signature and 8,394 by the company's published IP ranges. Check what we see for your own request.

By day

Which crawlers

CrawlerCompanyRequestsAsked to payIP check
meta-externalagentMeta28,998533Name only: the company publishes no IP list
GoogleOtherGoogle9,6582,807Real: IP matches the published list
AmazonbotAmazon4,2392,683Name only: the company publishes no IP list
GPTBotOpenAI73035Real: IP matches the published list
ClaudeBotAnthropic4871IP checked since Nov 17 (the company's list is new)
ElevenlabsCrawlerElevenLabs46880Name only: the company publishes no IP list
ShapBot3530Name only: the company publishes no IP list
Signed agent (achivx.com)1410Name only: the company publishes no IP list
Signed agent (prober.tersign.ai)1160Name only: the company publishes no IP list
Signed agent (brick.blue)1020Name only: the company publishes no IP list
Signed agent (crawler.exa.ai)850Name only: the company publishes no IP list
Signed agent (agenstry.com)720Name only: the company publishes no IP list
ChatGPT-UserOpenAI650Real: IP matches the published list
Signed agent (agentconcierge.biz)560Name only: the company publishes no IP list
Claude-User (fake)Anthropic350Fake: name doesn't match the company's addresses
OAI-SearchBotOpenAI241Real: IP matches the published list
Signed agent (agent.agentdrives.com)210Name only: the company publishes no IP list
GPTBot (fake)OpenAI2035Fake: name doesn't match the company's addresses
PerplexityBotPerplexity196Real: IP matches the published list
Signed agent (exauser.crawler.exa.ai)100Name only: the company publishes no IP list
ChatGPT-User (fake)OpenAI100Fake: name doesn't match the company's addresses
OAI-SearchBot (fake)OpenAI91Fake: name doesn't match the company's addresses
Claude-UserAnthropic90IP checked since Nov 17 (the company's list is new)
BytespiderByteDance90Name only: the company publishes no IP list
Signed agent (www.meta.com)70Name only: the company publishes no IP list
Signed agent (unfragile.ai)70Name only: the company publishes no IP list
Perplexity-User (fake)Perplexity70Fake: name doesn't match the company's addresses
DuckAssistBot (fake)DuckDuckGo70Fake: name doesn't match the company's addresses
CCBot (fake)Common Crawl70Fake: name doesn't match the company's addresses
PerplexityBot (fake)Perplexity66Fake: name doesn't match the company's addresses
YouBotYou.com50Name only: the company publishes no IP list
cohere-aiCohere40Name only: the company publishes no IP list
DuckAssistBot (disguised)DuckDuckGo40Disguised: AI company address, browser name
ClaudeBot (fake)Anthropic41Fake: name doesn't match the company's addresses
Claude-SearchBot (fake)Anthropic40Fake: name doesn't match the company's addresses
CCBotCommon Crawl40Real: IP matches the published list
Signed agent (bot.duckduckgo.com) (fake)20Fake: name doesn't match the company's addresses
Signed agent (shapbot.parallel.ai)10Name only: the company publishes no IP list
Signed agent (api.lyrenth.com)10Name only: the company publishes no IP list

What they read most

/browse/api15,235
/browse/skill6,427
/browse/mcp6,336
/browse/agent2,913
/browse/service2,041
/market218
/browse/listing97
/70

Which crawlers read the rules

Reads of the files where this site states its rules for crawlers, by AI crawlers whose address matches their company's published list (last 30 days).

Crawler/robots.txt/llms.txt/.well-known/ai-terms
Signed agent (agent.agentdrives.com)6740
OAI-SearchBot3800
Signed agent (brick.blue)12120
Signed agent (agenstry.com)1900
ClaudeBot901
Signed agent (exauser.crawler.exa.ai)370
Signed agent (crawler.exa.ai)460
PerplexityBot800
GoogleOther041
ShapBot301
Signed agent (prober.tersign.ai)200
CCBot110
Amazonbot001
ChatGPT-User010
Signed agent (achivx.com)100
Signed agent (unfragile.ai)100

How to read this

Only our marketplace listings (third-party APIs, MCP servers and agents) are priced: AI crawlers asking for one get a $0.001 price instead of the page. Our own pages stay free to AI on purpose, so assistants can read and cite them; that's why a crawler can have many requests and none asked to pay. Most don't pay yet; some stop, like the one in our case study. "Fake" means a request used a crawler's name from an address its company doesn't publish. "Name only" means the company (ByteDance, Meta, Amazon and others) publishes no address list, so the name can't be checked. Anthropic and Mistral started publishing their lists recently; their visits are checked by IP from Nov 17, 2026, and earlier ones were counted by name only.

See the same thing for your own site: check your server log in your browser, see whether someone copied your pages, or start the free report, no wallet needed.