PricingGet API key
Menu
ToolsPay-per-crawlAI crawlersMarketplaceDocsHow x402 worksPricingGet API key

State of AI crawler blocking

We read the robots.txt of 60 well-known websites on 2026-11-15 and checked 29 AI crawlers against each. Of the 53 sites whose robots.txt we could read, 37 (70%) block at least one AI crawler and 0 block all 29. Bytespider, CCBot and ClaudeBot are tied as the most blocked, each shut out by 32 of 53.

Check your own siteCharge crawlers instead of blocking

By type of site

CategoryBlock at least oneAverage blocked
News16 of 1716.7 of 29
Tech and developers4 of 94.4 of 29
Reference and education5 of 87.1 of 29
Shopping and services5 of 97.9 of 29
Social and community7 of 1016.7 of 29

Most blocked AI crawlers

CrawlerOperatorBlocked by
BytespiderByteDance32 of 53 (60%)
CCBotCommon Crawl32 of 53 (60%)
ClaudeBotAnthropic32 of 53 (60%)
meta-externalagentMeta28 of 53 (53%)
PerplexityBotPerplexity28 of 53 (53%)
AmazonbotAmazon26 of 53 (49%)
anthropic-aiAnthropic26 of 53 (49%)
DiffbotDiffbot26 of 53 (49%)
cohere-aiCohere25 of 53 (47%)
Applebot-ExtendedApple24 of 53 (45%)
Google-ExtendedGoogle24 of 53 (45%)
Claude-UserAnthropic23 of 53 (43%)
GPTBotOpenAI23 of 53 (43%)
PetalBotHuawei23 of 53 (43%)
YouBotYou.com23 of 53 (43%)
Claude-SearchBotAnthropic22 of 53 (42%)
FacebookBotMeta22 of 53 (42%)
Perplexity-UserPerplexity22 of 53 (42%)
ImagesiftBotImageSift21 of 53 (40%)
omgiliWebz.io21 of 53 (40%)
TimpibotTimpi20 of 53 (38%)
DuckAssistBotDuckDuckGo18 of 53 (34%)
MistralAI-UserMistral AI18 of 53 (34%)
GoogleOtherGoogle17 of 53 (32%)
ChatGPT-UserOpenAI16 of 53 (30%)
AI2BotAllen Institute for AI14 of 53 (26%)
OAI-SearchBotOpenAI13 of 53 (25%)
ElevenlabsBotElevenLabs0 of 53 (0%)
ElevenlabsCrawlerElevenLabs0 of 53 (0%)

Every site

SiteCategoryAI crawlers blocked
facebook.comSocial and community27 of 29
instagram.comSocial and community27 of 29
reddit.comSocial and community27 of 29
x.comSocial and community27 of 29
yelp.comShopping and services27 of 29
usatoday.comNews26 of 29
cnn.comNews25 of 29
imdb.comReference and education24 of 29
linkedin.comSocial and community24 of 29
pinterest.comSocial and community24 of 29
reuters.comNews24 of 29
amazon.comShopping and services23 of 29
nytimes.comNews22 of 29
theverge.comNews22 of 29
wsj.comNews22 of 29
arstechnica.comTech and developers21 of 29
newyorker.comNews19 of 29
wired.comNews19 of 29
nature.comReference and education18 of 29
theatlantic.comNews17 of 29
bloomberg.comNews15 of 29
theguardian.comNews15 of 29
economist.comNews14 of 29
ft.comNews14 of 29
apnews.comNews12 of 29
quora.comReference and education12 of 29
washingtonpost.comNews12 of 29
tripadvisor.comShopping and services11 of 29
tumblr.comSocial and community11 of 29
techcrunch.comTech and developers10 of 29
ebay.comShopping and services9 of 29
medium.comTech and developers8 of 29
axios.comNews6 of 29
goodreads.comReference and education2 of 29
coursera.orgReference and education1 of 29
github.comTech and developers1 of 29
ikea.comShopping and services1 of 29
arxiv.orgReference and education0 of 29
booking.comShopping and services0 of 29
britannica.comReference and education0 of 29
dev.toTech and developers0 of 29
discord.comSocial and community0 of 29
etsy.comShopping and services0 of 29
foxnews.comNews0 of 29
news.ycombinator.comTech and developers0 of 29
producthunt.comTech and developers0 of 29
substack.comTech and developers0 of 29
target.comShopping and services0 of 29
twitch.tvSocial and community0 of 29
walmart.comShopping and services0 of 29
wikipedia.orgReference and education0 of 29
wordpress.comTech and developers0 of 29
youtube.comSocial and community0 of 29

How we measured

For each site we fetched /robots.txt and applied RFC 9309 matching to the path / for each crawler's token, using the crawler's own group or else the * group. A crawler counts as blocked when / is disallowed for it. robots.txt is a request, not enforcement, and some sites also block crawlers at the network level, which this doesn't see. We couldn't read robots.txt for bbc.com, npr.org, politico.com, stackoverflow.com, khanacademy.org, sciencedirect.com, bestbuy.com; they're left out. Each site links to a live report.

Blocking is all or nothing: a blocked crawler can't cite or link to you either. Pay-per-crawl answers AI crawlers with HTTP 402 and a price, so the ones that pay get the page.