AI crawlers / which sites block them / AI companies

Which AI company sites block AI crawlers?

Of 29 AI company sites we checked on 2026-11-16, 3 (10%) block at least one AI crawler in robots.txt, and 0 (0%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
poe.com27 of 29AllowedBlockedAllowedBlockedBlocked2026-11-16
notion.so1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
together.ai1 of 29AllowedAllowedBlockedAllowedAllowed2026-11-16
ai21.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
anthropic.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
character.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
cohere.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
cursor.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
deepl.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
deepmind.google0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
elevenlabs.io0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
groq.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
huggingface.co0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
jasper.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
langchain.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
llamaindex.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
mistral.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
openai.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
perplexity.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
pinecone.io0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
replicate.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
replit.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
runwayml.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
scale.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
stability.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
weaviate.io0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
writer.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
x.ai0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
you.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 1 site in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: News · Reference · Shopping · Social and community · Developer tools · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software