AI crawlers / which sites block them / Reference

Which reference sites block AI crawlers?

Of 24 reference sites we checked on 2026-11-16, 17 (71%) block at least one AI crawler in robots.txt, and 11 (46%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
yelp.com29 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
imdb.com26 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
wikihow.com24 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
allrecipes.com22 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
investopedia.com22 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
glassdoor.com16 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
snopes.com16 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
healthline.com12 of 29BlockedBlockedAllowedAllowedBlocked2026-11-16
quora.com12 of 29BlockedBlockedAllowedBlockedAllowed2026-11-16
tripadvisor.com11 of 29BlockedBlockedBlockedAllowedBlocked2026-11-16
weather.com9 of 29AllowedAllowedAllowedAllowedBlocked2026-11-16
timeanddate.com5 of 29AllowedAllowedAllowedBlockedAllowed2026-11-16
zillow.com5 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
dictionary.com3 of 29BlockedAllowedAllowedAllowedBlocked2026-11-16
webmd.com3 of 29BlockedBlockedAllowedAllowedBlocked2026-11-16
goodreads.com2 of 29BlockedAllowedAllowedAllowedBlocked2026-11-16
who.int1 of 29AllowedAllowedAllowedAllowedBlocked2026-11-16
archive.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
britannica.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
howstuffworks.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
merriam-webster.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
wikipedia.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
wiktionary.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
worldometers.info0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 6 sites in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: News · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software