AI crawlers / which sites block them / News

Which news sites block AI crawlers?

Of 37 news sites we checked on 2026-11-16, 33 (89%) block at least one AI crawler in robots.txt, and 19 (51%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
usatoday.com26 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
cnn.com25 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
reuters.com25 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
telegraph.co.uk25 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
wsj.com24 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
lemonde.fr23 of 29AllowedBlockedBlockedAllowedBlocked2026-11-16
nbcnews.com22 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
nytimes.com22 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
theverge.com22 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
vox.com22 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
arstechnica.com21 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
dw.com21 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
spiegel.de21 of 29BlockedBlockedAllowedAllowedBlocked2026-11-16
bbc.com20 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
huffpost.com20 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
newyorker.com19 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
wired.com19 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
theatlantic.com17 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
bloomberg.com15 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
theguardian.com15 of 29AllowedBlockedAllowedBlockedBlocked2026-11-16
economist.com14 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
forbes.com14 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
ft.com14 of 29AllowedBlockedBlockedBlockedBlocked2026-11-16
apnews.com12 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
washingtonpost.com12 of 29AllowedBlockedAllowedBlockedBlocked2026-11-16
elpais.com11 of 29AllowedBlockedAllowedBlockedBlocked2026-11-16
techcrunch.com10 of 29BlockedBlockedAllowedAllowedBlocked2026-11-16
abcnews.go.com8 of 29BlockedAllowedBlockedAllowedBlocked2026-11-16
aljazeera.com7 of 29BlockedBlockedAllowedBlockedAllowed2026-11-16
axios.com6 of 29AllowedAllowedAllowedAllowedBlocked2026-11-16
businessinsider.com6 of 29AllowedBlockedAllowedAllowedBlocked2026-11-16
latimes.com6 of 29BlockedBlockedAllowedAllowedAllowed2026-11-16
cbsnews.com1 of 29BlockedAllowedAllowedAllowedAllowed2026-11-16
engadget.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
foxnews.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
independent.co.uk0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
time.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 3 sites in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software