AI crawlers / which sites block them / News
Which news sites block AI crawlers?
Of 37 news sites we checked on 2026-11-16, 33 (89%) block at least one AI crawler in robots.txt, and 19 (51%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| usatoday.com | 26 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| cnn.com | 25 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| reuters.com | 25 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| telegraph.co.uk | 25 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| wsj.com | 24 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| lemonde.fr | 23 of 29 | Allowed | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| nbcnews.com | 22 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| nytimes.com | 22 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| theverge.com | 22 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| vox.com | 22 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| arstechnica.com | 21 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| dw.com | 21 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| spiegel.de | 21 of 29 | Blocked | Blocked | Allowed | Allowed | Blocked | 2026-11-16 |
| bbc.com | 20 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| huffpost.com | 20 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| newyorker.com | 19 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| wired.com | 19 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| theatlantic.com | 17 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| bloomberg.com | 15 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| theguardian.com | 15 of 29 | Allowed | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| economist.com | 14 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| forbes.com | 14 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| ft.com | 14 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| apnews.com | 12 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| washingtonpost.com | 12 of 29 | Allowed | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| elpais.com | 11 of 29 | Allowed | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| techcrunch.com | 10 of 29 | Blocked | Blocked | Allowed | Allowed | Blocked | 2026-11-16 |
| abcnews.go.com | 8 of 29 | Blocked | Allowed | Blocked | Allowed | Blocked | 2026-11-16 |
| aljazeera.com | 7 of 29 | Blocked | Blocked | Allowed | Blocked | Allowed | 2026-11-16 |
| axios.com | 6 of 29 | Allowed | Allowed | Allowed | Allowed | Blocked | 2026-11-16 |
| businessinsider.com | 6 of 29 | Allowed | Blocked | Allowed | Allowed | Blocked | 2026-11-16 |
| latimes.com | 6 of 29 | Blocked | Blocked | Allowed | Allowed | Allowed | 2026-11-16 |
| cbsnews.com | 1 of 29 | Blocked | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| engadget.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| foxnews.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| independent.co.uk | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| time.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers. 3 sites in this group could not be read.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software