AI crawlers / which sites block them / Reference
Which reference sites block AI crawlers?
Of 24 reference sites we checked on 2026-11-16, 17 (71%) block at least one AI crawler in robots.txt, and 11 (46%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| yelp.com | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| imdb.com | 26 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| wikihow.com | 24 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| allrecipes.com | 22 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| investopedia.com | 22 of 29 | Allowed | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| glassdoor.com | 16 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| snopes.com | 16 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| healthline.com | 12 of 29 | Blocked | Blocked | Allowed | Allowed | Blocked | 2026-11-16 |
| quora.com | 12 of 29 | Blocked | Blocked | Allowed | Blocked | Allowed | 2026-11-16 |
| tripadvisor.com | 11 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| weather.com | 9 of 29 | Allowed | Allowed | Allowed | Allowed | Blocked | 2026-11-16 |
| timeanddate.com | 5 of 29 | Allowed | Allowed | Allowed | Blocked | Allowed | 2026-11-16 |
| zillow.com | 5 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| dictionary.com | 3 of 29 | Blocked | Allowed | Allowed | Allowed | Blocked | 2026-11-16 |
| webmd.com | 3 of 29 | Blocked | Blocked | Allowed | Allowed | Blocked | 2026-11-16 |
| goodreads.com | 2 of 29 | Blocked | Allowed | Allowed | Allowed | Blocked | 2026-11-16 |
| who.int | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Blocked | 2026-11-16 |
| archive.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| britannica.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| howstuffworks.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| merriam-webster.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| wikipedia.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| wiktionary.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| worldometers.info | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers. 6 sites in this group could not be read.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: News · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software