AI crawlers / which sites block them / Education
Which education sites block AI crawlers?
Of 24 education sites we checked on 2026-11-16, 8 (33%) block at least one AI crawler in robots.txt, and 4 (17%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| academia.edu | 20 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| nature.com | 18 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| ted.com | 15 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| science.org | 6 of 29 | Allowed | Allowed | Blocked | Allowed | Blocked | 2026-11-16 |
| jstor.org | 4 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| researchgate.net | 2 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| coursera.org | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| udacity.com | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| arxiv.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| berkeley.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| brilliant.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| cam.ac.uk | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| chegg.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| codecademy.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| columbia.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| cornell.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| duolingo.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| edx.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| harvard.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| mit.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| princeton.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| springer.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| stanford.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| yale.edu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers. 6 sites in this group could not be read.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: News · Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Government and public bodies · Entertainment and media · SaaS and business software