AI crawlers / which sites block them / Education

Which education sites block AI crawlers?

Of 24 education sites we checked on 2026-11-16, 8 (33%) block at least one AI crawler in robots.txt, and 4 (17%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
academia.edu20 of 29BlockedBlockedBlockedAllowedBlocked2026-11-16
nature.com18 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
ted.com15 of 29BlockedBlockedBlockedAllowedBlocked2026-11-16
science.org6 of 29AllowedAllowedBlockedAllowedBlocked2026-11-16
jstor.org4 of 29BlockedBlockedBlockedAllowedBlocked2026-11-16
researchgate.net2 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
coursera.org1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
udacity.com1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
arxiv.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
berkeley.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
brilliant.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
cam.ac.uk0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
chegg.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
codecademy.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
columbia.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
cornell.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
duolingo.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
edx.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
harvard.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
mit.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
princeton.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
springer.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
stanford.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
yale.edu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 6 sites in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: News · Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Government and public bodies · Entertainment and media · SaaS and business software