AI crawlers / which sites block them / Developer tools

Which developer sites block AI crawlers?

Of 38 developer sites we checked on 2026-11-16, 3 (8%) block at least one AI crawler in robots.txt, and 0 (0%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
postman.com5 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
geeksforgeeks.org4 of 29AllowedAllowedAllowedAllowedBlocked2026-11-16
github.com1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
angular.dev0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
apache.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
atlassian.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
bitbucket.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
cloudflare.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
crates.io0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
datadoghq.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
developer.mozilla.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
digitalocean.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
djangoproject.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
docker.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
freecodecamp.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
gitlab.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
go.dev0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
heroku.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
jetbrains.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
kubernetes.io0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
laravel.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
netlify.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
nginx.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
nodejs.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
packagist.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
pypi.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
python.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
react.dev0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
rubygems.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
rubyonrails.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
rust-lang.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
sentry.io0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
stripe.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
supabase.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
twilio.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
vercel.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
vuejs.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
w3schools.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 2 sites in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: News · Reference · Shopping · Social and community · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software