AI crawlers / which sites block them / Government and public bodies

Which government sites block AI crawlers?

Of 24 government sites we checked on 2026-11-16, 1 (4%) block at least one AI crawler in robots.txt, and 1 (4%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
congress.gov26 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
admin.ch0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
bundesregierung.de0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
canada.ca0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
cdc.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
census.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
europa.eu0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
fda.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
ftc.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
gouvernement.fr0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
gov.uk0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
imf.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
irs.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
loc.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
nasa.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
nhs.uk0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
noaa.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
oecd.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
sec.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
un.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
usa.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
whitehouse.gov0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
worldbank.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
wto.org0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 6 sites in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: News · Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Education · Entertainment and media · SaaS and business software