AI crawlers / which sites block them / Government and public bodies
Which government sites block AI crawlers?
Of 24 government sites we checked on 2026-11-16, 1 (4%) block at least one AI crawler in robots.txt, and 1 (4%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| congress.gov | 26 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| admin.ch | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| bundesregierung.de | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| canada.ca | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| cdc.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| census.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| europa.eu | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| fda.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| ftc.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| gouvernement.fr | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| gov.uk | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| imf.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| irs.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| loc.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| nasa.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| nhs.uk | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| noaa.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| oecd.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| sec.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| un.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| usa.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| whitehouse.gov | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| worldbank.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| wto.org | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers. 6 sites in this group could not be read.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: News · Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Travel · Education · Entertainment and media · SaaS and business software