AI crawlers / which sites block them / Social and community
Which social and community sites block AI crawlers?
Of 30 social and community sites we checked on 2026-11-16, 19 (63%) block at least one AI crawler in robots.txt, and 17 (57%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| facebook.com | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| instagram.com | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| nextdoor.com | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| reddit.com | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| threads.net | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| x.com | 29 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| lobste.rs | 28 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| linkedin.com | 26 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| pinterest.com | 26 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| flickr.com | 22 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| snapchat.com | 19 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| tiktok.com | 18 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| letterboxd.com | 13 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| mastodon.social | 13 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| tumblr.com | 11 of 29 | Allowed | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| medium.com | 8 of 29 | Blocked | Blocked | Allowed | Allowed | Allowed | 2026-11-16 |
| vimeo.com | 6 of 29 | Allowed | Allowed | Allowed | Allowed | Blocked | 2026-11-16 |
| strava.com | 4 of 29 | Blocked | Blocked | Blocked | Allowed | Allowed | 2026-11-16 |
| meetup.com | 1 of 29 | Blocked | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| bsky.app | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| dev.to | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| deviantart.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| discord.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| hashnode.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| news.ycombinator.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| producthunt.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| soundcloud.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| substack.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| twitch.tv | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| youtube.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: News · Reference · Shopping · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software