AI crawlers / which sites block them
Which websites block AI crawlers?
Of 347 well-known websites we checked on 2026-11-16, 121 (35%) block at least one AI crawler in robots.txt. The most blocked is CCBot, shut out by 91 (26%). None of them answers AI crawlers with a price yet.
By kind of site
| Kind of site | Sites checked | Block at least one | Block GPTBot |
|---|---|---|---|
| News | 37 | 33 (89%) | 19 (51%) |
| Reference | 24 | 17 (71%) | 11 (46%) |
| Shopping | 21 | 8 (38%) | 4 (19%) |
| Social and community | 30 | 19 (63%) | 17 (57%) |
| Developer tools | 38 | 3 (8%) | 0 (0%) |
| AI companies | 29 | 3 (10%) | 0 (0%) |
| Finance | 26 | 3 (12%) | 2 (8%) |
| Travel | 20 | 3 (15%) | 2 (10%) |
| Education | 24 | 8 (33%) | 4 (17%) |
| Government and public bodies | 24 | 1 (4%) | 1 (4%) |
| Entertainment and media | 29 | 17 (59%) | 11 (38%) |
| SaaS and business software | 45 | 6 (13%) | 3 (7%) |
Most blocked AI crawlers
| Crawler | Blocked by |
|---|---|
CCBot | 91 of 347 (26%) |
Bytespider | 90 of 347 (26%) |
ClaudeBot | 83 of 347 (24%) |
Diffbot | 74 of 347 (21%) |
GPTBot | 74 of 347 (21%) |
meta-externalagent | 74 of 347 (21%) |
anthropic-ai | 72 of 347 (21%) |
cohere-ai | 67 of 347 (19%) |
omgili | 67 of 347 (19%) |
Applebot-Extended | 66 of 347 (19%) |
Amazonbot | 65 of 347 (19%) |
FacebookBot | 64 of 347 (18%) |
Google-Extended | 63 of 347 (18%) |
PerplexityBot | 61 of 347 (18%) |
ImagesiftBot | 57 of 347 (16%) |
Every site
News
usatoday.com 26 · cnn.com 25 · reuters.com 25 · telegraph.co.uk 25 · wsj.com 24 · lemonde.fr 23 · nbcnews.com 22 · nytimes.com 22 · theverge.com 22 · vox.com 22 · arstechnica.com 21 · dw.com 21 · spiegel.de 21 · bbc.com 20 · huffpost.com 20 · newyorker.com 19 · wired.com 19 · theatlantic.com 17 · bloomberg.com 15 · theguardian.com 15 · economist.com 14 · forbes.com 14 · ft.com 14 · apnews.com 12 · washingtonpost.com 12 · elpais.com 11 · techcrunch.com 10 · abcnews.go.com 8 · aljazeera.com 7 · axios.com 6 · businessinsider.com 6 · latimes.com 6 · cbsnews.com 1 · engadget.com 0 · foxnews.com 0 · independent.co.uk 0 · time.com 0
Reference
yelp.com 29 · imdb.com 26 · wikihow.com 24 · allrecipes.com 22 · investopedia.com 22 · glassdoor.com 16 · snopes.com 16 · healthline.com 12 · quora.com 12 · tripadvisor.com 11 · weather.com 9 · timeanddate.com 5 · zillow.com 5 · dictionary.com 3 · webmd.com 3 · goodreads.com 2 · who.int 1 · archive.org 0 · britannica.com 0 · howstuffworks.com 0 · merriam-webster.com 0 · wikipedia.org 0 · wiktionary.org 0 · worldometers.info 0
Shopping
amazon.com 23 · ebay.com 9 · mercadolibre.com 7 · aliexpress.com 6 · chewy.com 1 · flipkart.com 1 · ikea.com 1 · temu.com 1 · bhphotovideo.com 0 · costco.com 0 · etsy.com 0 · homedepot.com 0 · newegg.com 0 · nike.com 0 · nordstrom.com 0 · rakuten.com 0 · shein.com 0 · shopify.com 0 · target.com 0 · walmart.com 0 · wayfair.com 0
Social and community
facebook.com 29 · instagram.com 29 · nextdoor.com 29 · reddit.com 29 · threads.net 29 · x.com 29 · lobste.rs 28 · linkedin.com 26 · pinterest.com 26 · flickr.com 22 · snapchat.com 19 · tiktok.com 18 · letterboxd.com 13 · mastodon.social 13 · tumblr.com 11 · medium.com 8 · vimeo.com 6 · strava.com 4 · meetup.com 1 · bsky.app 0 · dev.to 0 · deviantart.com 0 · discord.com 0 · hashnode.com 0 · news.ycombinator.com 0 · producthunt.com 0 · soundcloud.com 0 · substack.com 0 · twitch.tv 0 · youtube.com 0
Developer tools
postman.com 5 · geeksforgeeks.org 4 · github.com 1 · angular.dev 0 · apache.org 0 · atlassian.com 0 · bitbucket.org 0 · cloudflare.com 0 · crates.io 0 · datadoghq.com 0 · developer.mozilla.org 0 · digitalocean.com 0 · djangoproject.com 0 · docker.com 0 · freecodecamp.org 0 · gitlab.com 0 · go.dev 0 · heroku.com 0 · jetbrains.com 0 · kubernetes.io 0 · laravel.com 0 · netlify.com 0 · nginx.org 0 · nodejs.org 0 · packagist.org 0 · pypi.org 0 · python.org 0 · react.dev 0 · rubygems.org 0 · rubyonrails.org 0 · rust-lang.org 0 · sentry.io 0 · stripe.com 0 · supabase.com 0 · twilio.com 0 · vercel.com 0 · vuejs.org 0 · w3schools.com 0
AI companies
poe.com 27 · notion.so 1 · together.ai 1 · ai21.com 0 · anthropic.com 0 · character.ai 0 · cohere.com 0 · cursor.com 0 · deepl.com 0 · deepmind.google 0 · elevenlabs.io 0 · groq.com 0 · huggingface.co 0 · jasper.ai 0 · langchain.com 0 · llamaindex.ai 0 · mistral.ai 0 · openai.com 0 · perplexity.ai 0 · pinecone.io 0 · replicate.com 0 · replit.com 0 · runwayml.com 0 · scale.com 0 · stability.ai 0 · weaviate.io 0 · writer.com 0 · x.ai 0 · you.com 0
Finance
marketwatch.com 23 · cnbc.com 19 · seekingalpha.com 14 · bankofamerica.com 0 · bankrate.com 0 · binance.com 0 · capitalone.com 0 · chase.com 0 · citi.com 0 · coinbase.com 0 · creditkarma.com 0 · fidelity.com 0 · intuit.com 0 · kraken.com 0 · morningstar.com 0 · nerdwallet.com 0 · nyse.com 0 · paypal.com 0 · revolut.com 0 · robinhood.com 0 · sofi.com 0 · tradingview.com 0 · vanguard.com 0 · visa.com 0 · wellsfargo.com 0 · wise.com 0
Travel
airbnb.com 5 · lonelyplanet.com 1 · skyscanner.net 1 · aa.com 0 · agoda.com 0 · amtrak.com 0 · booking.com 0 · expedia.com 0 · hilton.com 0 · kayak.com 0 · lufthansa.com 0 · lyft.com 0 · marriott.com 0 · priceline.com 0 · rome2rio.com 0 · southwest.com 0 · trainline.com 0 · trivago.com 0 · uber.com 0 · vrbo.com 0
Education
academia.edu 20 · nature.com 18 · ted.com 15 · science.org 6 · jstor.org 4 · researchgate.net 2 · coursera.org 1 · udacity.com 1 · arxiv.org 0 · berkeley.edu 0 · brilliant.org 0 · cam.ac.uk 0 · chegg.com 0 · codecademy.com 0 · columbia.edu 0 · cornell.edu 0 · duolingo.com 0 · edx.org 0 · harvard.edu 0 · mit.edu 0 · princeton.edu 0 · springer.com 0 · stanford.edu 0 · yale.edu 0
Government and public bodies
congress.gov 26 · admin.ch 0 · bundesregierung.de 0 · canada.ca 0 · cdc.gov 0 · census.gov 0 · europa.eu 0 · fda.gov 0 · ftc.gov 0 · gouvernement.fr 0 · gov.uk 0 · imf.org 0 · irs.gov 0 · loc.gov 0 · nasa.gov 0 · nhs.uk 0 · noaa.gov 0 · oecd.org 0 · sec.gov 0 · un.org 0 · usa.gov 0 · whitehouse.gov 0 · worldbank.org 0 · wto.org 0
Entertainment and media
ign.com 25 · netflix.com 20 · pitchfork.com 19 · primevideo.com 18 · billboard.com 17 · hollywoodreporter.com 17 · rollingstone.com 17 · variety.com 17 · gamespot.com 14 · polygon.com 14 · hulu.com 13 · nba.com 9 · espn.com 8 · kotaku.com 8 · metacritic.com 5 · disney.com 2 · playstation.com 1 · apple.com 0 · crunchyroll.com 0 · fifa.com 0 · max.com 0 · mlb.com 0 · nfl.com 0 · paramountplus.com 0 · peacocktv.com 0 · rottentomatoes.com 0 · spotify.com 0 · steampowered.com 0 · xbox.com 0
SaaS and business software
sap.com 15 · canva.com 13 · figma.com 8 · amplitude.com 1 · calendly.com 1 · loom.com 1 · adobe.com 0 · airtable.com 0 · asana.com 0 · auth0.com 0 · box.com 0 · clickup.com 0 · databricks.com 0 · docusign.com 0 · dropbox.com 0 · freshworks.com 0 · google.com 0 · hubspot.com 0 · ibm.com 0 · intercom.com 0 · linear.app 0 · mailchimp.com 0 · microsoft.com 0 · miro.com 0 · mixpanel.com 0 · monday.com 0 · mongodb.com 0 · namecheap.com 0 · okta.com 0 · rippling.com 0 · salesforce.com 0 · segment.com 0 · shopify.dev 0 · slack.com 0 · snowflake.com 0 · squarespace.com 0 · trello.com 0 · typeform.com 0 · wix.com 0 · wordpress.com 0 · workday.com 0 · xero.com 0 · zapier.com 0 · zendesk.com 0 · zoom.us 0
The number after each site is how many of the 29 AI crawlers its robots.txt blocks. 53 more sites could not be read and are left out. The data is free to reuse with a link back (CC BY 4.0): download as JSON.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.