AI crawlers / which sites block them

Which websites block AI crawlers?

Of 347 well-known websites we checked on 2026-11-16, 121 (35%) block at least one AI crawler in robots.txt. The most blocked is CCBot, shut out by 91 (26%). None of them answers AI crawlers with a price yet.

By kind of site

Kind of siteSites checkedBlock at least oneBlock GPTBot
News3733 (89%)19 (51%)
Reference2417 (71%)11 (46%)
Shopping218 (38%)4 (19%)
Social and community3019 (63%)17 (57%)
Developer tools383 (8%)0 (0%)
AI companies293 (10%)0 (0%)
Finance263 (12%)2 (8%)
Travel203 (15%)2 (10%)
Education248 (33%)4 (17%)
Government and public bodies241 (4%)1 (4%)
Entertainment and media2917 (59%)11 (38%)
SaaS and business software456 (13%)3 (7%)

Most blocked AI crawlers

CrawlerBlocked by
CCBot91 of 347 (26%)
Bytespider90 of 347 (26%)
ClaudeBot83 of 347 (24%)
Diffbot74 of 347 (21%)
GPTBot74 of 347 (21%)
meta-externalagent74 of 347 (21%)
anthropic-ai72 of 347 (21%)
cohere-ai67 of 347 (19%)
omgili67 of 347 (19%)
Applebot-Extended66 of 347 (19%)
Amazonbot65 of 347 (19%)
FacebookBot64 of 347 (18%)
Google-Extended63 of 347 (18%)
PerplexityBot61 of 347 (18%)
ImagesiftBot57 of 347 (16%)

Every site

News

usatoday.com 26 · cnn.com 25 · reuters.com 25 · telegraph.co.uk 25 · wsj.com 24 · lemonde.fr 23 · nbcnews.com 22 · nytimes.com 22 · theverge.com 22 · vox.com 22 · arstechnica.com 21 · dw.com 21 · spiegel.de 21 · bbc.com 20 · huffpost.com 20 · newyorker.com 19 · wired.com 19 · theatlantic.com 17 · bloomberg.com 15 · theguardian.com 15 · economist.com 14 · forbes.com 14 · ft.com 14 · apnews.com 12 · washingtonpost.com 12 · elpais.com 11 · techcrunch.com 10 · abcnews.go.com 8 · aljazeera.com 7 · axios.com 6 · businessinsider.com 6 · latimes.com 6 · cbsnews.com 1 · engadget.com 0 · foxnews.com 0 · independent.co.uk 0 · time.com 0

Reference

yelp.com 29 · imdb.com 26 · wikihow.com 24 · allrecipes.com 22 · investopedia.com 22 · glassdoor.com 16 · snopes.com 16 · healthline.com 12 · quora.com 12 · tripadvisor.com 11 · weather.com 9 · timeanddate.com 5 · zillow.com 5 · dictionary.com 3 · webmd.com 3 · goodreads.com 2 · who.int 1 · archive.org 0 · britannica.com 0 · howstuffworks.com 0 · merriam-webster.com 0 · wikipedia.org 0 · wiktionary.org 0 · worldometers.info 0

Shopping

amazon.com 23 · ebay.com 9 · mercadolibre.com 7 · aliexpress.com 6 · chewy.com 1 · flipkart.com 1 · ikea.com 1 · temu.com 1 · bhphotovideo.com 0 · costco.com 0 · etsy.com 0 · homedepot.com 0 · newegg.com 0 · nike.com 0 · nordstrom.com 0 · rakuten.com 0 · shein.com 0 · shopify.com 0 · target.com 0 · walmart.com 0 · wayfair.com 0

Social and community

facebook.com 29 · instagram.com 29 · nextdoor.com 29 · reddit.com 29 · threads.net 29 · x.com 29 · lobste.rs 28 · linkedin.com 26 · pinterest.com 26 · flickr.com 22 · snapchat.com 19 · tiktok.com 18 · letterboxd.com 13 · mastodon.social 13 · tumblr.com 11 · medium.com 8 · vimeo.com 6 · strava.com 4 · meetup.com 1 · bsky.app 0 · dev.to 0 · deviantart.com 0 · discord.com 0 · hashnode.com 0 · news.ycombinator.com 0 · producthunt.com 0 · soundcloud.com 0 · substack.com 0 · twitch.tv 0 · youtube.com 0

Developer tools

postman.com 5 · geeksforgeeks.org 4 · github.com 1 · angular.dev 0 · apache.org 0 · atlassian.com 0 · bitbucket.org 0 · cloudflare.com 0 · crates.io 0 · datadoghq.com 0 · developer.mozilla.org 0 · digitalocean.com 0 · djangoproject.com 0 · docker.com 0 · freecodecamp.org 0 · gitlab.com 0 · go.dev 0 · heroku.com 0 · jetbrains.com 0 · kubernetes.io 0 · laravel.com 0 · netlify.com 0 · nginx.org 0 · nodejs.org 0 · packagist.org 0 · pypi.org 0 · python.org 0 · react.dev 0 · rubygems.org 0 · rubyonrails.org 0 · rust-lang.org 0 · sentry.io 0 · stripe.com 0 · supabase.com 0 · twilio.com 0 · vercel.com 0 · vuejs.org 0 · w3schools.com 0

AI companies

poe.com 27 · notion.so 1 · together.ai 1 · ai21.com 0 · anthropic.com 0 · character.ai 0 · cohere.com 0 · cursor.com 0 · deepl.com 0 · deepmind.google 0 · elevenlabs.io 0 · groq.com 0 · huggingface.co 0 · jasper.ai 0 · langchain.com 0 · llamaindex.ai 0 · mistral.ai 0 · openai.com 0 · perplexity.ai 0 · pinecone.io 0 · replicate.com 0 · replit.com 0 · runwayml.com 0 · scale.com 0 · stability.ai 0 · weaviate.io 0 · writer.com 0 · x.ai 0 · you.com 0

Finance

marketwatch.com 23 · cnbc.com 19 · seekingalpha.com 14 · bankofamerica.com 0 · bankrate.com 0 · binance.com 0 · capitalone.com 0 · chase.com 0 · citi.com 0 · coinbase.com 0 · creditkarma.com 0 · fidelity.com 0 · intuit.com 0 · kraken.com 0 · morningstar.com 0 · nerdwallet.com 0 · nyse.com 0 · paypal.com 0 · revolut.com 0 · robinhood.com 0 · sofi.com 0 · tradingview.com 0 · vanguard.com 0 · visa.com 0 · wellsfargo.com 0 · wise.com 0

Travel

airbnb.com 5 · lonelyplanet.com 1 · skyscanner.net 1 · aa.com 0 · agoda.com 0 · amtrak.com 0 · booking.com 0 · expedia.com 0 · hilton.com 0 · kayak.com 0 · lufthansa.com 0 · lyft.com 0 · marriott.com 0 · priceline.com 0 · rome2rio.com 0 · southwest.com 0 · trainline.com 0 · trivago.com 0 · uber.com 0 · vrbo.com 0

Education

academia.edu 20 · nature.com 18 · ted.com 15 · science.org 6 · jstor.org 4 · researchgate.net 2 · coursera.org 1 · udacity.com 1 · arxiv.org 0 · berkeley.edu 0 · brilliant.org 0 · cam.ac.uk 0 · chegg.com 0 · codecademy.com 0 · columbia.edu 0 · cornell.edu 0 · duolingo.com 0 · edx.org 0 · harvard.edu 0 · mit.edu 0 · princeton.edu 0 · springer.com 0 · stanford.edu 0 · yale.edu 0

Government and public bodies

congress.gov 26 · admin.ch 0 · bundesregierung.de 0 · canada.ca 0 · cdc.gov 0 · census.gov 0 · europa.eu 0 · fda.gov 0 · ftc.gov 0 · gouvernement.fr 0 · gov.uk 0 · imf.org 0 · irs.gov 0 · loc.gov 0 · nasa.gov 0 · nhs.uk 0 · noaa.gov 0 · oecd.org 0 · sec.gov 0 · un.org 0 · usa.gov 0 · whitehouse.gov 0 · worldbank.org 0 · wto.org 0

Entertainment and media

ign.com 25 · netflix.com 20 · pitchfork.com 19 · primevideo.com 18 · billboard.com 17 · hollywoodreporter.com 17 · rollingstone.com 17 · variety.com 17 · gamespot.com 14 · polygon.com 14 · hulu.com 13 · nba.com 9 · espn.com 8 · kotaku.com 8 · metacritic.com 5 · disney.com 2 · playstation.com 1 · apple.com 0 · crunchyroll.com 0 · fifa.com 0 · max.com 0 · mlb.com 0 · nfl.com 0 · paramountplus.com 0 · peacocktv.com 0 · rottentomatoes.com 0 · spotify.com 0 · steampowered.com 0 · xbox.com 0

SaaS and business software

sap.com 15 · canva.com 13 · figma.com 8 · amplitude.com 1 · calendly.com 1 · loom.com 1 · adobe.com 0 · airtable.com 0 · asana.com 0 · auth0.com 0 · box.com 0 · clickup.com 0 · databricks.com 0 · docusign.com 0 · dropbox.com 0 · freshworks.com 0 · google.com 0 · hubspot.com 0 · ibm.com 0 · intercom.com 0 · linear.app 0 · mailchimp.com 0 · microsoft.com 0 · miro.com 0 · mixpanel.com 0 · monday.com 0 · mongodb.com 0 · namecheap.com 0 · okta.com 0 · rippling.com 0 · salesforce.com 0 · segment.com 0 · shopify.dev 0 · slack.com 0 · snowflake.com 0 · squarespace.com 0 · trello.com 0 · typeform.com 0 · wix.com 0 · wordpress.com 0 · workday.com 0 · xero.com 0 · zapier.com 0 · zendesk.com 0 · zoom.us 0

The number after each site is how many of the 29 AI crawlers its robots.txt blocks. 53 more sites could not be read and are left out. The data is free to reuse with a link back (CC BY 4.0): download as JSON.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.