AI crawlers / which sites block them / Shopping

Which shopping sites block AI crawlers?

Of 21 shopping sites we checked on 2026-11-16, 8 (38%) block at least one AI crawler in robots.txt, and 4 (19%) block GPTBot.

SiteAI crawlers blockedGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotChecked
amazon.com23 of 29BlockedBlockedBlockedBlockedBlocked2026-11-16
ebay.com9 of 29BlockedBlockedAllowedBlockedBlocked2026-11-16
mercadolibre.com7 of 29BlockedBlockedAllowedBlockedAllowed2026-11-16
aliexpress.com6 of 29BlockedBlockedBlockedAllowedBlocked2026-11-16
chewy.com1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
flipkart.com1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
ikea.com1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
temu.com1 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
bhphotovideo.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
costco.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
etsy.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
homedepot.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
newegg.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
nike.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
nordstrom.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
rakuten.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
shein.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
shopify.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
target.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
walmart.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16
wayfair.com0 of 29AllowedAllowedAllowedAllowedAllowed2026-11-16

Open a site for the full list of 29 crawlers. 9 sites in this group could not be read.

How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.

Other kinds of site: News · Reference · Social and community · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software