AI crawlers / which sites block them / Shopping
Which shopping sites block AI crawlers?
Of 21 shopping sites we checked on 2026-11-16, 8 (38%) block at least one AI crawler in robots.txt, and 4 (19%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| amazon.com | 23 of 29 | Blocked | Blocked | Blocked | Blocked | Blocked | 2026-11-16 |
| ebay.com | 9 of 29 | Blocked | Blocked | Allowed | Blocked | Blocked | 2026-11-16 |
| mercadolibre.com | 7 of 29 | Blocked | Blocked | Allowed | Blocked | Allowed | 2026-11-16 |
| aliexpress.com | 6 of 29 | Blocked | Blocked | Blocked | Allowed | Blocked | 2026-11-16 |
| chewy.com | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| flipkart.com | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| ikea.com | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| temu.com | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| bhphotovideo.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| costco.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| etsy.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| homedepot.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| newegg.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| nike.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| nordstrom.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| rakuten.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| shein.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| shopify.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| target.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| walmart.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| wayfair.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers. 9 sites in this group could not be read.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: News · Reference · Social and community · Developer tools · AI companies · Finance · Travel · Education · Government and public bodies · Entertainment and media · SaaS and business software