AI crawlers / which sites block them / Travel
Which travel sites block AI crawlers?
Of 20 travel sites we checked on 2026-11-16, 3 (15%) block at least one AI crawler in robots.txt, and 2 (10%) block GPTBot.
| Site | AI crawlers blocked | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Checked |
|---|---|---|---|---|---|---|---|
| airbnb.com | 5 of 29 | Blocked | Blocked | Allowed | Allowed | Allowed | 2026-11-16 |
| lonelyplanet.com | 1 of 29 | Blocked | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| skyscanner.net | 1 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| aa.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| agoda.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| amtrak.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| booking.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| expedia.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| hilton.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| kayak.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| lufthansa.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| lyft.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| marriott.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| priceline.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| rome2rio.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| southwest.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| trainline.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| trivago.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| uber.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
| vrbo.com | 0 of 29 | Allowed | Allowed | Allowed | Allowed | Allowed | 2026-11-16 |
Open a site for the full list of 29 crawlers. 10 sites in this group could not be read.
How we know: we read each site's public robots.txt and test 29 AI crawler names against it. A site can also block crawlers at its firewall, which nobody outside can see, so "allows" here means "robots.txt doesn't say no". Each site is rechecked about once a week.
Other kinds of site: News · Reference · Shopping · Social and community · Developer tools · AI companies · Finance · Education · Government and public bodies · Entertainment and media · SaaS and business software