AI crawlers / Google-Extended
Google-Extended: what it is and how to block or charge it
Google-Extended is run by Google. A robots.txt token, not a separate crawler: it controls whether content Googlebot already crawls may be used for Gemini and other Google AI models. Blocking it does not affect Google Search.
| Operator | |
|---|---|
| Purpose | Collects content for AI training |
| robots.txt token | Google-Extended |
| Same operator | GoogleOther |
Block Google-Extended in robots.txt
User-agent: Google-Extended
Disallow: /Because Google-Extended is only a robots.txt token, robots.txt is the only way to control it; it never shows up in your server logs.
Or charge Google-Extended per page
Google-Extended can't be charged directly, because it's a permission token rather than a visitor. Pay-per-crawl works on crawlers that fetch pages themselves, like pay-per-crawl GPTBot and ClaudeBot.
Is Google-Extended reading your site?
See the full list of AI crawlers. Facts here follow each operator's own documentation; tell us if one changes.