Google-Extended
Google crawler, type: ai training.
- 3,283,977sites name it in robots.txt
- 2,111,819block it fully (64.3%)
- 905,615block some paths
- 266,543allow it explicitly
About this crawler
Robots.txt control token only: manages whether content Google crawls may be used for training future Gemini models (Gemini Apps, Vertex AI API) and for grounding.
- robots.txt token
google-extended- Respects robots.txt
- unknown
- Operator page
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Page: Google-Extended has no separate HTTP user agent string; crawling uses existing Google user agent strings.
- Does not affect inclusion in Google Search and is not a ranking signal.
- Not listed on the special-case crawlers page; found on the common crawlers page.
Including sites that block it only through the * group, 7.17% of all sites with a robots.txt keep it out.