opencrawlers

Google-Extended

Google crawler, type: ai training.

About this crawler

Robots.txt control token only: manages whether content Google crawls may be used for training future Gemini models (Gemini Apps, Vertex AI API) and for grounding.

robots.txt token
google-extended
Respects robots.txt
unknown
Operator page
https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Including sites that block it only through the * group, 7.17% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind