opencrawlers

googleother

Google crawler, type: ai training.

About this crawler

Generic crawler used by various Google product teams to fetch publicly accessible content, e.g. one-off crawls for internal research and development.

User-Agent
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
robots.txt token
googleother
Respects robots.txt
yes (third party: source)
Operator page
https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Including sites that block it only through the * group, 4.27% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind