opencrawlers

google-cloudvertexbot

Google crawler, type: ai training.

About this crawler

Crawls requested by site owners for building Vertex AI Agents; it has no effect on Google Search or other products.

User-Agent
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) (third party: source)
robots.txt token
google-cloudvertexbot
Respects robots.txt
yes
Published IP ranges
https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Operator page
https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Including sites that block it only through the * group, 4.24% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind