google-cloudvertexbot
Google crawler, type: ai training.
- 62,089sites name it in robots.txt
- 48,791block it fully (78.6%)
- 6,212block some paths
- 7,086allow it explicitly
About this crawler
Crawls requested by site owners for building Vertex AI Agents; it has no effect on Google Search or other products.
- User-Agent
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search)(third party: source)- robots.txt token
google-cloudvertexbot- Respects robots.txt
- yes
- Published IP ranges
- https://developers.google.com/static/crawling/ipranges/common-crawlers.json
- Operator page
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Page shows Google-CloudVertexBot as a UA substring; full UA string not captured in the fetch, so left null
- robots.txt can target Google-CloudVertexBot (page also lists Googlebot as a token for it)
- IP file is the common-crawlers list
Including sites that block it only through the * group, 4.24% of all sites with a robots.txt keep it out.