youbot
You.com crawler, type: ai search.
- 1,046,420sites name it in robots.txt
- 157,612block it fully (15.1%)
- 812,540block some paths
- 76,268allow it explicitly
About this crawler
YouBot is You.com's web crawler that automatically discovers and indexes web pages to provide real-time, accurate search results for You.com users.
- User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36- robots.txt token
youbot- Respects robots.txt
- yes
- Operator page
- https://you.com/docs/youbot
- Page states YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings; robots.txt is cached for 30 minutes.
- Page states legitimate requests come from 68.67.112.0/24 (listed on the page, no separate IP list URL); reverse DNS pattern youbot-{IP-with-hyphens}.search.you.com.
- Supports verification via Cloudflare Web Bot Auth signatures; contact spider@you.com. docs.you.com/youbot redirects (301) to you.com/docs/youbot.
Including sites that block it only through the * group, 4.32% of all sites with a robots.txt keep it out.