opencrawlers

youbot

You.com crawler, type: ai search.

About this crawler

YouBot is You.com's web crawler that automatically discovers and indexes web pages to provide real-time, accurate search results for You.com users.

User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36
robots.txt token
youbot
Respects robots.txt
yes
Operator page
https://you.com/docs/youbot

Including sites that block it only through the * group, 4.32% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind