claude-searchbot
Anthropic crawler, type: ai search.
- 152,128sites name it in robots.txt
- 63,950block it fully (42%)
- 42,247block some paths
- 45,931allow it explicitly
About this crawler
Navigates the web to improve search result quality for users.
- User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com)(third party: source)- robots.txt token
claude-searchbot- Respects robots.txt
- yes
- Published IP ranges
- https://claude.com/crawling/bots.json
- Operator page
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Separate bot from ClaudeBot (training) and Claude-User (user-initiated).
Including sites that block it only through the * group, 4.25% of all sites with a robots.txt keep it out.