CCBot
Common Crawl crawler, type: ai training.
- 3,367,481sites name it in robots.txt
- 2,243,509block it fully (66.6%)
- 878,480block some paths
- 245,492allow it explicitly
About this crawler
Crawler of Common Crawl, a non-profit that produces and maintains an open repository of web crawl data.
- User-Agent
CCBot/2.0 (https://commoncrawl.org/faq/)- robots.txt token
ccbot- Respects robots.txt
- yes
- Published IP ranges
- https://index.commoncrawl.org/ccbot.json
- Operator page
- https://commoncrawl.org/ccbot
- Page gives User-agent: CCBot / Disallow: / as the blocking rule (compliance stated implicitly via robots.txt blocking)
- Page warns of crawlers falsely identifying as CCBot; verify with reverse DNS against .crawl.commoncrawl.org
- An opt-out registry exists
Including sites that block it only through the * group, 7.34% of all sites with a robots.txt keep it out.