opencrawlers

CCBot

Common Crawl crawler, type: ai training.

About this crawler

Crawler of Common Crawl, a non-profit that produces and maintains an open repository of web crawl data.

User-Agent
CCBot/2.0 (https://commoncrawl.org/faq/)
robots.txt token
ccbot
Respects robots.txt
yes
Published IP ranges
https://index.commoncrawl.org/ccbot.json
Operator page
https://commoncrawl.org/ccbot

Including sites that block it only through the * group, 7.34% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind