diffbot
Diffbot crawler, type: ai training.
- 304,734sites name it in robots.txt
- 221,071block it fully (72.5%)
- 15,613block some paths
- 68,050allow it explicitly
About this crawler
Diffbot crawls the web generally to build a search engine and knowledge graph, similar to Googlebot or Bingbot.
- User-Agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36 (compatible; Diffbot-User/1.0; +https://www.diffbot.com)(third party: source)- robots.txt token
diffbot- Respects robots.txt
- yes
- Operator page
- https://www.diffbot.com/docs/crawl/faq/robots-txt
- States it is not used for AI training and that Diffbot does not crawl pages for training generative AI foundation models.
- A second agent, Diffbot-User, covers requests made on behalf of a human user of Diffbot software.
- Robots.txt can be overridden in specific cases such as a partnership with the site; page text fetched did not show a full UA string.
Including sites that block it only through the * group, 4.48% of all sites with a robots.txt keep it out.