opencrawlers

diffbot

Diffbot crawler, type: ai training.

About this crawler

Diffbot crawls the web generally to build a search engine and knowledge graph, similar to Googlebot or Bingbot.

User-Agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36 (compatible; Diffbot-User/1.0; +https://www.diffbot.com) (third party: source)
robots.txt token
diffbot
Respects robots.txt
yes
Operator page
https://www.diffbot.com/docs/crawl/faq/robots-txt

Including sites that block it only through the * group, 4.48% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind