opencrawlers

brightbot

Bright Data crawler, type: ai training.

About this crawler

BrightBot is Bright Data's crawler; domain owners control its crawl behavior with a collectors.txt file set in the Bright Data Webmaster Console.

User-Agent
Brightbot 1.0
robots.txt token
brightbot
Respects robots.txt
unknown
Operator page
https://docs.brightdata.com/general/webmaster-console.md

Including sites that block it only through the * group, 4.2% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind