opencrawlers

cohere-training-data-crawler

Cohere crawler, type: ai training.

About this crawler

Web crawler operated by Cohere to download training data for its LLMs supporting its enterprise AI products.

robots.txt token
cohere-training-data-crawler
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/cohere-training-data-crawler

Including sites that block it only through the * group, 4.29% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind