cohere-training-data-crawler
Cohere crawler, type: ai training.
- 907,998sites name it in robots.txt
- 106,341block it fully (11.7%)
- 798,076block some paths
- 3,581allow it explicitly
About this crawler
Web crawler operated by Cohere to download training data for its LLMs supporting its enterprise AI products.
- robots.txt token
cohere-training-data-crawler- Respects robots.txt
- yes
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/cohere-training-data-crawler
- Third-party source (Known Agents / Dark Visitors); operator confirmation not seen.
- Source gives only the token as the user agent, no full UA string.
- Source says it is expected to follow robots.txt and has no published verification method.
Including sites that block it only through the * group, 4.29% of all sites with a robots.txt keep it out.