friendlycrawler
unknown crawler, type: ai training.
- 99,303sites name it in robots.txt
- 61,268block it fully (61.7%)
- 5,960block some paths
- 32,075allow it explicitly
About this crawler
AI-oriented web crawling; specific downstream use remains unclear.
- User-Agent
FriendlyCrawler- robots.txt token
friendlycrawler- Respects robots.txt
- unknown
- Source (third party, not confirmed by the operator)
- https://botcrawl.com/bots/friendlycrawler/
- Third-party source (botcrawl.com); operator listed as Unknown, so who runs it is not established.
- Source says robots.txt behavior is not sufficiently established and the identity is low confidence and spoofable.
- knownagents.com/agents/friendlycrawler returned 404; no operator page found.
Including sites that block it only through the * group, 4.26% of all sites with a robots.txt keep it out.