opencrawlers

friendlycrawler

unknown crawler, type: ai training.

About this crawler

AI-oriented web crawling; specific downstream use remains unclear.

User-Agent
FriendlyCrawler
robots.txt token
friendlycrawler
Respects robots.txt
unknown
Source (third party, not confirmed by the operator)
https://botcrawl.com/bots/friendlycrawler/

Including sites that block it only through the * group, 4.26% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind