velenpublicwebcrawler
Velen crawler, type: ai training.
- 149,611sites name it in robots.txt
- 110,405block it fully (73.8%)
- 7,092block some paths
- 32,114allow it explicitly
About this crawler
Hunter's robot analyzes public web pages for business data used in Hunter's lead generation tools.
- User-Agent
Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)- robots.txt token
velenpublicwebcrawler- Respects robots.txt
- yes
- Operator page
- https://hunter.io/robot
- Page says it strictly respects robots.txt disallow and allow rules.
- Never visits more than one page every two seconds per website.
- Operated by Velen; page maintained by Hunter; no IP list.
Including sites that block it only through the * group, 4.33% of all sites with a robots.txt keep it out.