opencrawlers

omgili

Webz.io crawler, type: ai training.

About this crawler

Webz.io describes a crawler pair (Webzio and Webzio-extended) that collects news, blogs, discussions and reviews for its data products; Webzio-extended tags data as usable or not usable for AI/ML training.

User-Agent
omgili (third party: source)
robots.txt token
omgili
Respects robots.txt
yes
Operator page
https://webz.io/bot.html

Including sites that block it only through the * group, 4.49% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind