omgilibot
Webz.io crawler, type: ai training.
- 1,016,525sites name it in robots.txt
- 172,351block it fully (17%)
- 801,231block some paths
- 42,943allow it explicitly
About this crawler
Webz.io describes a crawler pair (Webzio and Webzio-extended) that collects news, blogs, discussions and reviews for its data products; Webzio-extended tags data as usable or not usable for AI/ML training.
- User-Agent
omgilibot(third party: source)- robots.txt token
omgilibot- Respects robots.txt
- yes
- Operator page
- https://webz.io/bot.html
- Page names Omgilibot as the legacy crawler, Webzio as current crawler and Webzio-extended as the validation bot.
- States it only provides data tagged as explicitly permitted for AI purposes.
- Page gives no full user-agent strings and no IP list.
Including sites that block it only through the * group, 4.36% of all sites with a robots.txt keep it out.