opencrawlers

phindbot

Phind crawler, type: ai search.

About this crawler

AI answer engine for developers; makes isolated, on-demand requests for real-time web search rather than crawling sites.

User-Agent
PhindBot
robots.txt token
phindbot
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/phindbot

Including sites that block it only through the * group, 4.22% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind