opencrawlers

oai-searchbot

OpenAI crawler, type: ai search.

About this crawler

Used to surface websites in search results in ChatGPT's search features.

User-Agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
robots.txt token
oai-searchbot
Respects robots.txt
yes
Published IP ranges
https://openai.com/searchbot.json
Operator page
https://developers.openai.com/api/docs/bots

Including sites that block it only through the * group, 4.43% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind