oai-searchbot
OpenAI crawler, type: ai search.
- 522,887sites name it in robots.txt
- 205,892block it fully (39.4%)
- 89,579block some paths
- 227,416allow it explicitly
About this crawler
Used to surface websites in search results in ChatGPT's search features.
- User-Agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot- robots.txt token
oai-searchbot- Respects robots.txt
- yes
- Published IP ranges
- https://openai.com/searchbot.json
- Operator page
- https://developers.openai.com/api/docs/bots
- Opting out prevents appearance in ChatGPT search answers; roughly 24 hour update lag.
- Separate from GPTBot (training).
Including sites that block it only through the * group, 4.43% of all sites with a robots.txt keep it out.