GPTBot
OpenAI crawler, type: ai training.
- 4,002,047sites name it in robots.txt
- 2,591,834block it fully (64.8%)
- 1,046,980block some paths
- 363,233allow it explicitly
About this crawler
Crawls content that may be used to make OpenAI generative AI foundation models more useful and safe, i.e. training.
- User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot- robots.txt token
gptbot- Respects robots.txt
- yes
- Published IP ranges
- https://openai.com/gptbot.json
- Operator page
- https://developers.openai.com/api/docs/bots
- Disallowing GPTBot signals that site content should not be used to train generative models.
- Version number may change.
Including sites that block it only through the * group, 7.86% of all sites with a robots.txt keep it out.