opencrawlers

GPTBot

OpenAI crawler, type: ai training.

About this crawler

Crawls content that may be used to make OpenAI generative AI foundation models more useful and safe, i.e. training.

User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
robots.txt token
gptbot
Respects robots.txt
yes
Published IP ranges
https://openai.com/gptbot.json
Operator page
https://developers.openai.com/api/docs/bots

Including sites that block it only through the * group, 7.86% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind