applebot-extended
Apple crawler, type: ai training.
- 3,036,559sites name it in robots.txt
- 1,959,285block it fully (64.5%)
- 932,662block some paths
- 144,612allow it explicitly
About this crawler
Control token only, no crawler of its own: it tells Apple whether content already crawled by Applebot may be used to train Apple foundation models for generative AI features.
- robots.txt token
applebot-extended- Respects robots.txt
- yes
- Published IP ranges
- https://search.developer.apple.com/applebot.json
- Operator page
- https://support.apple.com/en-us/119829
- Applebot-Extended does not crawl; the actual crawler is Applebot (UA ends with Applebot/0.1; +http://www.apple.com/go/applebot)
- Disallowing Applebot-Extended opts out of training use but Applebot still powers Spotlight, Siri and Safari search
- IP list URL is the Applebot list; Apple's iTMS crawler does not follow robots.txt
Including sites that block it only through the * group, 6.97% of all sites with a robots.txt keep it out.