opencrawlers

meta-externalagent

Meta crawler, type: ai training.

About this crawler

Crawls the web for use cases such as training AI models or improving products by indexing content directly.

User-Agent
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
robots.txt token
meta-externalagent
Respects robots.txt
yes
Operator page
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers

Including sites that block it only through the * group, 7.2% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind