meta-externalagent
Meta crawler, type: ai training.
- 3,184,301sites name it in robots.txt
- 2,108,759block it fully (66.2%)
- 942,998block some paths
- 132,544allow it explicitly
About this crawler
Crawls the web for use cases such as training AI models or improving products by indexing content directly.
- User-Agent
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)- robots.txt token
meta-externalagent- Respects robots.txt
- yes
- Operator page
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
- Page also lists the shorter UA meta-externalagent/1.1; UA URL shown relative on page, expanded here to the page host
- Page says site managers can block it with a robots.txt disallow
- Page did not give an IP list URL in the fetched summary
Including sites that block it only through the * group, 7.2% of all sites with a robots.txt keep it out.