meta-externalfetcher
Meta crawler, type: ai assistant.
- 133,933sites name it in robots.txt
- 74,919block it fully (55.9%)
- 15,689block some paths
- 43,325allow it explicitly
About this crawler
Fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities.
- User-Agent
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)- robots.txt token
meta-externalfetcher- Respects robots.txt
- no
- Operator page
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
- Page says it may bypass robots.txt rules because fetches are user-requested
- Page also lists the shorter UA meta-externalfetcher/1.1
- Blocking via robots.txt is therefore not guaranteed
Including sites that block it only through the * group, 4.27% of all sites with a robots.txt keep it out.