meta-externalagent
Meta-ExternalAgent is Meta's crawler for uses such as training foundation AI models and improving products by indexing content directly. Meta says you can block it with a robots.txt disallow. Meta AI search uses a different crawler, meta-webindexer, which this rule does not cover.
What it does
Meta says the Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. Meta documents blocking it by adding a disallow rule for meta-externalagent to robots.txt.
It is separate from meta-webindexer, which Meta uses to improve Meta AI search results, and from Meta-ExternalFetcher, which fetches links on a user's request.
In our 2026-11 study, 19.5% of popular sites with a robots.txt blocked it. →
robots.txt token
meta-externalagent
User-agent string
meta-externalagent/1.1How to allow it
User-agent: meta-externalagent
Allow: /How to block it
User-agent: meta-externalagent
Disallow: /How to verify real visits
Meta does not publish IP ranges or a reverse DNS method for this crawler in its crawler documentation, so a user agent match alone cannot prove a request is genuine. Meta directs questions about its crawlers to webmasters@meta.com.