We read the robots.txt of the 10,000 most popular sites. Here is what they do about AI crawlers
Of the 6,192 domains in the Tranco top 10,000 that serve a robots.txt, 23.2% block at least one AI training crawler and 12.3% block at least one AI search or user crawler. 51.6% of the sites that block training also block AI search. Only 11.2% opt out of training while allowing every AI search crawler.
In November we read the robots.txt of the 1,000 most popular domains. This time we read 10,000, with the same method, and added a check of how the picture changes as sites get smaller. The headline holds: blocking training and blocking search are treated as one decision, although they are two.
On this page
How did we measure it?
We took the top 10,000 domains of the Tranco list (list ID JZWZY) and fetched each domain's home page, robots.txt and llms.txt once on 21 November 2026, as AISeeWhy's own user agent. We judged each robots.txt by RFC 9309 for the home page (/) and for 16 AI and search crawlers.
6,192 of the 10,000 domains served a robots.txt (a successful answer that is not an HTML page). 1,121 answered with a 4xx, which means no rules and every crawler allowed, and 2,687 did not serve one (no HTTPS site, errors, timeouts or an HTML page instead). Every percentage below is of the 6,192. The list includes CDNs and APIs that serve no website.
What did we find?
- 23.2% (1,437 domains) block at least one AI training crawler.
- 12.3% (762) block at least one AI search or user crawler, so the pages should not be used or cited by the answers that rely on it.
- 51.6% of the domains that block a training crawler (742 of 1,437) also block at least one AI search or user crawler.
- Only 11.2% (695) take the selective route: they block training crawlers and allow every AI search and user crawler.
- 6.5% (401) state their AI preferences with
Content-Signal. - 12.4% of all 10,000 domains (1,240) publish an
llms.txt. - For comparison, 2.0% (124) block
Googleboton the home page.
Which crawlers are blocked most often?
Share of the 6,192 domains whose robots.txt disallows the home page for the crawler:
Bytespider18.7%,CCBot18.5%,GPTBot17.4%,ClaudeBot16.5%.Meta-ExternalAgent15.5%,Google-Extended14.9%,Applebot-Extended14.6%.PerplexityBot10.4% is the AI search crawler blocked most often, ahead ofOAI-SearchBot7.0% andClaude-SearchBot7.0%.- The crawlers that open a page when a person asks:
ChatGPT-User8.9%,Perplexity-User7.7%,Claude-User7.4%.
Do bigger sites block more?
Yes, and they mix up training and search more often. We split the list by rank:
- Ranks 1 to 1,000 (560 domains with a robots.txt): 28.6% block training, 18.6% block AI search, and 63.1% of the training blockers also block AI search.
- Ranks 1,001 to 3,000 (1,256 domains): 23.5%, 13.5% and 55.6%.
- Ranks 3,001 to 10,000 (4,376 domains): 22.4%, 11.2% and 48.6%.
The selective route is rare at every size: 10.5%, 10.4% and 11.5% of the domains in the three groups.
Has anything changed since our November 12 report?
We re-ran the first 1,000 domains with the same list. The result is stable within about a point: 28.6% block training (29.9% on 12 November), 18.6% block AI search (19.5%), and 63.1% of the training blockers also block AI search (63.2%). The denominators differ slightly (560 domains with a robots.txt now, 518 then) because this run follows a domain's redirect to its main host before fetching.
What does it mean for your robots.txt?
Most AI companies document separate crawlers for training and for search and say the settings are independent. Disallowing GPTBot leaves OAI-SearchBot alone, and disallowing ClaudeBot leaves Claude-SearchBot alone. If you want to opt out of training and still be cited, block the training tokens and allow the search and user tokens, and repeat any private paths in every group, because a named group does not inherit from *.
How reliable are these numbers?
They are robots.txt rules for the home page, read by one parser on one day. They do not measure firewall blocks, which can refuse a crawler that robots.txt allows. They cannot tell what each site intended, and the Tranco list includes domains that serve no website. A rule that blocks a search crawler is not proof of a mistake, only of a choice that is often made without knowing the difference. You may quote the figures with a link to this page.
Next steps
- robots.txt tester and checker for AI crawlers
- llms.txt checker and validator
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
What share of popular sites block AI crawlers?
Among the 6,192 domains of the Tranco top 10,000 that serve a robots.txt, 23.2% block at least one AI training crawler and 12.3% block at least one AI search or user crawler.
Do sites that block AI training also block AI search?
Often. 51.6% of the sites that block a training crawler also block at least one AI search or user crawler, although the vendors document them as separate crawlers with independent settings.
Which AI crawler is blocked most often?
Among training crawlers Bytespider (18.7%) and CCBot (18.5%), then GPTBot (17.4%). Among AI search crawlers PerplexityBot is blocked most often, on 10.4% of the domains.
Why do the numbers differ from your 1,000-site report?
The first 1,000 domains give almost the same percentages (28.6% and 18.6% against 29.9% and 19.5%). The 10,000-site figures are lower because smaller sites block less.
Sources
- AISeeWhy: AI Crawler Access Report, 2026-11
- Tranco: a research-oriented top sites ranking
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers