We read the robots.txt of the 10,000 most popular sites. Here is what they do about AI crawlers

Published

Short answer

Of the 6,192 domains in the Tranco top 10,000 that serve a robots.txt, 23.2% block at least one AI training crawler and 12.3% block at least one AI search or user crawler. 51.6% of the sites that block training also block AI search. Only 11.2% opt out of training while allowing every AI search crawler.

In November we read the robots.txt of the 1,000 most popular domains. This time we read 10,000, with the same method, and added a check of how the picture changes as sites get smaller. The headline holds: blocking training and blocking search are treated as one decision, although they are two.

On this page
  1. How did we measure it?
  2. What did we find?
  3. Which crawlers are blocked most often?
  4. Do bigger sites block more?
  5. Has anything changed since our November 12 report?
  6. What does it mean for your robots.txt?
  7. How reliable are these numbers?

How did we measure it?

We took the top 10,000 domains of the Tranco list (list ID JZWZY) and fetched each domain's home page, robots.txt and llms.txt once on 21 November 2026, as AISeeWhy's own user agent. We judged each robots.txt by RFC 9309 for the home page (/) and for 16 AI and search crawlers.

6,192 of the 10,000 domains served a robots.txt (a successful answer that is not an HTML page). 1,121 answered with a 4xx, which means no rules and every crawler allowed, and 2,687 did not serve one (no HTTPS site, errors, timeouts or an HTML page instead). Every percentage below is of the 6,192. The list includes CDNs and APIs that serve no website.

What did we find?

Which crawlers are blocked most often?

Share of the 6,192 domains whose robots.txt disallows the home page for the crawler:

Do bigger sites block more?

Yes, and they mix up training and search more often. We split the list by rank:

The selective route is rare at every size: 10.5%, 10.4% and 11.5% of the domains in the three groups.

Has anything changed since our November 12 report?

We re-ran the first 1,000 domains with the same list. The result is stable within about a point: 28.6% block training (29.9% on 12 November), 18.6% block AI search (19.5%), and 63.1% of the training blockers also block AI search (63.2%). The denominators differ slightly (560 domains with a robots.txt now, 518 then) because this run follows a domain's redirect to its main host before fetching.

What does it mean for your robots.txt?

Most AI companies document separate crawlers for training and for search and say the settings are independent. Disallowing GPTBot leaves OAI-SearchBot alone, and disallowing ClaudeBot leaves Claude-SearchBot alone. If you want to opt out of training and still be cited, block the training tokens and allow the search and user tokens, and repeat any private paths in every group, because a named group does not inherit from *.

How reliable are these numbers?

They are robots.txt rules for the home page, read by one parser on one day. They do not measure firewall blocks, which can refuse a crawler that robots.txt allows. They cannot tell what each site intended, and the Tranco list includes domains that serve no website. A rule that blocks a search crawler is not proof of a mistake, only of a choice that is often made without knowing the difference. You may quote the figures with a link to this page.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

What share of popular sites block AI crawlers?

Among the 6,192 domains of the Tranco top 10,000 that serve a robots.txt, 23.2% block at least one AI training crawler and 12.3% block at least one AI search or user crawler.

Do sites that block AI training also block AI search?

Often. 51.6% of the sites that block a training crawler also block at least one AI search or user crawler, although the vendors document them as separate crawlers with independent settings.

Which AI crawler is blocked most often?

Among training crawlers Bytespider (18.7%) and CCBot (18.5%), then GPTBot (17.4%). Among AI search crawlers PerplexityBot is blocked most often, on 10.4% of the domains.

Why do the numbers differ from your 1,000-site report?

The first 1,000 domains give almost the same percentages (28.6% and 18.6% against 29.9% and 19.5%). The 10,000-site figures are lower because smaller sites block less.

Sources

More articles