AI Crawler Access Report, 2026-11
Short answer
29.9% of popular sites with a robots.txt block at least one AI training crawler, and 19.5% block at least one AI search crawler, which keeps those pages out of AI answers. Only 11.0% block training while allowing every AI search crawler. Data: robots.txt of the top 1000 domains, 2026-11-12.
We read the robots.txt of the 1000 most popular domains (Tranco list) on 2026-11-12 and applied the rules for the home page to 16 AI and search crawlers, the way RFC 9309 says crawlers read them.
What did we find?
- 29.9% of sites with a robots.txt block at least one AI training crawler (GPTBot, ClaudeBot, CCBot, Google-Extended and others).
- 19.5% block at least one AI search or user crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and others). These crawlers are asked not to read those home pages, so the AI answers that rely on them should not use or cite them.
- 57 sites (11.0%) take the selective route: they block training crawlers but allow every AI search and user crawler.
- Only 18 sites (3.5%) state their AI preferences with Content-Signal.
- 132 of the 1000 domains (13.2%) publish an llms.txt; 117 of them follow the llmstxt.org format.
- For comparison, 14 sites (2.7%) block Googlebot on the home page.
Which AI crawlers are blocked most often?
| Crawler | Type | Sites blocking it | Share | Named in robots.txt |
|---|---|---|---|---|
| CCBot | AI training | 125 | 24.1% | 121 |
| Bytespider | AI training | 125 | 24.1% | 111 |
| GPTBot | AI training | 111 | 21.4% | 137 |
| ClaudeBot | AI training | 110 | 21.2% | 130 |
| Meta-ExternalAgent | AI training | 101 | 19.5% | 93 |
| Google-Extended | AI training | 94 | 18.1% | 114 |
| Applebot-Extended | AI training | 93 | 18.0% | 90 |
| PerplexityBot | Search | 86 | 16.6% | 121 |
| Perplexity-User | User fetch | 74 | 14.3% | 67 |
| Claude-User | User fetch | 68 | 13.1% | 75 |
| Claude-SearchBot | Search | 67 | 12.9% | 76 |
| ChatGPT-User | User fetch | 67 | 12.9% | 101 |
| OAI-SearchBot | Search | 58 | 11.2% | 91 |
| Applebot | Search | 20 | 3.9% | 39 |
| Bingbot | Search | 17 | 3.3% | 62 |
| Googlebot | Search | 14 | 2.7% | 81 |
How did we measure it?
- Domains: the top 1000 of the Tranco top 1M, downloaded 2026-11-12. The list ranks registrable domains, so it includes CDNs, APIs and trackers that serve no website.
- 518 domains returned a robots.txt over HTTPS; 103 answered 4xx (no file: every crawler is allowed) and 379 did not serve one (no HTTPS site, errors or timeouts). Percentages are of the 518 sites with a robots.txt.
- "Blocked" means the rules disallow the home page (/) for that crawler token. "Named" counts files with a group for that token rather than only User-agent: *.
- This is what robots.txt asks for. It does not show firewall or bot-protection blocks, which often stop AI crawlers that robots.txt allows; the full AISeeWhy check tests those.
How do I check my own site?
See which crawlers your robots.txt allows with the free robots.txt checker, validate your llms.txt, or run the full check to test firewalls and what crawlers actually receive.
Data: AISeeWhy, 2026-11-12. You may quote these numbers with a link to this page.