Four in ten business websites turn AI crawlers away
In our audits of 165 business websites in the US, UK and France, 38% turned away at least one AI crawler at the server, almost always a training or scraper bot. About one site in ten also refused an AI search crawler such as PerplexityBot or OAI-SearchBot, the crawlers that can lead to a citation. Check yours with one request per crawler.
You ask ChatGPT about your own product and it names a competitor, a review site and a forum thread. Sometimes that is a question of content. Often it is a firewall answering AI crawlers with a no, which nobody on your team ever saw because the site works in a browser.
On this page
What did we test?
We request a page several times from the same address: once as an ordinary browser, and once each as 14 AI and search crawlers, including GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-SearchBot from Anthropic and PerplexityBot from Perplexity. If the browser gets the page and a crawler gets a 401, 403, 406, 429, 451, 503 or a challenge page, the site is turning that crawler away.
The sample is 218 business websites in the US, UK and France, audited from 18 to 21 November 2026. We could run the full test on 165. The other 53 refused even our plain browser request or did not answer, so we left them out instead of guessing.
What did we find?
- 38% of the testable sites (63 of 165) turned away at least one AI crawler.
- Every one of those sites refused at least one training or scraper crawler.
Bytespiderwas refused by 35% of all sites,ClaudeBotby 13% andGPTBotby 11%. - 10% of sites (16) also refused at least one AI search crawler, and 5% (8) refused a crawler that opens a page live when a person asks the assistant about it.
- The search crawlers that mattered most were
PerplexityBot(11 sites) andOAI-SearchBot(5 sites). - Cloudflare was behind 32 of the 66 sites that refused any crawler we tested. For another 30 we could not tell which firewall answered.
Why does it happen, and when is it a choice?
A lot of this is probably deliberate. Many owners do not want their content used to train models, and bot-protection settings that block scrapers are easy to switch on. That is a fair choice.
The trouble is that AI companies run separate crawlers for training and for search. Blocking the training crawler does not require blocking the search one, and the search crawler is the one that can lead to your page being cited in an answer.
Do the biggest sites do better?
Not really. In our robots.txt report on the 1,000 most popular domains, 29.9% of the sites with a robots.txt block at least one AI training crawler, and 63.2% of those also block at least one AI search or user crawler. Only 11.0% block training while allowing every AI search crawler, which is the setup most of them probably wanted.
How do I check my own site in ten minutes?
- Open
yoursite.com/robots.txt. For each AI crawler you care about, see whether a group names it and what the group says. A group that names a crawler replaces the general*group completely. - Test what the server does, not what the file says. Request your key page as a browser and as the crawler. A 403, a 429, a page titled
Just a moment...or a response much smaller than the browser's is the signature of a firewall block. - Ask the assistant. With search on, ask ChatGPT or Perplexity the questions your buyers ask and see which sites it cites. If access is fine and you are still missing, the problem is content, not access.
How reliable are these numbers?
Treat them as a pattern, not a census. The sample is not random: these are companies found through ad libraries, job boards and map listings. We test by user agent from one address, while the real crawlers come from the addresses their vendors publish, so some firewalls would treat the real ones differently. And a block is not always a mistake.
Next steps
- robots.txt tester and checker for AI crawlers
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
What share of websites block AI crawlers?
In our audit of 165 testable business websites, 38% turned away at least one AI crawler at the server, mostly training and scraper bots. About 10% also refused an AI search crawler.
Does blocking GPTBot keep my site out of ChatGPT search?
No. OpenAI says each robots.txt setting for its crawlers is independent: GPTBot relates to training, OAI-SearchBot to ChatGPT search. Blocking one does not block the other.
How can a site block a crawler without robots.txt?
A firewall, CDN or bot-protection rule can answer a crawler with a 403, a 429 or a challenge page. robots.txt never shows this, so it has to be tested by requesting the page as the crawler.
Is turning away a crawler always a mistake?
No. Many owners want to keep training crawlers and scrapers out. The mistake is blocking search crawlers by accident, because those are the ones that can cite your page.
Sources
- AISeeWhy: AI Crawler Access Report, 2026-11
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Cloudflare: Manage AI crawlers (AI Crawl Control)