How to check if a website blocks AI crawlers
Paste any address and see whether it blocks GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers: in robots.txt, in the firewall, or both.
Free · No signup · Results in 30 seconds
Which crawlers are blocked
Training, search and user crawlers are shown separately, because blocking GPTBot does not block OAI-SearchBot, and blocking ClaudeBot does not block Claude-SearchBot.
The line that decides
For each crawler, robots.txt is read by RFC 9309 rules: the group that names it, or User-agent: * as the fallback, and the rule that wins.
Firewall refusals
The page is requested as 14 crawlers and as a normal browser. Blocks and challenge pages are flagged and the firewall that answered is named.
A fix for every finding
Each finding comes with the change to make. The free check shows your score and the first finding; the full report is $29, one time.
A site can block AI crawlers in two places: robots.txt, which asks crawlers to stay away, and the server or firewall, which actually refuses them. Read robots.txt for each crawler's token, falling back to User-agent: *, then fetch the page as the crawler and compare it with a browser. A browser extension does the robots.txt part for 16 AI crawlers in one click.
Check by hand in 3 steps
- Pick the crawler
Each AI company runs separate ones. Training:
GPTBot,ClaudeBot. Search:OAI-SearchBot,Claude-SearchBot,PerplexityBot. On request of a person:ChatGPT-User,Claude-User,Perplexity-User. - Read robots.txt for it
Open
https://example.com/robots.txtand find the group that names the crawler. If no group does, theUser-agent: *group applies; if there is neither, everything is allowed. - Fetch the page as the crawler
Request the page with the crawler's user agent and compare status and size with a browser's. A 403, a challenge page or a much smaller 200 means the firewall refuses it.
Frequently asked questions
Is a site without robots.txt open to AI crawlers?
Yes, as far as robots.txt goes. A 404 for robots.txt means there are no rules, so every crawler may fetch every page. A firewall can still block crawlers.
Does the extension show firewall blocks?
No. It reads robots.txt from your browser, which shows what the site asks crawlers to do. Firewall and bot protection blocks need a request sent as the crawler, which the full check on aiseewhy.com does.
Can I check a site that is not mine?
Yes. robots.txt is public, and the extension reads only the robots.txt and llms.txt files the site publishes, the same files every crawler reads.
Read the full guide
Whether you are checking your own site after a change or looking at someone else's before you rely on it, the question "does this site block AI crawlers?" has a precise answer. It depends on which crawler you mean, because AI companies run separate crawlers for training, for search and for live visits, and a site can treat each one differently. This guide shows three checks, from the quickest to the most complete.
Where can a site block AI crawlers?
There are three layers, and a crawler has to pass all of them to use a page.
- robots.txt: a public file of rules that well-behaved crawlers read before they fetch anything. It is a request, not a lock.
- The server, CDN or firewall: it decides what is actually sent back. A bot protection rule can answer a crawler with a 403, a 429 or a challenge page even when robots.txt allows it.
- The page itself: if the main text appears only after JavaScript runs, a crawler that reads the initial HTML gets an almost empty page, although nothing blocked it.
Most quick checks only look at the first layer. That is a good start, as long as you remember the other two.
How do I read robots.txt for one AI crawler?
Open https://example.com/robots.txt in a browser and find the group that applies to the crawler. A group starts with one or more User-agent lines. The crawler uses the group that names its token, such as GPTBot; matching is not case-sensitive. If no group names it, it uses the User-agent: * group. If there is neither, everything is allowed.
Inside the group, the rule with the longest matching path wins, and when an Allow and a Disallow are equally long, Allow wins. So the file below blocks GPTBot from the whole site but lets every other crawler in:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: / Also look at how the file answers. Under RFC 9309, a 404 means there are no rules, so everything is allowed. A server error (5xx) or no answer at all means a crawler must treat the whole site as disallowed. A robots.txt address that returns an HTML page is a common mistake: crawlers find no valid rules in it.
How do I check all AI crawlers at once?
Reading the file by hand gets slow once you care about more than one crawler. Each AI company runs several, with different jobs:
- Training crawlers collect content for model training:
GPTBot,ClaudeBot,CCBot,Bytespider,meta-externalagent, plus theGoogle-ExtendedandApplebot-Extendedtokens, which only appear in robots.txt. - Search crawlers index pages so AI answers can cite them:
OAI-SearchBot,Claude-SearchBot,PerplexityBot. - User crawlers fetch a page when a person asks the assistant to:
ChatGPT-User,Claude-User,Perplexity-User.
The free AI Crawler Checker extension for Chrome does this in one click. Open any page, click the icon, and it reads that site's robots.txt and shows, for each of 16 crawlers, whether the page is allowed and which line decides it. It also shows Content-Signal preferences, whether the site has an llms.txt, and the sitemaps listed in robots.txt. It runs in your browser, needs no account, and reads only the site you click it on.
Without Chrome, the free robots.txt checker on aiseewhy.com gives the same answer for a URL you paste.
How do I check whether the firewall lets them through?
robots.txt shows what the site asks for. To see what a crawler actually gets, request the page twice, once as a browser and once with the crawler's user agent, and compare the status code and the size of the response:
# As a browser
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
https://example.com/page
# As OAI-SearchBot
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
https://example.com/page If the browser gets 200 and the crawler gets 403, 429 or 503, or a much smaller page, a firewall or bot protection treats the crawler differently. One limit applies to every outside test: it comes from your network, not from the AI company's. A firewall that trusts crawlers by their published IP addresses can treat the real crawler differently, so your server logs are the final word.
The full check on aiseewhy.com fetches the page as all 16 crawlers next to a real browser, names the firewall or rule that blocks a crawler, and compares the text each one received.
What result should I aim for?
It depends on what you want. If you want to be cited in ChatGPT search, Claude and Perplexity, the search and user crawlers must be allowed in robots.txt and must get the real page from the server. If you do not want your content used for training, block only the training tokens. These are separate choices: blocking GPTBot does not block OAI-SearchBot, and blocking ClaudeBot does not block Claude-SearchBot.
Many sites mix them up. In our November 2026 report on the 1,000 most popular domains, 63% of the sites that block a training crawler also block at least one AI search or user crawler, which keeps their pages out of those AI answers.
Next steps
- robots.txt tester and checker for AI crawlers
- llms.txt checker and validator
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Sources
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: Common crawlers, including Google-Extended
- AISeeWhy: AI Crawler Access Report, 2026-11