What do AI crawlers actually get from your site? A 66-line script
Request your page as a browser and as each AI crawler, then compare. A 403, a 429 or a challenge page for the crawler only, or a 200 that is much smaller than the browser's page, means the firewall refuses it even when robots.txt allows it. A 66-line Node script does this for seven crawlers and reads robots.txt by RFC 9309 rules.
robots.txt tells a crawler what it may fetch. It says nothing about what your firewall or CDN actually sends back. This post gives you a short script that shows both side by side, so you can see where the two disagree.
On this page
Why does robots.txt not tell you what a crawler gets?
robots.txt is a request. A firewall, CDN or bot-protection rule decides what is really sent back, and it can answer OAI-SearchBot or PerplexityBot with a 403, a 429 or a "Just a moment..." page while your own browser visits look perfectly normal.
The check is to request the same URL as a browser and as the crawler, and compare. A 403, 429 or 503 for the crawler only, a page titled Just a moment..., or a 200 response that is much smaller than the browser's all point to the firewall.
The script
Save it as ai-crawler-view.mjs. It needs Node 18 or newer and no packages. It reads robots.txt once, fetches your page as a browser, then as seven AI crawlers: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot and Perplexity-User.
#!/usr/bin/env node
// ai-crawler-view: what do AI crawlers get from YOUR site, compared with a browser?
// Usage: node ai-crawler-view.mjs https://your-site.com/page [--expect "a sentence from your page"]
// Node 18+, no dependencies. Run it against sites you own: it sends requests with crawler user agents.
const BROWSER = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";
const BOTS = [ // [robots.txt token, user agent]; Anthropic prints no full strings, so Claude gets a token-based one
["OAI-SearchBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"],
["ChatGPT-User", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"],
["GPTBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"],
["Claude-SearchBot", "Mozilla/5.0 (compatible; Claude-SearchBot/1.0)"],
["ClaudeBot", "Mozilla/5.0 (compatible; ClaudeBot/1.0)"],
["PerplexityBot", "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"],
["Perplexity-User", "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"],
];
const BLOCK = new Set([401, 403, 406, 429, 451, 503, 999]);
const CHALLENGE = /Just a moment\.\.\.|Attention Required! \| Cloudflare|\/cdn-cgi\/challenge-platform|_Incapsula_Resource|Access Denied/i;
const [url, ...rest] = process.argv.slice(2);
const expect = rest.includes("--expect") ? rest[rest.indexOf("--expect") + 1] : "";
if (!url) { console.error("usage: node ai-crawler-view.mjs https://site/page [--expect \"text\"]"); process.exit(2); }
async function get(u, ua) {
try {
const r = await fetch(u, { headers: { "user-agent": ua }, redirect: "follow", signal: AbortSignal.timeout(15000) });
return { status: r.status, body: (await r.text()).slice(0, 500000) };
} catch (e) { return { status: 0, body: "", error: e.cause?.code || e.name }; }
}
// robots.txt, RFC 9309 in short: the group naming the token wins, else "*"; longest match wins; Allow wins a tie.
function robotsVerdict(txt, token, path) {
const groups = []; let cur = null;
for (const raw of txt.split(/\r?\n/)) {
const m = raw.replace(/#.*/, "").match(/^\s*([A-Za-z-]+)\s*:\s*(.*?)\s*$/); if (!m) continue;
const [k, v] = [m[1].toLowerCase(), m[2]];
if (k === "user-agent") { if (!cur || cur.rules.length) groups.push(cur = { agents: [], rules: [] }); cur.agents.push(v.toLowerCase()); }
else if (cur && (k === "allow" || k === "disallow") && v) cur.rules.push([k, v]);
}
const g = groups.find(x => x.agents.includes(token.toLowerCase())) || groups.find(x => x.agents.includes("*"));
if (!g) return "no rule";
let best = null;
for (const [k, p] of g.rules) {
const re = new RegExp("^" + p.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\\\$$/, "$"));
if (re.test(path) && (!best || p.length > best[1].length || (p.length === best[1].length && k === "allow"))) best = [k, p];
}
return best ? `${best[0]}: ${best[1]}` : "no rule";
}
const target = new URL(url);
const robots = await get(new URL("/robots.txt", target).href, BROWSER);
const robotsOk = robots.status >= 200 && robots.status < 300 && !/<html/i.test(robots.body);
const control = await get(url, BROWSER);
const cLen = control.body.length;
console.log(`control (browser): HTTP ${control.status}, ${cLen} bytes${expect ? `, expected text ${control.body.includes(expect) ? "found" : "MISSING"}` : ""}`);
if (control.status === 0 || BLOCK.has(control.status)) console.log(" the browser request itself failed or was refused: results below are not meaningful");
console.log(`robots.txt: ${robots.status >= 500 ? "server error (RFC 9309: crawlers must assume everything is disallowed)" : robotsOk ? "read" : "none or not valid"}\n`);
console.log("crawler robots.txt HTTP bytes verdict");
for (const [token, ua] of BOTS) {
const r = await get(url, ua);
const rule = robotsOk ? robotsVerdict(robots.body, token, target.pathname + target.search) : "no rule";
const challenge = CHALLENGE.test(r.body.slice(0, 20000)) && r.body.length < 20000;
const verdict = BLOCK.has(r.status) || challenge ? "BLOCKED by the server"
: r.status >= 400 || r.status === 0 ? `error ${r.status || r.error}`
: r.body.length < cLen * 0.5 ? "THIN: much smaller than the browser's page"
: expect && !r.body.includes(expect) ? "text MISSING from the HTML" : "ok";
console.log(`${token.padEnd(17)} ${rule.padEnd(19)} ${String(r.status).padEnd(5)} ${String(r.body.length).padEnd(7)} ${verdict}${rule.startsWith("disallow") ? " (robots.txt asks it to stay away)" : ""}`);
} Anthropic does not print full user agent strings, so the two Claude lines use the crawler token only. A firewall that matches on the token reacts the same way.
How do I run it?
Run it against a page of your own site. The optional --expect flag checks that a sentence from your main content is present in the raw HTML each crawler receives. If it is missing, scripts add the text after load, and many crawlers will not see it.
node ai-crawler-view.mjs https://your-site.com/pricing/ --expect "a sentence from your page" Run it only on sites you own or are allowed to test. It sends nine requests: robots.txt, the browser and one per crawler.
What does the output look like?
This is the result against a deliberately hostile local test server that challenges OAI-SearchBot, rate-limits ClaudeBot, serves PerplexityBot an empty shell and disallows GPTBot in robots.txt:
control (browser): HTTP 200, 2618 bytes, expected text found
robots.txt: read
crawler robots.txt HTTP bytes verdict
OAI-SearchBot allow: / 403 31 BLOCKED by the server
ChatGPT-User allow: / 200 2618 ok
GPTBot disallow: / 200 2618 ok (robots.txt asks it to stay away)
Claude-SearchBot allow: / 200 2618 ok
ClaudeBot allow: / 429 9 BLOCKED by the server
PerplexityBot allow: / 200 36 THIN: much smaller than the browser's page
Perplexity-User allow: / 200 2618 ok On aiseewhy.com every crawler gets the full page, with the same status and size as the browser.
How do I read the table?
- The robots.txt column is the permission you gave: the rule that applies to that crawler under RFC 9309, which is the group naming the crawler, else
*, then the longest matching rule, withAllowwinning a tie. - The HTTP and bytes columns are what the server did.
GPTBotshows a 200 because the server answers anyone who ignores robots.txt; the rule next to it is what a well-behaved crawler obeys. - The dangerous rows are the opposite:
allow: /next toBLOCKEDorTHIN. That is a site that says yes and answers no.
What can the script not tell you?
- It tests user agents from your own IP address, not the real crawlers. Many firewalls verify a crawler by the IP ranges its vendor publishes, so a real
OAI-SearchBotmay be let through when your request is refused, or the other way around. Use it to find rules that react to the user agent, then check the vendors' published IP lists. - It does not run JavaScript, on purpose: it shows the HTML a crawler that does not render would see.
- A clean result is not a recommendation. Being readable is not the same as being cited.
What do we see on real websites?
AISeeWhy runs this kind of check for 14 crawlers. Across 165 business websites in the US, UK and France that we could test, 38% (63 sites) turned away at least one AI crawler at the server while a browser from the same address got the page. Every one of those sites refused at least one training or scraper bot, and 10% (16 sites) also refused at least one AI search crawler, such as PerplexityBot (11 sites) or OAI-SearchBot (5).
The sites are not a random sample, and 53 more could not be tested because even a plain browser request was refused or unanswered. Treat it as a pattern worth checking on your own site, not as a census.
Next steps
- robots.txt tester and checker for AI crawlers
- AI crawler IP check
- See an example full report
- AI crawler monitoring
- Plans and prices
Crawlers in this article
Frequently asked questions
Is it safe to run this against any site?
Run it only against sites you own or are allowed to test. It sends requests with crawler user agents, which some bot defenses treat as hostile.
Why does GPTBot show ok when robots.txt disallows it?
robots.txt is a request, not a lock. The server answers anyone who ignores it, so the HTTP column says ok. The rule next to it shows what a well-behaved crawler obeys.
Does the script run JavaScript?
No, on purpose. It shows the HTML that a crawler that does not render pages receives. The --expect flag tells you whether your main text is already in that HTML.
Does a clean result mean ChatGPT will cite my page?
No. Access is necessary but not sufficient. To see whether an assistant names your site, ask it the questions your buyers ask and note which sites it cites.
Sources
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Cloudflare: Manage AI crawlers (AI Crawl Control)