What do AI crawlers actually get from your site? A 66-line script

Published

Short answer

Request your page as a browser and as each AI crawler, then compare. A 403, a 429 or a challenge page for the crawler only, or a 200 that is much smaller than the browser's page, means the firewall refuses it even when robots.txt allows it. A 66-line Node script does this for seven crawlers and reads robots.txt by RFC 9309 rules.

robots.txt tells a crawler what it may fetch. It says nothing about what your firewall or CDN actually sends back. This post gives you a short script that shows both side by side, so you can see where the two disagree.

On this page
  1. Why does robots.txt not tell you what a crawler gets?
  2. The script
  3. How do I run it?
  4. What does the output look like?
  5. How do I read the table?
  6. What can the script not tell you?
  7. What do we see on real websites?

Why does robots.txt not tell you what a crawler gets?

robots.txt is a request. A firewall, CDN or bot-protection rule decides what is really sent back, and it can answer OAI-SearchBot or PerplexityBot with a 403, a 429 or a "Just a moment..." page while your own browser visits look perfectly normal.

The check is to request the same URL as a browser and as the crawler, and compare. A 403, 429 or 503 for the crawler only, a page titled Just a moment..., or a 200 response that is much smaller than the browser's all point to the firewall.

The script

Save it as ai-crawler-view.mjs. It needs Node 18 or newer and no packages. It reads robots.txt once, fetches your page as a browser, then as seven AI crawlers: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot and Perplexity-User.

#!/usr/bin/env node
// ai-crawler-view: what do AI crawlers get from YOUR site, compared with a browser?
// Usage: node ai-crawler-view.mjs https://your-site.com/page [--expect "a sentence from your page"]
// Node 18+, no dependencies. Run it against sites you own: it sends requests with crawler user agents.
const BROWSER = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";
const BOTS = [ // [robots.txt token, user agent]; Anthropic prints no full strings, so Claude gets a token-based one
  ["OAI-SearchBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"],
  ["ChatGPT-User", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"],
  ["GPTBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"],
  ["Claude-SearchBot", "Mozilla/5.0 (compatible; Claude-SearchBot/1.0)"],
  ["ClaudeBot", "Mozilla/5.0 (compatible; ClaudeBot/1.0)"],
  ["PerplexityBot", "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"],
  ["Perplexity-User", "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"],
];
const BLOCK = new Set([401, 403, 406, 429, 451, 503, 999]);
const CHALLENGE = /Just a moment\.\.\.|Attention Required! \| Cloudflare|\/cdn-cgi\/challenge-platform|_Incapsula_Resource|Access Denied/i;

const [url, ...rest] = process.argv.slice(2);
const expect = rest.includes("--expect") ? rest[rest.indexOf("--expect") + 1] : "";
if (!url) { console.error("usage: node ai-crawler-view.mjs https://site/page [--expect \"text\"]"); process.exit(2); }

async function get(u, ua) {
  try {
    const r = await fetch(u, { headers: { "user-agent": ua }, redirect: "follow", signal: AbortSignal.timeout(15000) });
    return { status: r.status, body: (await r.text()).slice(0, 500000) };
  } catch (e) { return { status: 0, body: "", error: e.cause?.code || e.name }; }
}

// robots.txt, RFC 9309 in short: the group naming the token wins, else "*"; longest match wins; Allow wins a tie.
function robotsVerdict(txt, token, path) {
  const groups = []; let cur = null;
  for (const raw of txt.split(/\r?\n/)) {
    const m = raw.replace(/#.*/, "").match(/^\s*([A-Za-z-]+)\s*:\s*(.*?)\s*$/); if (!m) continue;
    const [k, v] = [m[1].toLowerCase(), m[2]];
    if (k === "user-agent") { if (!cur || cur.rules.length) groups.push(cur = { agents: [], rules: [] }); cur.agents.push(v.toLowerCase()); }
    else if (cur && (k === "allow" || k === "disallow") && v) cur.rules.push([k, v]);
  }
  const g = groups.find(x => x.agents.includes(token.toLowerCase())) || groups.find(x => x.agents.includes("*"));
  if (!g) return "no rule";
  let best = null;
  for (const [k, p] of g.rules) {
    const re = new RegExp("^" + p.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\\\$$/, "$"));
    if (re.test(path) && (!best || p.length > best[1].length || (p.length === best[1].length && k === "allow"))) best = [k, p];
  }
  return best ? `${best[0]}: ${best[1]}` : "no rule";
}

const target = new URL(url);
const robots = await get(new URL("/robots.txt", target).href, BROWSER);
const robotsOk = robots.status >= 200 && robots.status < 300 && !/<html/i.test(robots.body);
const control = await get(url, BROWSER);
const cLen = control.body.length;
console.log(`control (browser): HTTP ${control.status}, ${cLen} bytes${expect ? `, expected text ${control.body.includes(expect) ? "found" : "MISSING"}` : ""}`);
if (control.status === 0 || BLOCK.has(control.status)) console.log("  the browser request itself failed or was refused: results below are not meaningful");
console.log(`robots.txt: ${robots.status >= 500 ? "server error (RFC 9309: crawlers must assume everything is disallowed)" : robotsOk ? "read" : "none or not valid"}\n`);
console.log("crawler           robots.txt          HTTP  bytes   verdict");
for (const [token, ua] of BOTS) {
  const r = await get(url, ua);
  const rule = robotsOk ? robotsVerdict(robots.body, token, target.pathname + target.search) : "no rule";
  const challenge = CHALLENGE.test(r.body.slice(0, 20000)) && r.body.length < 20000;
  const verdict = BLOCK.has(r.status) || challenge ? "BLOCKED by the server"
    : r.status >= 400 || r.status === 0 ? `error ${r.status || r.error}`
    : r.body.length < cLen * 0.5 ? "THIN: much smaller than the browser's page"
    : expect && !r.body.includes(expect) ? "text MISSING from the HTML" : "ok";
  console.log(`${token.padEnd(17)} ${rule.padEnd(19)} ${String(r.status).padEnd(5)} ${String(r.body.length).padEnd(7)} ${verdict}${rule.startsWith("disallow") ? "  (robots.txt asks it to stay away)" : ""}`);
}

Anthropic does not print full user agent strings, so the two Claude lines use the crawler token only. A firewall that matches on the token reacts the same way.

How do I run it?

Run it against a page of your own site. The optional --expect flag checks that a sentence from your main content is present in the raw HTML each crawler receives. If it is missing, scripts add the text after load, and many crawlers will not see it.

node ai-crawler-view.mjs https://your-site.com/pricing/ --expect "a sentence from your page"

Run it only on sites you own or are allowed to test. It sends nine requests: robots.txt, the browser and one per crawler.

What does the output look like?

This is the result against a deliberately hostile local test server that challenges OAI-SearchBot, rate-limits ClaudeBot, serves PerplexityBot an empty shell and disallows GPTBot in robots.txt:

control (browser): HTTP 200, 2618 bytes, expected text found
robots.txt: read

crawler           robots.txt          HTTP  bytes   verdict
OAI-SearchBot     allow: /            403   31      BLOCKED by the server
ChatGPT-User      allow: /            200   2618    ok
GPTBot            disallow: /         200   2618    ok  (robots.txt asks it to stay away)
Claude-SearchBot  allow: /            200   2618    ok
ClaudeBot         allow: /            429   9       BLOCKED by the server
PerplexityBot     allow: /            200   36      THIN: much smaller than the browser's page
Perplexity-User   allow: /            200   2618    ok

On aiseewhy.com every crawler gets the full page, with the same status and size as the browser.

How do I read the table?

What can the script not tell you?

What do we see on real websites?

AISeeWhy runs this kind of check for 14 crawlers. Across 165 business websites in the US, UK and France that we could test, 38% (63 sites) turned away at least one AI crawler at the server while a browser from the same address got the page. Every one of those sites refused at least one training or scraper bot, and 10% (16 sites) also refused at least one AI search crawler, such as PerplexityBot (11 sites) or OAI-SearchBot (5).

The sites are not a random sample, and 53 more could not be tested because even a plain browser request was refused or unanswered. Treat it as a pattern worth checking on your own site, not as a census.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Is it safe to run this against any site?

Run it only against sites you own or are allowed to test. It sends requests with crawler user agents, which some bot defenses treat as hostile.

Why does GPTBot show ok when robots.txt disallows it?

robots.txt is a request, not a lock. The server answers anyone who ignores it, so the HTTP column says ok. The rule next to it shows what a well-behaved crawler obeys.

Does the script run JavaScript?

No, on purpose. It shows the HTML that a crawler that does not render pages receives. The --expect flag tells you whether your main text is already in that HTML.

Does a clean result mean ChatGPT will cite my page?

No. Access is necessary but not sufficient. To see whether an assistant names your site, ask it the questions your buyers ask and note which sites it cites.

Sources

More articles