Roboti za AI hupata nini hasa kutoka tovuti yako? Skripti ya mistari 66
Omba ukurasa wako kama kivinjari na kama kila roboti ya AI, kisha linganisha. Ukipata 403, 429 au ukurasa wa changamoto kwa roboti pekee, au 200 iliyo ndogo zaidi kuliko ukurasa wa kivinjari, ngome ya usalama inaikataa roboti hiyo hata kama robots.txt inairuhusu. Skripti ya Node ya mistari 66 hufanya hivyo kwa roboti saba na husoma robots.txt kwa kanuni za RFC 9309.
robots.txt huiambia roboti ya kutambaa kile inachoruhusiwa kuchukua. Haisemi lolote kuhusu kile ngome yako ya usalama au CDN hurudisha kwa kweli. Chapisho hili linakupa skripti fupi inayoonyesha vyote viwili kando kwa kando, ili uone mahali vinapotofautiana.
Kwenye ukurasa huu
Kwa nini robots.txt haikuambii roboti inapata nini?
robots.txt ni ombi tu. Ngome ya usalama, CDN au kanuni ya ulinzi dhidi ya roboti ndiyo huamua kinachotumwa kweli, na inaweza kujibu OAI-SearchBot au PerplexityBot kwa 403, 429 au ukurasa wa "Just a moment..." huku ziara zako za kivinjari zikionekana kawaida kabisa.
Njia ya kukagua ni kuomba URL ileile kama kivinjari na kama roboti, kisha kulinganisha. 403, 429 au 503 kwa roboti pekee, ukurasa wenye kichwa Just a moment..., au jibu la 200 lililo dogo zaidi kuliko la kivinjari, vyote vinaelekeza kwenye ngome ya usalama.
Skripti
Ihifadhi kama ai-crawler-view.mjs. Inahitaji Node 18 au mpya zaidi na haihitaji vifurushi vyovyote. Husoma robots.txt mara moja, kisha huchukua ukurasa wako kama kivinjari, halafu kama roboti saba za AI: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot na Perplexity-User.
#!/usr/bin/env node
// ai-crawler-view: what do AI crawlers get from YOUR site, compared with a browser?
// Usage: node ai-crawler-view.mjs https://your-site.com/page [--expect "a sentence from your page"]
// Node 18+, no dependencies. Run it against sites you own: it sends requests with crawler user agents.
const BROWSER = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";
const BOTS = [ // [robots.txt token, user agent]; Anthropic prints no full strings, so Claude gets a token-based one
["OAI-SearchBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"],
["ChatGPT-User", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"],
["GPTBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"],
["Claude-SearchBot", "Mozilla/5.0 (compatible; Claude-SearchBot/1.0)"],
["ClaudeBot", "Mozilla/5.0 (compatible; ClaudeBot/1.0)"],
["PerplexityBot", "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"],
["Perplexity-User", "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"],
];
const BLOCK = new Set([401, 403, 406, 429, 451, 503, 999]);
const CHALLENGE = /Just a moment\.\.\.|Attention Required! \| Cloudflare|\/cdn-cgi\/challenge-platform|_Incapsula_Resource|Access Denied/i;
const [url, ...rest] = process.argv.slice(2);
const expect = rest.includes("--expect") ? rest[rest.indexOf("--expect") + 1] : "";
if (!url) { console.error("usage: node ai-crawler-view.mjs https://site/page [--expect \"text\"]"); process.exit(2); }
async function get(u, ua) {
try {
const r = await fetch(u, { headers: { "user-agent": ua }, redirect: "follow", signal: AbortSignal.timeout(15000) });
return { status: r.status, body: (await r.text()).slice(0, 500000) };
} catch (e) { return { status: 0, body: "", error: e.cause?.code || e.name }; }
}
// robots.txt, RFC 9309 in short: the group naming the token wins, else "*"; longest match wins; Allow wins a tie.
function robotsVerdict(txt, token, path) {
const groups = []; let cur = null;
for (const raw of txt.split(/\r?\n/)) {
const m = raw.replace(/#.*/, "").match(/^\s*([A-Za-z-]+)\s*:\s*(.*?)\s*$/); if (!m) continue;
const [k, v] = [m[1].toLowerCase(), m[2]];
if (k === "user-agent") { if (!cur || cur.rules.length) groups.push(cur = { agents: [], rules: [] }); cur.agents.push(v.toLowerCase()); }
else if (cur && (k === "allow" || k === "disallow") && v) cur.rules.push([k, v]);
}
const g = groups.find(x => x.agents.includes(token.toLowerCase())) || groups.find(x => x.agents.includes("*"));
if (!g) return "no rule";
let best = null;
for (const [k, p] of g.rules) {
const re = new RegExp("^" + p.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\\\$$/, "$"));
if (re.test(path) && (!best || p.length > best[1].length || (p.length === best[1].length && k === "allow"))) best = [k, p];
}
return best ? `${best[0]}: ${best[1]}` : "no rule";
}
const target = new URL(url);
const robots = await get(new URL("/robots.txt", target).href, BROWSER);
const robotsOk = robots.status >= 200 && robots.status < 300 && !/<html/i.test(robots.body);
const control = await get(url, BROWSER);
const cLen = control.body.length;
console.log(`control (browser): HTTP ${control.status}, ${cLen} bytes${expect ? `, expected text ${control.body.includes(expect) ? "found" : "MISSING"}` : ""}`);
if (control.status === 0 || BLOCK.has(control.status)) console.log(" the browser request itself failed or was refused: results below are not meaningful");
console.log(`robots.txt: ${robots.status >= 500 ? "server error (RFC 9309: crawlers must assume everything is disallowed)" : robotsOk ? "read" : "none or not valid"}\n`);
console.log("crawler robots.txt HTTP bytes verdict");
for (const [token, ua] of BOTS) {
const r = await get(url, ua);
const rule = robotsOk ? robotsVerdict(robots.body, token, target.pathname + target.search) : "no rule";
const challenge = CHALLENGE.test(r.body.slice(0, 20000)) && r.body.length < 20000;
const verdict = BLOCK.has(r.status) || challenge ? "BLOCKED by the server"
: r.status >= 400 || r.status === 0 ? `error ${r.status || r.error}`
: r.body.length < cLen * 0.5 ? "THIN: much smaller than the browser's page"
: expect && !r.body.includes(expect) ? "text MISSING from the HTML" : "ok";
console.log(`${token.padEnd(17)} ${rule.padEnd(19)} ${String(r.status).padEnd(5)} ${String(r.body.length).padEnd(7)} ${verdict}${rule.startsWith("disallow") ? " (robots.txt asks it to stay away)" : ""}`);
} Anthropic haichapishi mfuatano kamili wa user agent, kwa hiyo mistari miwili ya Claude hutumia tokeni ya roboti pekee. Ngome inayolinganisha tokeni hiyo hujibu vivyo hivyo.
Ninaiendeshaje?
Iendeshe dhidi ya ukurasa wa tovuti yako mwenyewe. Chaguo la hiari --expect hukagua kama sentensi kutoka maudhui yako makuu ipo kwenye HTML ghafi anayopokea kila roboti. Ikikosekana, hati huongeza maandishi baada ya ukurasa kupakia, na roboti nyingi hazitayaona.
node ai-crawler-view.mjs https://your-site.com/pricing/ --expect "a sentence from your page" Iendeshe tu kwenye tovuti unazomiliki au unaruhusiwa kuzipima. Hutuma maombi tisa: robots.txt, kivinjari na moja kwa kila roboti.
Matokeo yanaonekanaje?
Haya ni matokeo dhidi ya seva ya majaribio ya ndani yenye uadui wa makusudi, inayoipa OAI-SearchBot changamoto, kuweka kikomo kwa ClaudeBot, kuipa PerplexityBot ganda tupu, na kukataza GPTBot kwenye robots.txt:
control (browser): HTTP 200, 2618 bytes, expected text found
robots.txt: read
crawler robots.txt HTTP bytes verdict
OAI-SearchBot allow: / 403 31 BLOCKED by the server
ChatGPT-User allow: / 200 2618 ok
GPTBot disallow: / 200 2618 ok (robots.txt asks it to stay away)
Claude-SearchBot allow: / 200 2618 ok
ClaudeBot allow: / 429 9 BLOCKED by the server
PerplexityBot allow: / 200 36 THIN: much smaller than the browser's page
Perplexity-User allow: / 200 2618 ok Kwenye aiseewhy.com kila roboti hupata ukurasa kamili, wenye hali na ukubwa uleule kama kivinjari.
Ninasomaje jedwali?
- Safu ya robots.txt ni ruhusa uliyotoa: kanuni inayotumika kwa roboti hiyo chini ya RFC 9309, yaani kundi linalotaja roboti hiyo, vinginevyo
*, kisha kanuni ndefu zaidi inayolingana, ambapoAllowhushinda sare. - Safu za HTTP na bytes ni kile seva ilichofanya.
GPTBotinaonyesha 200 kwa sababu seva hujibu yeyote anayepuuza robots.txt; kanuni iliyo kando yake ndiyo roboti inayofuata taratibu huitii. - Safu hatari ni kinyume chake:
allow: /kando yaBLOCKEDauTHIN. Hiyo ni tovuti inayosema ndiyo lakini inajibu hapana.
Skripti haiwezi kukuambia nini?
- Inapima user agent kutoka anwani yako ya IP, si roboti halisi. Ngome nyingi huthibitisha roboti kwa safu za IP ambazo muuzaji wake huchapisha, kwa hiyo
OAI-SearchBothalisi inaweza kuruhusiwa wakati ombi lako linakataliwa, au kinyume chake. Itumie kupata kanuni zinazoitikia user agent, kisha kagua orodha za IP zilizochapishwa na wauzaji. - Haiendeshi JavaScript, kwa makusudi: inaonyesha HTML ambayo roboti isiyoonyesha kurasa ingeiona.
- Matokeo safi si pendekezo. Kusomeka si sawa na kunukuliwa.
Tunaona nini kwenye tovuti halisi?
AISeeWhy hufanya ukaguzi wa aina hii kwa roboti 14. Kati ya tovuti 165 za biashara nchini Marekani, Uingereza na Ufaransa tulizoweza kuzipima, 38% (tovuti 63) ziligeuza angalau roboti moja ya AI kwenye seva wakati kivinjari kutoka anwani ileile kilipata ukurasa. Kila tovuti hizo ilikataa angalau roboti moja ya mafunzo au ya kuvuna data, na 10% (tovuti 16) pia zilikataa angalau roboti moja ya utafutaji wa AI, kama PerplexityBot (tovuti 11) au OAI-SearchBot (5).
Tovuti hizi si sampuli ya nasibu, na nyingine 53 hazikuweza kupimwa kwa sababu hata ombi la kawaida la kivinjari lilikataliwa au halikujibiwa. Chukulia hili kama mwelekeo unaostahili kukaguliwa kwenye tovuti yako, si sensa.
Hatua zinazofuata
- Kikagua na kijaribu cha robots.txt kwa crawler za AI
- Ukaguzi wa IP ya crawler za AI
- Tazama mfano wa ripoti kamili
- Ufuatiliaji wa crawler za AI
- Mipango na bei
Crawler katika makala hii
Maswali yanayoulizwa mara kwa mara
Je, ni salama kuiendesha dhidi ya tovuti yoyote?
Iendeshe tu dhidi ya tovuti unazomiliki au unaruhusiwa kuzipima. Hutuma maombi yenye user agent za roboti, ambazo baadhi ya ulinzi dhidi ya roboti huzichukulia kuwa za uadui.
Kwa nini GPTBot inaonyesha ok wakati robots.txt inaikataza?
robots.txt ni ombi, si kufuli. Seva hujibu yeyote anayeipuuza, kwa hiyo safu ya HTTP inaonyesha ok. Kanuni iliyo kando yake inaonyesha kile roboti inayofuata taratibu huitii.
Je, skripti inaendesha JavaScript?
Hapana, kwa makusudi. Inaonyesha HTML anayopokea roboti isiyoonyesha kurasa. Chaguo la --expect hukuambia kama maandishi yako makuu tayari yapo kwenye HTML hiyo.
Je, matokeo safi yanamaanisha ChatGPT itanukuu ukurasa wangu?
Hapana. Ufikiaji ni muhimu lakini hautoshi. Ili kuona kama msaidizi anataja tovuti yako, mwulize maswali ambayo wateja wako huuliza na uandike tovuti anazonukuu.
Vyanzo
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Cloudflare: Manage AI crawlers (AI Crawl Control)