วิธีบล็อกการฝึกโมเดลของ AI แต่ยังอยู่ในคำตอบของการค้นหาด้วย AI

Q: Will blocking GPTBot remove my site from ChatGPT answers?

No. OpenAI says GPTBot is for training and that each robots.txt setting is independent. ChatGPT search uses OAI-SearchBot; as long as that is allowed and not blocked by your firewall, your pages remain eligible.

Q: Does blocking Google-Extended remove me from AI Overviews?

Not according to Google's documentation. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews draw on the Search index. Google offers a separate Search Console control to exclude a site from AI Overviews and AI Mode.

Q: Is there a single line that blocks all AI training?

No standard line does. Content-Signal: ai-train=no states the preference for all crawlers, but whether a crawler honors it depends on the operator. Named groups for each training token are the documented way.

เผยแพร่เมื่อ 2026-11-12

คำตอบสั้น ๆ

ผู้ให้บริการ AI ส่วนใหญ่ใช้โทเค็น robots.txt แยกกันสำหรับการฝึกโมเดลและการค้นหา ให้ห้ามโทเค็นการฝึกโมเดล เช่น GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent และ CCBot และอนุญาตโทเค็นสำหรับการค้นหาและผู้ใช้ เช่น OAI-SearchBot, Claude-SearchBot และ PerplexityBot เพิ่มบรรทัด Content-Signal เพื่อแสดงความต้องการเดียวกันเป็นถ้อยคำ

เจ้าของเว็บไซต์หลายคนต้องการสองอย่างพร้อมกัน คือไม่ต้องการให้นำเนื้อหาไปฝึกโมเดล AI แต่ต้องการให้ผู้ช่วย AI ค้นหา อ้างอิง และลิงก์ไปยังหน้าเว็บของตน เมื่อผู้ให้บริการรายใหญ่แยกหน้าที่เหล่านี้ไปไว้ในโปรแกรมรวบรวมข้อมูลหรือโทเค็นคนละตัว robots.txt จึงแสดงได้ทั้งสองทางเลือก ข้อควรระวังคือชื่อต่างกันไปตามผู้ให้บริการ บางรายไม่เข้ากับรูปแบบนี้ และบรรทัดที่ผิดเพียงบรรทัดเดียวอาจทำให้คุณหายไปจากคำตอบของ AI แทนที่จะหายไปจากข้อมูลฝึกโมเดล

ในหน้านี้

What is the difference between a training crawler and a search crawler?
Which tokens control training and which control search?
What does the robots.txt look like?
How many sites already do this?
What does Content-Signal add?
Where does the split not work cleanly?
How do I check that the split works?

What is the difference between a training crawler and a search crawler?

A training crawler collects pages that may be used to build or improve AI models. Training by itself does not produce a citation or a link. A search crawler builds an index that an AI assistant uses to find sources for an answer, and those sources are what it can cite. A user-triggered fetcher opens a page live because someone asked about it.

The vendors document these roles separately and say the settings are independent. OpenAI, for example, says each robots.txt setting for its crawlers is independent of the others. So a training opt-out, done right, leaves your search visibility alone.

Which tokens control training and which control search?

These are the tokens the vendors document, grouped by what they control:

OpenAI: GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot is for ChatGPT search, and ChatGPT-User handles user actions; OpenAI says robots.txt may not apply to it.
Anthropic: ClaudeBot collects content that could contribute to training. Claude-SearchBot improves Claude's search results, and Claude-User fetches pages for users. Anthropic says its bots honor robots.txt.
Google: Google-Extended is a robots.txt token with no crawler of its own. It controls use in training future Gemini models and in grounding for Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search, which is controlled through Googlebot.
Apple: Applebot-Extended does not crawl; it controls whether content crawled by Applebot is used to train Apple's foundation models. Apple says pages that disallow it can still appear in its search results.
Meta: meta-externalagent crawls for uses such as training foundation models, while meta-webindexer improves Meta AI search results.
Common Crawl: CCBot builds Common Crawl's public web archive. Cloudflare's managed robots.txt lists it among the AI training crawlers it disallows.
Perplexity: PerplexityBot and Perplexity-User serve search and user requests. Perplexity says neither is used to collect training data, so there is no Perplexity training token to block.

What does the robots.txt look like?

Several User-agent lines in a row form one group under RFC 9309, so the rules apply to each of them. This file opts out of the training tokens above, allows the search and user agents, and keeps one private path closed for everyone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
Allow: /
Disallow: /account/

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Two details matter. First, a named group does not inherit from *, so any private paths have to be repeated in the search group. Second, do not add Googlebot or Applebot to the training group: they are the search crawlers, and blocking them removes you from Google Search and Apple's search features. Changes are not instant: OpenAI says about 24 hours for search, and Perplexity says up to 24 hours.

How many sites already do this?

Fewer than you might expect. We read the robots.txt files of the 1,000 most popular domains on the Tranco list on 2026-11-12. Of the 518 that served a robots.txt, 155 (29.9%) disallow the home page for at least one AI training crawler. Only 57 (11.0%) take the selective route: they block training crawlers and allow every AI search and user crawler we tested. In all, 101 sites (19.5%) block at least one AI search or user crawler.

The crawler-by-crawler numbers show the same split. GPTBot is blocked on 111 sites (21.4%) and OAI-SearchBot on 58 (11.2%). ClaudeBot is blocked on 110 (21.2%) and Claude-SearchBot on 67 (12.9%). Google-Extended is blocked on 94 (18.1%), Googlebot on 14 (2.7%). Only 18 sites (3.5%) use Content-Signal.

The study looks at robots.txt rules for the home page only. It does not measure firewall blocks, and it does not show what each site intended.

What does Content-Signal add?

Content Signals is a robots.txt extension published by Cloudflare at contentsignals.org. A Content-Signal line states how content may be used, with three signals: search (building a search index and showing links and short excerpts), ai-input (putting content into AI models in real time, such as grounding or retrieval for generative answers) and ai-train (training or fine-tuning models).

Note that the definition of search excludes AI-generated search summaries. If you want to appear in AI answers, set ai-input=yes as well, as in the example above. Cloudflare's managed robots.txt adds search=yes, ai-train=no by default; a signal you leave out neither grants nor restricts that use.

These are stated preferences. The Content Signals site says they do not prevent crawling at a technical level, and Cloudflare notes that robots.txt compliance is voluntary. None of the OpenAI, Anthropic, Google or Perplexity crawler pages we checked say whether they read Content-Signal, so treat it as an addition to the per-crawler groups, not a replacement.

Where does the split not work cleanly?

A few cases need care:

Google AI Overviews and AI Mode draw on the Search index that Googlebot builds, so Google-Extended does not take you out of them. Google's Search Console help adds that Google-Extended limits training of the models behind Search's generative AI features, a scope its crawler documentation does not mention. Google now offers a separate Search Console setting to exclude a site from AI Overviews and AI Mode.
OpenAI says that if a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both purposes. Disallowing GPTBot is what signals that content should not be used for training.
Opt-outs look forward. Anthropic, for example, describes disallowing ClaudeBot as excluding your site's future content from training datasets.
User-triggered fetchers sit in between. OpenAI and Perplexity say robots.txt may not apply to ChatGPT-User and generally does not apply to Perplexity-User.
Firewalls do not read robots.txt. A broad bot-blocking rule can stop the search crawlers you allowed. If you enforce the training block at the firewall, match the training agents by name and verify them by their published IP ranges.

How do I check that the split works?

Read your robots.txt as each crawler would: find the group that names it, fall back to * only if none does, and apply the longest matching rule. Then request a page with each search crawler's user agent and confirm it gets a 200 with the full content, not a 403 or a challenge page.

AISeeWhy does both for 14 crawlers, grouped as training, search and user agents, and flags the most common mistake: blocking a search token such as PerplexityBot or Claude-SearchBot while meaning to block training.

คำถามที่พบบ่อย

Will blocking GPTBot remove my site from ChatGPT answers?

No. OpenAI says GPTBot is for training and that each robots.txt setting is independent. ChatGPT search uses OAI-SearchBot; as long as that is allowed and not blocked by your firewall, your pages remain eligible.

Does blocking Google-Extended remove me from AI Overviews?

Not according to Google's documentation. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews draw on the Search index. Google offers a separate Search Console control to exclude a site from AI Overviews and AI Mode.

Is there a single line that blocks all AI training?

No standard line does. Content-Signal: ai-train=no states the preference for all crawlers, but whether a crawler honors it depends on the operator. Named groups for each training token are the documented way.

Should I block Perplexity to stop AI training?

Perplexity says its agents are not used to collect training data. Blocking PerplexityBot reduces your visibility in Perplexity's search results without serving a training opt-out.