robots.txt for AI crawlers: what to allow, what to block

Q: Does blocking GPTBot or ClaudeBot hurt my visibility in AI answers?

Not directly. Both vendors use separate search crawlers, OAI-SearchBot and Claude-SearchBot, for search. Blocking the training crawler is a policy choice.

Q: Do AI crawlers support Crawl-delay?

It is not part of RFC 9309. Anthropic says it supports Crawl-delay as a non-standard extension; for other vendors, check their documentation.

Q: Where must robots.txt be located?

At the root of each host, for example https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.

Published 2026-11-12

Short answer

Give each AI crawler you care about its own User-agent group. Allow search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to be cited, and decide separately about training tokens such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. A named group replaces the * group for that crawler.

robots.txt is a plain text file that dates back to 1994 and became a formal standard, RFC 9309, in 2022. Most AI vendors say their crawlers follow it, and several publish separate tokens for search and for training. That makes robots.txt the simplest place to state an AI policy, provided the rules mean what you think they mean.

On this page

Which robots.txt tokens do AI crawlers use?
What does a good AI robots.txt look like?
How do crawlers pick which rules apply to them?
What are the most common robots.txt mistakes?
Can I state an AI usage policy beyond allow and disallow?
How long do changes take, and how do I check them?

Which robots.txt tokens do AI crawlers use?

These are the tokens the vendors document. Matching is case-insensitive under RFC 9309.

OpenAI: OAI-SearchBot (ChatGPT search), GPTBot (training), ChatGPT-User (user-requested visits; OpenAI says robots.txt may not apply).
Anthropic: Claude-SearchBot (search), ClaudeBot (training), Claude-User (user-requested visits).
Perplexity: PerplexityBot (search), Perplexity-User (user-requested visits; Perplexity says it generally ignores robots.txt).
Google: Googlebot (Search, including its AI features) and Google-Extended, a token with no crawler of its own that controls use in Gemini training and grounding.
Apple: Applebot (Spotlight, Siri, Safari search) and Applebot-Extended, which does not crawl but controls use for training Apple's foundation models.
Meta: meta-externalagent (training and product indexing) and meta-webindexer (Meta AI search).
Common Crawl: CCBot, which builds Common Crawl's free public archive of web data.

What does a good AI robots.txt look like?

This example lets AI search and user fetches in, opts out of the main training tokens and keeps one private area closed for everyone:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /
Disallow: /account/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row form one group, so the rules below them apply to each of those crawlers. If you do want your content used for training, simply leave the second group out. Blocking Google-Extended does not affect Google Search: Google says it does not impact inclusion in Search and is not a ranking signal. Likewise, Apple says pages that disallow Applebot-Extended can still appear in its search results.

How do crawlers pick which rules apply to them?

RFC 9309 defines this precisely, and the details explain most surprises:

A crawler looks for the group whose User-agent matches its product token, ignoring case. If several groups match, their rules are combined into one.
Only if no group matches does the crawler follow the User-agent: * group. A named group does not inherit anything from *.
Within the group, the rule with the longest matching path wins. Allow: /blog/ beats Disallow: / for /blog/post.
If an Allow and a Disallow rule match with equal length, Allow should be used.
* in a path matches any characters and $ anchors the end, so Disallow: /*.pdf$ covers every PDF.

What are the most common robots.txt mistakes?

Expecting a named group to inherit from *. A group for OAI-SearchBot with only Allow: / opens every path, including ones you disallowed under *.
A robots.txt address that returns an HTML page, often a 404 page served with status 200. Crawlers find no valid rules in it.
A robots.txt that returns a server error. RFC 9309 says a crawler must then assume the whole site is disallowed. A 404 is different: it means there are no restrictions.
Blocking a search token while meaning to block training, for example disallowing PerplexityBot or Claude-SearchBot to avoid model training.
Relying on robots.txt to stop user-requested fetches. OpenAI and Perplexity both say their user agents may not follow it.
Forgetting subdomains. Each host has its own robots.txt; Anthropic's documentation reminds site owners to add rules for each subdomain.

Can I state an AI usage policy beyond allow and disallow?

Cloudflare has proposed a Content-Signal line for robots.txt with three signals: search (building a search index and showing links and short excerpts), ai-input (using content in real time for AI answers) and ai-train (training or fine-tuning models). Cloudflare's managed robots.txt can add it for you.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Like everything in robots.txt, these are preferences, not technical controls. Cloudflare itself notes that robots.txt compliance is voluntary. The line is optional and does no harm, but it does not replace the per-crawler groups above.

How long do changes take, and how do I check them?

RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, and OpenAI and Perplexity both say changes can take about a day to apply. Remember also that robots.txt is not access control: a firewall can still block a crawler that your rules allow.

AISeeWhy parses your robots.txt by RFC 9309 for each crawler it tests, flags files that are unreachable or served as HTML, shows where your rules and your server disagree and gives a ready diff for the change.

Frequently asked questions

Does blocking GPTBot or ClaudeBot hurt my visibility in AI answers?

Not directly. Both vendors use separate search crawlers, OAI-SearchBot and Claude-SearchBot, for search. Blocking the training crawler is a policy choice.

Does Google-Extended stop AI Overviews?

No. Google says AI features in Search use Googlebot and the usual Search controls such as nosnippet and noindex. Google-Extended covers Gemini training and grounding in other Google products and does not affect Search.

Do AI crawlers support Crawl-delay?

It is not part of RFC 9309. Anthropic says it supports Crawl-delay as a non-standard extension; for other vendors, check their documentation.

Where must robots.txt be located?

At the root of each host, for example https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.