What is Content-Signal in robots.txt, and does anyone follow it?

Published

Short answer

Content-Signal is an optional robots.txt line, published by Cloudflare, that states how your content may be used after it is fetched: search for search results, ai-input for feeding it into AI answers and ai-train for training models, each set to yes or no, as in Content-Signal: search=yes, ai-input=yes, ai-train=no. It expresses a preference and blocks nothing. As of November 2026, none of the OpenAI, Anthropic, Google or Perplexity crawler pages say they read it, while the large AI companies have backed a different, training-only token instead.

robots.txt was built to say where a crawler may go, not what may be done with what it reads. Once the same fetch could feed a search index, a live AI answer and a training set, that gap mattered. Cloudflare published the Content Signals Policy on 24 September 2025 under a CC0 licence to fill it, and said at the time that its customers had already turned on managed robots.txt for over 3.8 million domains, a feature that now adds the line for them.

On this page
  1. What do search, ai-input and ai-train mean?
  2. How do I write a Content-Signal line in robots.txt?
  3. What does the policy text above the line say?
  4. Is Content-Signal already in my robots.txt?
  5. Do AI crawlers follow Content-Signal?
  6. Should I add Content-Signal to my robots.txt?
  7. How can I check what my robots.txt tells each crawler?

What do search, ai-input and ai-train mean?

The policy defines three signals. The wording below follows contentsignals.org and the policy text Cloudflare publishes:

Each signal takes yes or no. In the policy's terms, yes means content may be collected for that use, no means it may not, and a signal you leave out neither grants nor restricts that use. Cloudflare's default sets search and ai-train and leaves ai-input unset, because it says it does not know each customer's preference for that signal and does not want to guess.

The split between search and ai-input is the one most people miss. If you want your pages used in AI answers, search=yes alone does not say so. ai-input=yes does.

How do I write a Content-Signal line in robots.txt?

Content-Signal goes inside a user agent group, next to Allow and Disallow, with the signals separated by commas. contentsignals.org also shows a form that starts with a path, so one section of a site can carry different signals from the rest:

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# Search only for the blog
User-Agent: *
Content-Signal: /blog/ search=yes, ai-input=no, ai-train=no
Allow: /blog/

Allow and Disallow keep working exactly as before, and Content-Signal does not replace them. RFC 9309 lets crawlers interpret records that are not part of the protocol, as long as that does not interfere with the defined rules, so a parser that does not know the line should skip it.

Two details are worth knowing. First, the line belongs to a group. Under RFC 9309 a crawler that has a group with its own name ignores the User-agent: * group entirely, and the Content Signals pages do not describe any exception for this line. If you give Googlebot or OAI-SearchBot a group of their own, repeat the Content-Signal line there, as contentsignals.org does in its per-crawler example. Second, Google Search Console may report Syntax not understood for the line. Cloudflare's documentation mentions this and says it has observed no impact on crawling or SEO. Google's robots.txt documentation lists only user-agent, allow, disallow and sitemap as supported fields.

What does the policy text above the line say?

The Content Signals Policy is more than one line. Cloudflare's generator on contentsignals.org and its managed robots.txt both put a block of comments above the signals. It opens by saying that, as a condition of accessing the website, you agree to abide by the content signals, then defines the three signals and the yes, no and missing cases.

It ends with a sentence in capitals stating that any restrictions expressed via content signals are express reservations of rights under Article 4 of EU Directive 2019/790 on copyright in the Digital Single Market. Cloudflare's announcement adds that the signals may carry legal weight in various jurisdictions.

contentsignals.org itself warns that courts and regulators may conclude robots.txt files do not impose enforceable legal obligations, and suggests consulting a lawyer. Treat the comment block as a written statement of your position, not as a contract.

Is Content-Signal already in my robots.txt?

If your site is on Cloudflare, it may be, in one of two ways:

The use=reference part is recent. Cloudflare describes use as an optional fourth field it is testing, with three values: immediate (interact, but store and reuse nothing), reference (index, excerpt and link back) and full (summarize and reproduce). It is not one of the three original signals, and none of the crawler documentation we checked mentions it.

Because Cloudflare adds these lines at the edge, the file in your repository can look unchanged while the file crawlers receive is different. Fetch the live https://yourdomain.com/robots.txt before adding your own line, so the response does not state two different policies.

Adoption is still low among large sites. In our November 2026 study of the 1,000 most popular domains, 18 of the 518 sites that served a robots.txt (3.5%) used Content-Signal.

Do AI crawlers follow Content-Signal?

Not according to any documentation we could find. On 17 November 2026 we checked the crawler pages of OpenAI, Anthropic, Google and Perplexity, and none of them mentions Content-Signal. Cloudflare itself describes the signals as preferences, not technical countermeasures against scraping, and says some companies might simply ignore them.

What the AI companies have documented instead is a separate mechanism based on work in the IETF AI Preferences working group: a policy token, __usage_ai-training, written as an ordinary User-Agent group. It covers training only, and the Internet-Draft that defines it says a User-Agent: * group does not count as using the token:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Support is uneven and partly still ahead. Anthropic says it has recognised a general machine-readable training signal since 30 October 2026, is rolling it out gradually, and that a rule naming ClaudeBot takes priority over it. Microsoft documents the __usage_ai-training token for Bing, with its new preferences taking effect as pages are crawled from 1 February 2027. Google says it aims to support the shared control in 2027 and plans to treat a training opt-out through it as a disallow of Google-Extended when that token is not named. OpenAI published an announcement the same day saying it will honor the shared signal, without a date.

So ai-train=no and __usage_ai-training say similar things, but only the second has documented support from AI companies. The per-crawler tokens, such as GPTBot and ClaudeBot, remain the controls each vendor applies today.

Should I add Content-Signal to my robots.txt?

Add it if you want your usage policy written down in one place that machines and people can read. It also states an ai-input preference for every crawler in a single line, which the vendor tokens can only express crawler by crawler. It costs one line, and parsers that do not know it skip it.

Do not rely on it alone. To keep content out of training, also disallow the documented training tokens, and consider the shared training token; our guide to blocking AI training while staying in AI search lists the tokens per vendor. For anything that ignores robots.txt, only a firewall rule enforces a block.

Finally, check that the line agrees with the rest of the file. ai-input=no next to an Allow group for OAI-SearchBot or PerplexityBot sends mixed messages: you let the AI search crawlers in while saying their answers may not use what they fetch.

How can I check what my robots.txt tells each crawler?

The free AISeeWhy robots.txt checker takes a page address or a pasted robots.txt and shows, for each AI and search crawler, whether that page may be fetched, so you can confirm that the live file, including anything Cloudflare prepended, says what you meant. The free robots.txt generator writes a file from your choices.

Free tools

Next steps

Crawlers in this article

Frequently asked questions

Is Content-Signal part of the robots.txt standard?

No. RFC 9309 defines User-agent, Allow and Disallow and lets crawlers interpret other records. Content-Signal comes from Cloudflare's Content Signals Policy, released under a CC0 licence. Separately, the IETF AI Preferences working group is drafting its own Content-Usage rule for robots.txt, which is still an Internet-Draft, not a standard.

Does ai-train=no stop GPTBot or ClaudeBot?

Not as far as the vendors document. OpenAI and Anthropic describe control through robots.txt rules that name GPTBot and ClaudeBot, and neither crawler page mentions Content-Signal. To opt out of training with either, disallow the token by name.

What is the difference between Content-Signal and __usage_ai-training?

Content-Signal is a line inside a group and covers search, AI input and training. __usage_ai-training is a policy token used as its own User-Agent group and covers training only. Anthropic, Microsoft, Google and OpenAI have published statements about the shared training signal; none of them documents Content-Signal.

Why does Google Search Console say Syntax not understood?

Google's robots.txt documentation supports only user-agent, allow, disallow and sitemap, so other lines can be flagged. Cloudflare says it has observed no impact on crawling rates or SEO from these reports.

What does use=reference mean?

It is a fourth field Cloudflare is testing. reference means index, excerpt and link back. Cloudflare adds it to managed robots.txt files, and no crawler documentation we checked mentions it.

Sources

More articles