What is llms.txt, and is it worth adding?

Published 2026-11-12

Short answer

llms.txt is a proposed Markdown file at /llms.txt that gives AI tools a short summary of your site and a list of links to its most useful pages. It is a community proposal, not a standard, and the major AI search crawlers do not say they use it. It is cheap to add, but it does not replace crawlable HTML.

llms.txt gets a lot of attention, often presented as the robots.txt of AI. It is something different: robots.txt states what crawlers may fetch, while llms.txt is a reading guide that points AI tools to the content worth reading. Here is what the proposal says, what is known about its use, and how to write one.

On this page

Where does llms.txt come from?
What goes into an llms.txt file?
Who actually reads llms.txt today?
Should I add one anyway?
How do I write a good llms.txt?
How does AISeeWhy treat llms.txt?

Where does llms.txt come from?

Jeremy Howard proposed the /llms.txt file in September 2024, and the proposal is maintained at llmstxt.org. Its aim is to standardise one file that helps language models and agents use a website at inference time, for example when a coding assistant needs a library's documentation, rather than for training.

The idea responds to a real problem. Web pages carry navigation, scripts, banners and layout. A short, curated Markdown file with links to clean versions of the key pages is easier for a model to work with inside a limited context window.

What goes into an llms.txt file?

The file is Markdown with a fixed order. Only the first element is required:

An H1 with the name of the site or project (required).
A blockquote with a short summary of what the site is.
Optional paragraphs or lists with more detail, without headings.
H2 sections, each with a list of Markdown links in the form [name](url), optionally followed by a colon and a note.
An optional section named Optional for secondary links that a tool can skip when it needs less context.

# Example Co

> Example Co makes scheduling software for clinics. This file lists our product, pricing and help pages.

## Product

- [Features](https://example.com/features): What the app does
- [Pricing](https://example.com/pricing): Plans and prices in USD

## Help

- [Getting started](https://example.com/help/start): Setup in 10 minutes

## Optional

- [Changelog](https://example.com/changelog)

The proposal also suggests offering Markdown versions of pages at the same URL with .md added, for example page.html.md, so tools can fetch clean text.

Who actually reads llms.txt today?

This is where it pays to be careful. The llmstxt.org site says the files are used most heavily for software documentation, where coding agents follow them. OpenAI, Anthropic, Google and Perplexity do not list llms.txt among the signals their search crawlers use in the crawler documentation we reviewed.

Google goes further for its own Search features. Its guidance on AI Overviews and AI Mode says you do not need to create new machine-readable files, AI text files or markup to appear in them. For Google Search, llms.txt has no stated role.

So the honest answer is: some AI tools, mostly coding assistants and agents that are pointed at a site, may read it. The crawlers that decide whether AI search cites your page work from your HTML.

You can see for yourself whether anything reads your file. Search your server or CDN logs for requests to /llms.txt and look at the user agents. If no requests appear over a few weeks, that tells you more about its value for your site than any general claim.

Should I add one anyway?

For most sites it is a low-cost, low-certainty addition. It takes a few minutes, it cannot hurt your search visibility and it may help agents that look for it. It is most useful for documentation, developer tools and products with a lot of reference material.

What it cannot do is fix access problems. If a firewall blocks AI crawlers, your robots.txt disallows them or your content only appears after JavaScript runs, llms.txt changes nothing. Fix those first.

How do I write a good llms.txt?

Put it at https://yourdomain.com/llms.txt and serve it as plain text with status 200, not an HTML page.
Write the summary the way you would explain the site to a new colleague in two sentences: what it is, who it is for.
List the pages that answer real questions: product, pricing, documentation, policies. Skip tag pages and archives.
Use absolute URLs and keep the list short enough to stay useful; a few dozen links is plenty for most sites.
Keep it current. A file that links to removed pages is worse than none.
Do not use it to state permissions. Access rules belong in robots.txt.

How does AISeeWhy treat llms.txt?

AISeeWhy checks whether your site has an llms.txt and whether the audited page is listed in it. Because AI crawlers rarely read it today, a missing file is a P3 finding, the lowest priority, and the report includes a draft built from your page title, description and sitemap that you can edit and publish.

Frequently asked questions

Is llms.txt an official standard?

No. It is a community proposal published at llmstxt.org. It is not an IETF or W3C standard, and no AI vendor is bound to read it.

Does llms.txt block AI training?

No. It is a list of links, not a set of permissions. To opt out of training, use the training tokens in robots.txt, such as GPTBot, ClaudeBot or Google-Extended.

Do I need llms.txt to appear in Google AI Overviews?

No. Google says you do not need AI text files or special markup to appear in AI Overviews or AI Mode. The page must be indexed and eligible to show a snippet.

Can llms.txt live in a subfolder?

The current version of the proposal allows a file at a subpath, such as /docs/llms.txt, covering the pages under that path. Most sites only need one file at the root.