What is llms-full.txt, and should you publish one next to llms.txt?
llms-full.txt is a single text file that holds the full text of a site's documentation, usually in Markdown, published next to the short /llms.txt index. It is not part of the llms.txt proposal at llmstxt.org but a convention that documentation sites adopted. Publish one if you run developer documentation that agents load in bulk; for most business sites, a good llms.txt and clean pages are enough.
Our article on what llms.txt is covers the index file itself: its format, where it comes from and who reads it. This one is about its larger sibling. If you have seen llms-full.txt on a documentation site, or a checker asked whether you have one, here is what the file is, what the proposal does and does not say, and whether it is worth the work.
On this page
Is llms-full.txt part of the llms.txt specification?
No. The proposal at llmstxt.org, now in version 2 (modified 10 August 2026), defines only files named llms.txt, at the site root or at any path, where a file covers the pages under its path. It also proposes Markdown versions of individual pages, at the page URL with .md added (page.html.md) or with the extension replaced (page.md), and index.html.md or index.md for URLs without a file name.
A single file with all the content appears nowhere in the specification. Version 1 described a tool, llms_txt2ctx, that expanded an llms.txt into one context file. Version 2 removed that tooling from the proposal and states the expected use instead: agents read or search the llms.txt, then follow the links they need. The FastHTML documentation, the proposal's own example, still serves an expanded file named llms-ctx-full.txt.
So llms-full.txt is a naming convention. There is no agreed format beyond "all the pages, one after another".
How is llms-full.txt different from llms.txt and .md pages?
Version 2 of the proposal adds a way for agents to find the Markdown version of a page and the llms.txt that covers it, using standard link relations in an HTTP header or in the page's HTML:
llms.txtis an index: a title, a summary and lists of links. The proposal says it should stay small enough to fit in a model's context..mdpages are clean copies of single pages. They are part of the proposal, and an agent fetches only the ones it needs.llms-full.txtis every page joined into one file. One request gives an agent everything, at the cost of size.
# HTTP response header on https://example.com/docs/setup.html
Link: </docs/setup.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"
# The same as HTML link elements in the page <head>
<link rel="alternate" type="text/markdown" href="/docs/setup.html.md">
<link rel="describedby" href="/docs/llms.txt"> There is no equivalent link relation for llms-full.txt. Sites that publish one usually link to it from their llms.txt.
Who publishes llms-full.txt?
Developer documentation sites, mostly. When we checked on 17 November 2026, these files were live:
- Cloudflare:
developers.cloudflare.com/llms-full.txt, about 71 MB. Its root llms.txt is 17 KB and links to a separate llms.txt for each product. - Anthropic:
platform.claude.com/llms-full.txt, about 53 MB. The 102 KB llms.txt links to it at the end. - OpenAI:
developers.openai.com/llms-full.txt, about 9.7 MB, described in the file as a single-file Markdown export. Its llms.txt is 6 KB. - Perplexity:
docs.perplexity.ai/llms-full.txt, about 4 MB, with aSource:line giving the URL of each page. Its llms.txt is 46 KB.
All four are API and product documentation, read by developers and coding agents. We did not count llms-full.txt in our November 2026 study of the 1,000 most popular domains; 132 of them published an llms.txt, and 117 of those followed the llmstxt.org format.
Do AI crawlers read llms.txt or llms-full.txt?
Not according to any documentation we could find. The crawler pages of OpenAI, Anthropic and Perplexity describe their bots and robots.txt tokens, and none of them says those crawlers read llms.txt or llms-full.txt on other sites. Google says you do not need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode.
Publishing these files for their own documentation is not the same as reading yours. llmstxt.org says the files are used most heavily for software documentation, where coding agents follow them. Chrome's Lighthouse includes an llms.txt audit, but it only flags a server error when fetching /llms.txt; a missing file counts as not applicable, and the audit does not mention llms-full.txt.
If you want evidence for your own site, search your server logs for requests to /llms-full.txt and look at the user agents.
How big can llms-full.txt be?
No limit is documented, because no specification defines the file. The practical limit is the reader. llmstxt.org notes that context windows, while larger than before, are still too small for most websites in their entirety. A file of several megabytes cannot be read in one pass, so an agent has to search it or read it in parts, which is what the llms.txt index and .md pages already allow.
If you publish one anyway:
- Split large documentation by product or section, as Cloudflare does with its per-product llms.txt files, rather than shipping one huge file.
- Put a source URL above each page, so an answer built from the file can still point to the right page.
- Include only public content. The file is easy to copy whole, and robots.txt rules apply to it like any other URL, including training crawler blocks.
- Regenerate it on every deploy. A stale full copy is worse than none, because it repeats outdated facts in one convenient place.
How do I generate and check llms-full.txt?
Start from a valid llms.txt and Markdown versions of your pages; llms-full.txt is then a build step. This script reads your llms.txt, fetches every linked .md page and joins them with a source line for each:
import re
import urllib.request
SITE_INDEX = "https://example.com/llms.txt"
HEADERS = {"User-Agent": "llms-full-builder/1.0"}
def get(url):
req = urllib.request.Request(url, headers=HEADERS)
with urllib.request.urlopen(req, timeout=20) as resp:
return resp.read().decode("utf-8", "replace")
index = get(SITE_INDEX)
links = re.findall(r"^\s*-\s*\[([^\]]+)\]\((https?://[^)\s]+)\)", index, re.M)
parts = [index.strip()]
for title, url in links:
if not url.endswith(".md"):
continue # include only pages that have a clean Markdown version
parts.append(f"# {title}\nSource: {url}\n\n{get(url).strip()}")
with open("llms-full.txt", "w", encoding="utf-8") as f:
f.write("\n\n---\n\n".join(parts) + "\n")
print(len(parts) - 1, "pages,", sum(len(p) for p in parts), "characters") Then check the result. The file should return status 200 as text/plain or text/markdown, not an HTML page, and its page count should match the links in your llms.txt. The free AISeeWhy llms.txt generator helps you write the index, and the free llms.txt validator checks it against the llmstxt.org format: title, summary, sections and links, with line numbers for each problem. llms-full.txt has no format to validate against, so the checks above are the useful ones.
Should llms-full.txt be a priority?
For most sites, no. Both files help only after a reader can reach your pages. The free AISeeWhy check fetches a page as 16 AI and search crawlers next to a browser and shows whether robots.txt, a firewall or JavaScript stops them, which decides more about AI visibility than either text file.
Fixes for this problem
- No llms.txt
- llms.txt does not start with a title
- Page is not listed in llms.txt
- llms.txt is too large: 1200 KB
Next steps
- robots.txt tester and checker for AI crawlers
- llms.txt checker and validator
- See an example full report
- AI crawler monitoring
- Plans and prices
Frequently asked questions
Is llms-full.txt required by the llms.txt proposal?
No. The proposal defines llms.txt and Markdown versions of pages. llms-full.txt is a convention that some documentation sites follow.
What is the difference between llm.txt and llms.txt?
The proposal's file name is llms.txt, with an s. A file at /llm.txt does not follow the proposal, and tools that look for llms.txt will not find it.
Where should llms-full.txt go?
No specification says. The examples above put it next to llms.txt at the root of the documentation site, and link to it from llms.txt so readers can find it.
Does llms-full.txt help with Google AI Overviews?
Google says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. Pages must be indexed and eligible to show a snippet.
Can I check llms-full.txt with an llms.txt checker?
Only partly. An llms.txt checker validates the index format, which a full-content file does not follow. For llms-full.txt, check the status code, the content type, the size and that it matches your current pages.
Sources
- llmstxt.org: The /llms.txt file, v2
- llmstxt.org: Changes
- Chrome for Developers: Lighthouse llms.txt audit
- Google Search Central: AI features and your website
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Cloudflare: llms-full.txt
- Anthropic: llms-full.txt
- OpenAI: llms-full.txt
- Perplexity: llms-full.txt
- AISeeWhy: AI Crawler Access Report, 2026-11