About llms.txt
What is llms.txt?
An llms.txt file is a short markdown document at the root of your site, at /llms.txt, that points a language model at the pages worth reading and says in one line what each one covers. It was proposed by Jeremy Howard of Answer.AI in late 2024.
The problem it addresses is real. A model reading a modern web page gets navigation, cookie notices, related-post widgets and markup wrapped around a small amount of actual content, and it has a limited window to work with. A curated list of clean pages with descriptions is a far better starting point than a crawl of the whole site.
The format is deliberately plain: a title, an optional summary, then headed sections of annotated links. That is the whole specification, and the llms.txt generator above produces exactly that shape.
What the file looks like
- An H1 with the site or project name. This is the only required line.
- An optional blockquote summarising the site in a sentence.
- Optional prose giving any context a reader needs first.
- H2 sections, each a markdown list of links written as name, URL and a short description.
- An optional section headed "Optional", for links that can be skipped when space is short.
A companion file, llms-full.txt, holds the full text of those pages inlined rather than linked. It is a bigger commitment to maintain and worth leaving until the short version is doing something for you.
A standard in the process of being adopted
llms.txt is new, and it is at the stage where publishers have moved first. Documentation sites took it up early, support has appeared across a growing list of frameworks and tools, and the number of sites publishing one climbs steadily.
The AI providers have not made public commitments about reading it yet, which is normal for a convention at this age - the same was true of XML sitemaps and of structured data before either became routine.
That makes it a sensible early move. It takes an afternoon to write, it costs nothing to keep, and being in place before support settles is worth more than waiting to be certain.
What makes a good one
The descriptions do the work. "Pricing" tells a model nothing it could not infer from the URL, whereas "Pricing tiers, what is included at each level, and how usage is counted" describes what is actually on the page and makes it selectable for a specific question.
Be selective as well. A file listing four hundred URLs is a sitemap with extra steps, and it defeats the purpose, which is to say which pages matter. Twenty well-described links beat the entire site.
Point at the cleanest version of each page. If you publish documentation in both a heavily styled and a plain markdown form, link the plain one.
Common mistakes
- Dumping the whole sitemap in, which removes the curation that gives the file its value.
- Descriptions that restate the link text instead of describing the page.
- Relative paths, which are ambiguous once the file is read anywhere but on your site.
- Writing it once and never revisiting it as the site changes.
- Treating it as a substitute for the page content being good, which it is not.
How this relates to AI search
Writing one is a useful exercise in itself, because it forces you to decide which twenty pages actually represent the site. That is the same question that decides which of your pages gets cited in an AI answer.
It works best alongside the rest of the groundwork: clear headings, content that answers a question in its first paragraph, structured data, and pages that are fast and reachable. A curated index helps a system that already wants to read you, and those other things are what make it want to.
Do both and they reinforce each other.
Using the llms.txt generator
Enter your site name, URL and a one-line summary, then add a section for each part of the site and the links worth including. Relative paths are expanded against your site URL, so you can type /docs and get the full address.
The output is editable before you copy or download it, which is where the descriptions usually get their final pass. Save the result as llms.txt in the root of your site so it resolves at yourdomain.com/llms.txt.
Where to go next
The file is an index, so the pages behind it are what matter. Check those pages are reachable with the
robots.txt checker, confirm your
XML sitemap is complete and current, and make sure the pages themselves carry proper
schema markup.