What is llms.txt? Definition and Examples
llms.txt is a proposed standard: a markdown file at a site's root that gives AI models a curated map of the most important pages, so assistants can find and cite the right content.
llms.txt is a proposed standard: a markdown file at your site’s root that gives AI models a curated map of your most important pages, so assistants can find and cite the right content instead of guessing. It was proposed in September 2024 by Jeremy Howard of Answer.AI. The honest status matters as much as the definition: a small minority of sites ship one, mostly documentation sites, and no major AI engine has committed to reading it. We publish one ourselves, and we will tell you exactly why that is still worth doing.
How llms.txt works#
The format is deliberately simple, plain markdown a human can read:
- An H1 with the site or project name, followed by a short blockquote summary of what the site is.
- Sections of curated links, each with a one-line description of what the page covers and why it matters.
- An optional companion file, llms-full.txt, which inlines the complete text of the key pages so a model can read everything in one fetch.
The file lives at the root, like tryprefer.com/llms.txt. The idea: models have limited context and crawl budgets, so a curated map beats forcing them to reconstruct your site’s structure from raw HTML.
The honest status: who reads it today#
As of mid-2026, no major AI engine has announced that it uses llms.txt. OpenAI, Anthropic, Google, and Perplexity all publish crawler documentation; none of it commits to fetching the file. Google’s search team has publicly compared llms.txt to the old keywords meta tag, a signal sites volunteer that engines have little reason to trust. Adoption is real but concentrated in developer documentation, where tooling generates the file automatically.
So why ship one? Three reasons, none of them hype. It costs minutes. It is the single best curation exercise you can run (deciding which pages explain your company forces clarity most sites never reach). And if an engine does start reading it, you are already on the record. That is the whole case. Anyone promising rankings from llms.txt is selling something.
llms.txt vs robots.txt vs sitemap.xml#
The distinction that trips people up: llms.txt does not control anything. Blocking or allowing AI crawlers is still robots.txt’s job (check yours with our free AI Crawler Access Checker). llms.txt only suggests what to read once a crawler is in.
Example#
Our own file at tryprefer.com/llms.txt is generated at build time from the same source of truth as the site: the company summary, current pricing, the product pages, the research, and the playbook, each with a one-line description. When any fact changes, the file regenerates with it, so it never drifts from what the site says.
How to ship one#
The fastest path is our free llms.txt generator: it builds a clean, curated file from your pages in the browser, no signup. The wider practice of making your content easy for models to read and cite is answer engine optimization; llms.txt is one small, cheap piece of it.
How Prefer helps with llms.txt#
Prefer’s audit checks whether you have an llms.txt, whether AI crawlers can reach your site at all, and, far more importantly, whether the engines actually cite you, because the file is the cheapest item on that list, not the one that moves the number. Run a free AI visibility audit to see the whole picture.
Ship an llms.txt because it costs ten minutes and forces you to name your best pages, not because engines have promised to read it. Today, none have.
- Do AI crawlers actually read llms.txt?
- What is the difference between llms.txt and robots.txt?
- How do I create an llms.txt file?
Frequently asked questions
What is llms.txt in simple terms?
A plain markdown file at yoursite.com/llms.txt that lists your most important pages with one-line descriptions, so an AI model can read a clean map of your site instead of crawling everything and guessing what matters.
Do AI engines actually read llms.txt?
No major engine has committed to it. OpenAI, Anthropic, Google, and Perplexity have not announced support, and Google's search team has publicly compared it to the old keywords meta tag. Treat it as low-cost insurance and a useful curation exercise, not a visibility lever with proven effect.
How is llms.txt different from robots.txt and sitemap.xml?
robots.txt controls which crawlers may access which paths. sitemap.xml is a machine-readable inventory of every URL for indexing. llms.txt is a human-readable, curated reading list for AI models: not every page, just the ones that best explain who you are and what you offer.
What is llms-full.txt?
A companion file from the same proposal that inlines the full text of your key pages in one document, so a model can read the content directly without fetching each URL. It is optional and larger; llms.txt is the map, llms-full.txt is the map plus the territory.
How do I create an llms.txt file?
List your site name, a one-paragraph summary, then sections of links with one-line descriptions, in markdown. Or generate one: our free llms.txt generator builds a clean file from your pages in the browser, no signup.