Developers now ask an AI assistant before they open your docs. Tools such as Cursor and Claude Code fetch documentation pages while someone is coding, pull the text into a limited context window, and answer from whatever they found. If your site is a maze of navigation menus, cookie banners and scripts, the agent wastes its budget or guesses.
llms.txt is a small Markdown file that gives those agents a curated map of your documentation. This guide explains what it is, how it differs from llms-full.txt, sitemap.xml and robots.txt, how to write one with a template, how to generate it in CI, and what the evidence says about whether it actually helps.
What is llms.txt?
llms.txt is a plain Markdown file served from the root of your site, at yoursite.com/llms.txt. Jeremy Howard proposed it in 2024 as a way to point large language models at the most important parts of a site Mintlify. Think of it as a table of contents written for a machine: a title, a short summary, and lists of links with a one-line description of each page.
It is a convention, not a web standard. No search engine or AI vendor is required to read it, and some tools do while others ignore it. What it gives you is a predictable place to say "start here" to any agent that looks.
Why documentation teams care
• Agents read differently from people. They work inside token limits and treat a page as data to consume, not a site to browse Fern. One documentation-portal study reported that AI coding agents often squeeze their reading into one or two HTTP requests, so a curated index helps them pick the right page first time summary on Emergent Mind.
• The fallback is your sitemap. Without llms.txt, most AI tools start from sitemap.xml, which lists every URL and says nothing about which pages matter Mintlify.
• Markdown is cheaper to read than HTML. A clean text file removes the parsing overhead of navigation, scripts and layout, so more of the context window goes to your content.
If you are also building an AI assistant on top of your docs, pair this with the retrieval ideas in our guide to generative AI in technical documentation.