Ship an llms.txt and Markdown Mirrors for Your Docs
An llms.txt is a Markdown file at the root of your site that tells AI tools what your product is and which pages to read. Write a short summary, a block of facts people tend to get wrong, and grouped links to Markdown co
An llms.txt is a Markdown file at the root of your site that tells AI tools what your product is and which pages to read. Write a short summary, a block of facts people tend to get wrong, and grouped links to Markdown copies of your pages. Then generate it in the same step as your sitemap so it never drifts.
I maintain the one on demovexa.com. Today it is 447 lines. A blog post on our site isn't finished until it shows up there and its Markdown copy loads. Here is what I'd tell someone writing their first one.
Key takeaways
- llms.txt is a proposal from llmstxt.org, not a standard. No assistant has promised to read it.
- Open with the facts that get misquoted: what the product is, platforms, requirements, version.
- Link to
.mdcopies of your pages and serve them astext/markdown. - Put things a model can skip under an
## Optionalheading. - Build the file from the same index as your sitemap, then check it after each deploy.
Contents
- What is llms.txt, and does anyone read it?
- What should go at the top?
- How should you group the links?
- Why serve Markdown mirrors?
- How do you stop it going stale?
- FAQ
What is llms.txt, and does anyone read it?
It's a plain Markdown file served at /llms.txt. The format was proposed at llmstxt.org in 2024. Only one part is required: an H1 with the name of the project. After that comes a blockquote summary, some optional free text, and H2 sections that each hold a list of links with a short note after each one.
Who fetches it? I don't have a clean answer, and I'd be careful with anyone who claims to. The cost is one file. A tool that does look for it gets your own words instead of a scrape of your nav bar. That trade was easy for us.
This post adapts our longer piece on getting your product cited by AI assistants, which covers the page side of the same problem.
What should go at the top?
Put the facts that assistants most often get wrong. For a product, that's usually what it is, where it runs, what it needs and what it costs.
Ours opens with the summary, then a "Key facts" list. A trimmed version:
# DemoVexa
> DemoVexa turns a software product's code into polished product videos ...
> on the user's own computer.
Key facts:
- What it is: a desktop app for macOS, Windows and Linux that makes
product demo videos from a software product's own code.
- Output: MP4 (H.264, 30 fps) in 16:9 (1920x1080), 9:16 and 1:1.
- Requirements: macOS 13 or newer on Apple Silicon, Windows 10 or 11
(64-bit), or Linux x64.
- Version: 0.4.6.
## Product
- [Pricing](https://demovexa.com/pricing): plans, limits and billing questions
Write each fact so it still makes sense when quoted alone. "Runs locally" is vague. "The code folder is never uploaded" can be checked.
One warning from our own file. That Version: line is wrong the moment we ship a release, unless the build writes it. If you can't automate a fact, leave it out. A stale fact in a file built for machines is worse than no fact.
How should you group the links?
Group them by the question a reader brings, and make each note say what the page answers. Don't repeat the title in the note.
| Section | What goes in it | Example note |
|---|---|---|
| Product | Home, features, pricing, FAQ | "plans, limits and billing questions" |
| Docs, split by task | Getting started, setup, each workflow step | "render full-quality MP4 files to download" |
| Blog | One line per post, newest first | The post's meta description |
| Optional | Full docs dump, changelog, legal pages | "every documentation page as Markdown" |
Our docs links are split into sections like "Getting started", "Workspaces" and "Making videos", and each one points at a .md URL.
Now the honest part. Ours is 74 KB, which is far bigger than the proposal has in mind. The blog list alone has 49 entries. If I were starting today I'd keep the main file to product and docs, and move the blog list under Optional. Big files eat the context window of whatever reads them.
If you want one file with everything, add llms-full.txt and link it from Optional. Ours holds every docs page as Markdown and runs to roughly 400 KB.
Why serve Markdown mirrors?
Because HTML pages carry a lot of noise. Scripts, nav, footers, consent banners. A Markdown copy at the same path plus .md gives a reader just the text.
On our site, every blog URL has a twin. The .md version returns text/markdown and starts with front matter: title, description, date, author and a source line with the canonical URL. That last field matters. When someone copies the text, the original address travels with it.
Check yours like this:
curl -sI https://demovexa.com/blog/demo-video-sample-data.md | grep -i content-type
# content-type: text/markdown; charset=utf-8
The same idea applies to video. A how-to clip with proper subtitle files gives machines the spoken steps as text.
How do you stop it going stale?
Treat llms.txt as a build output, not a hand-edited file. Ours sits on the same publish checklist as the sitemap.
- Generate llms.txt from the same content index that builds
sitemap.xml. - Generate the
.mdmirrors at build time, from the same source as the HTML. - Pull facts like pricing and requirements from the data your product pages use, so they can't disagree.
- After each deploy, check the new page is listed and its mirror loads.
Step 4 is a few lines of shell:
slug=demo-video-sample-data
curl -s https://demovexa.com/llms.txt | grep -q "$slug" \
&& echo "listed" || echo "missing from llms.txt"
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" \
"https://demovexa.com/blog/$slug.md"
Run it in CI after every deploy, so a missing page fails loudly instead of quietly.
FAQ
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may fetch. llms.txt is a reading list. It doesn't allow or block anything.
Will llms.txt help my Google rankings?
I've seen no evidence that it does. Google ranks your HTML pages. Treat llms.txt as something for AI tools, and keep doing normal SEO.
Should the Markdown mirror be the canonical page?
No. Keep the HTML page canonical and put its URL in the mirror's front matter.
How long should llms.txt be?
Short enough to read in one pass. Product and docs first. Anything long goes in llms-full.txt.
Baltej Singh builds DemoVexa, a desktop app and CLI that turns a local codebase into narrated product demo videos (how it works). He writes from Delaware, USA.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.