How to write an llms.txt for your side project
An llms.txt file is a short Markdown map of your site, published at /llms.txt (or under a path such as /docs/llms.txt). The idea, set out in the community proposal, is to give coding agents and chat assistants a concise
An llms.txt file is a short Markdown map of your site, published at /llms.txt (or under a path such as /docs/llms.txt). The idea, set out in the community proposal, is to give coding agents and chat assistants a concise place to start, instead of asking them to parse a whole HTML site through navigation, ads, and JavaScript.
It is not an official web standard. Putting a file at /llms.txt does not guarantee that any model, crawler, or answer engine will read it, cite you, or mention your project. Treat it as a courtesy file for agents that already choose to look, the same way robots.txt is a courtesy file for crawlers that choose to honour it.
That is still enough reason to write a careful one for a side project, especially if you have docs or an API that an agent might need mid-task.
The format, in plain terms
The proposal uses Markdown on purpose, so a person and a parser can both read it. The shape is strict enough to be useful:
- An
H1with the project name. This is the only required section. - A blockquote with a short summary β the one or two facts someone needs before the rest of the file makes sense.
- Optional notes as ordinary paragraphs or lists. No extra headings in this block.
- One or more
H2sections, each containing a list of links in the form[name](url), optionally followed by: a short note. - An
## Optionalsection, by convention, for links an agent can skip when context is tight.
A good file stays small. The detail lives behind the links. Where you can, those links should point at clean, text-friendly pages β Markdown versions of docs are ideal β not at a marketing page wrapped in a cookie banner.
A worked example
Imagine a small tool called ShipLog: a CLI that turns pull requests into a changelog. A useful /llms.txt is curated, not a dump of every URL in the sitemap:
# ShipLog
> ShipLog is a CLI and GitHub Action that turns merged pull requests into a human-readable changelog. It is not a commit-message linter and it does not publish releases for you.
ShipLog reads PR titles, labels and breaking-change notes. Configuration lives in `.shiplog.yml` at the repo root.
## Docs
- [Quick start](https://shiplog.example.com/docs/quick-start.md): Install the CLI, generate a changelog from the last 20 merged PRs, and add the GitHub Action.
- [Configuration](https://shiplog.example.com/docs/config.md): All `.shiplog.yml` keys, including label maps and ignore rules.
- [GitHub Action](https://shiplog.example.com/docs/github-action.md): Workflow inputs, permissions, and a minimal YAML example.
## Optional
- [Changelog style guide](https://shiplog.example.com/docs/style.md): Voice and heading conventions if you want the output to match your docs.
- [Pricing](https://shiplog.example.com/pricing): The CLI is free for public repos; the Action minutes pack is optional.
A few editorial choices are doing the work here:
- The summary says what the tool is and what it is not. That saves an agent a wrong assumption.
- Docs come first, in the order a new user would need them.
- Each link has a one-line note, so the agent can pick a page without fetching all of them.
- Pricing sits under Optional. It matters, but it is not what you want loaded first when someone asks "how do I configure the label map?"
- The docs links point at
.mdversions. If you do not yet serve Markdown, link the HTML pages, then add.mdmirrors later. The proposal suggestspage.mdorpage.html.mdnext to the HTML, plus arel="alternate"link if you can.
After you publish the file, do the test the proposal itself recommends: give an agent only your llms.txt and ask it questions about the project. If it cannot find the install steps, or it invents a flag you do not have, the file needs a clearer link or a better note β not a longer sales paragraph.
Start from your sitemap, then edit
You do not have to write the first draft from a blank page. If you already have a sitemap.xml, you can pull titles and meta descriptions into a Markdown list and then cut ruthlessly.
A free generator that does exactly that first step is the llms.txt Generator. Enter your site. It reads /sitemap.xml when it can (or homepage links if there is no sitemap) and builds a draft from each page's title and meta description. The free run covers the homepage plus 8 pages. A one-time β¬9 Pro unlock covers the homepage plus 40 pages. Pages without a description are listed by title only, which is a useful hint: write those descriptions before you publish the file.
What it will not do β and what no generator should claim to do β is curate the file for you, invent Markdown mirrors you have not published, or get your project mentioned by an AI assistant. You still download or copy the draft, edit it, and upload it as /llms.txt on your own site.
A raw draft often looks like a flat ## Pages list. That is fine as a starting point. Your edit pass should:
- Drop URLs an agent does not need (tag archives, legal mirrors, thin marketing variants).
- Group the rest under headings such as Docs, API, and Optional.
- Rewrite the summary in your own words, including the gotchas.
- Add the missing meta descriptions, or write the notes by hand.
- Point at
.mdversions where they exist.
Put the file where an agent can actually fetch it
Save the edited file at the site root, or at the path it describes (/docs/llms.txt for documentation only). On a static host or in Next.js, that is usually a file in public/llms.txt. Then confirm you are not accidentally serving an HTML 404 or a soft-404 "page not found" document with HTTP 200:
curl -sI "https://yourproject.com/llms.txt"
curl -s "https://yourproject.com/llms.txt" | head
You want HTTP 200 and a Content-Type of text/plain or text/markdown, not text/html. If your host defaults unknown files to HTML, set the type explicitly. In a common Nginx snippet:
location = /llms.txt {
default_type text/plain;
charset utf-8;
}
Or in a Cloudflare Worker or any Response:
return new Response(llmsTxt, {
headers: { "content-type": "text/plain; charset=utf-8" },
});
If you already have robots.txt rules for AI crawlers, keep the two files' jobs separate. robots.txt says who may crawl. llms.txt says what is worth reading once someone is allowed in. A welcoming llms.txt does nothing if a Disallow blocks the agents you hoped would use it, and the reverse is also true: allowing a crawler is not the same as giving it a clear map.
Write the file as if a careful human will read it in one sitting. That is the version most likely to help a model, too.
I build PIT Toolkit.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.