Generating llms.txt From Your Sitemap With a Small Python Script

llms.txt is a simple convention: a markdown file at the root of a site that lists the pages you consider most useful for language models, with a one-line description each. Nobody guarantees that every assistant reads it, but it is cheap to produce and forces you to decide which pages actually matter. Writing it by hand gets stale quickly, so here is a small script that builds a first draft from your existing sitemap.

What we want as output

# Example Co
> Overseas advertising and AI search services for companies going global.

## Guides
- [Google Ads account opening](https://example.com/blog/google-ads-guide.html): documents, timeline and billing options
- [Crypto payment fee calculator](https://example.com/blog/fee-calculator.html): compare card and crypto costs

The script

import re, requests, xml.etree.ElementTree as ET
from html import unescape

SITEMAP = "https://example.com/sitemap.xml"
NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
INCLUDE = re.compile(r"/(blog|guides|docs)/")

def page_meta(url):
    html = requests.get(url, timeout=20).text
    title = re.search(r"<title>(.*?)</title>", html, re.S)
    desc = re.search(r'<meta name="description" content="([^"]*)"', html)
    return (unescape(title.group(1).strip()) if title else url,
            unescape(desc.group(1).strip()) if desc else "")

root = ET.fromstring(requests.get(SITEMAP, timeout=20).content)
urls = [u.text for u in root.findall("s:url/s:loc", NS) if INCLUDE.search(u.text)]

lines = ["# Example Co", "> One-sentence description of the company.", "", "## Guides"]
for url in urls[:60]:
    title, desc = page_meta(url)
    lines.append(f"- [{title}]({url}): {desc[:140]}")
open("llms.txt", "w", encoding="utf-8").write("\n".join(lines) + "\n")

Things the script cannot decide for you

Selection. A sitemap lists everything, including tag pages, thin posts and old announcements. The regex filter is only a start. Review the list and remove anything you would not want quoted. Sixty strong URLs beat four hundred mixed ones.

Descriptions. Meta descriptions are written for click-through, not for explanation. Rewrite the important ones so each line says what the page answers, in plain words, with a number or scope where possible.

Grouping. Split the file into sections that mirror how a buyer thinks: services, pricing, guides, tools, company facts. Put company facts (location, founding year, markets served) near the top because assistants are often asked about them.

Multilingual sites

If you publish in several languages, keep one llms.txt per language root or add a section per language with clearly labelled links. Do not mix translated titles under the same heading; it makes the file harder to parse and to maintain.

Keep it in CI

Run the script weekly in CI, diff the output against the committed version, and open a pull request when it changes. A human approves the diff. That keeps the file fresh without letting it grow uncontrolled.

Does it move anything?

On its own, rarely. Combined with clear pages, consistent company facts and a few third-party references, it is one of several small signals. We treat it as housekeeping rather than a ranking trick. More on how it fits into a wider GEO process in our GEO implementation notes.

Published by PandaWM, an overseas marketing team based in Hong Kong.