robots.txt, llms.txt, and cats.txt are not the same job

Three text files keep showing up in the same AI SEO slide. They share a root path and a .txt extension, and that is almost all they share.

robots.txt states which paths a given crawler may fetch. llms.txt is an optional Markdown map that helps agents find curated pages when they choose to look. cats.txt was invented as satire, yet it still cleared the same “proofs” people use to sell the second file: bots fetched it, Google indexed it, models repeated it, and ChatGPT endorsed it. If those observations prove a protocol, they also prove your office cat is a ranking factor. Adding all three and calling the stack complete does not answer the client brief. The work is clearer when you name which job you are buying before you invoice it, then match the artefact to that job.

What robots.txt actually controls

robots.txt is the long-standing Robots Exclusion Protocol file at the site root. Compliant crawlers read user-agent rules, Allow / Disallow paths, and often a sitemap pointer. Google’s own robots.txt guide is blunt about the limits: the file guides crawl behaviour, but it is not a security boundary, and it is not a reliable way to keep a URL out of Google Search if other sites link to it. For that you need noindex, authentication, or removal.

For AI crawlers, the same file is still the control that well-behaved bots document. OpenAI’s GPTBot, Anthropic’s crawlers, and similar agents that honour REP will respect a Disallow aimed at their user-agent. That does not mean every scraper is polite, and it does not mean a disallowed URL vanishes from every index, but it does mean crawl preference lives here, not in a Markdown wishlist.

If the ticket is “stop this bot hammering /api” or “leave staging alone”, you are in robots.txt territory. Speed and timeouts still matter once a bot is allowed, because a crawler that times out on a shell page never reaches the content you hoped it would quote. We wrote up that fetch foundation in Why AI Crawlers Need Fast, Crawlable Pages.

What llms.txt is for (and is not)

llms.txt is Jeremy Howard’s proposal for a Markdown overview at /llms.txt (or under a path such as /docs/llms.txt). The format is deliberately simple: a site title, a short summary, then headed lists of links to cleaner markdown versions of important pages. The pitch is orientation for agents that need a small, curated entry point, especially on documentation sites where coding agents waste tokens on nav chrome.

It does not grant or deny crawl access, and it does not replace robots.txt, sitemap.xml, or structured data. Google’s AI features documentation says you do not need new machine-readable AI text files or markup to appear in AI Overviews or AI Mode, and that there are no extra technical requirements beyond ordinary Search eligibility. Google does not document llms.txt as a ranking or inclusion control for those features, so if the business case is “rank better in Google’s generative results”, this file is not the lever to buy.

Lighthouse’s experimental Agentic Browsing category can still check for an llms.txt as a discoverability signal for agents. That is a tooling conversation, not a Search ranking claim. We go through the scoring and what to monitor in Lighthouse Agentic Browsing: How to rank in chatbots. Shipping the file because an audit row went amber is a product decision. Shipping it because a deck promised ChatGPT citations is a different claim, and it needs different evidence.

What cats.txt proved about “evidence”

Mark Williams-Cook published cats.txt as a joke standard: declare your office cats, job titles, and a PurrLevel. He was not proposing taxonomy for chatbots. He was showing that the industry’s favourite proofs for llms.txt also certified nonsense. Bots crawled the file, Google indexed it, and models returned cat biographies that existed only in that text. ChatGPT, for a while, cheerfully recommended cats.txt as a ranking tactic, then later converged on “it is a joke” once the surrounding web said so.

Keep the lesson, not the file. A fetch in the logs means a crawler requested a URL; an indexed .txt means Google found words at a URL. An LLM repeating a fact means retrieval found a page that ranked, and a model endorsing a tactic means the training and chat corpus is full of people selling that tactic. None of those observations, on their own, prove that a consumer chatbot honours a special protocol when it answers a user. cats.txt does not belong on a client delivery checklist, but it is useful in a training deck when someone treats crawl, index, and chat echo as proof of GEO impact.

Three files, three jobs

A sitemap still lists indexable URLs for discovery, and schema still describes entities on the page. Neither becomes redundant because someone published a new root file with a catchy name. Those older artefacts still do discovery and meaning work that a Markdown map cannot replace on its own.

Which file earns the ticket this week

A clearer ticket starts from the outcome the client named, not from a template that lists every root-path .txt file.

Crawl control for GPTBot, Googlebot, or a CDN bot-management policy still belongs in robots.txt, with a check in the logs that the right user-agent rules are live. An llms.txt can be low-cost housekeeping when agents and coding tools need a short map of docs, especially on documentation hosts that generate one automatically. “Proof that AI search will cite us because a .txt exists” is a different brief, and it needs a different success metric. Citation and answer inclusion still depend on crawlability, content quality, and retrieval, not on inventing a parallel exclusion file.

Agencies inherit another failure mode: shipping every trendy file so the audit screenshot looks complete. Three green checkmarks for three unrelated jobs is not a strategy. A clearer setup is one owner for crawl rules, one decision on whether an agent map is worth maintaining, and no claim that satire is a ranking lever.

For teams who still want AI-search visibility without buying a GEO SaaS stack, the useful checks stay ordinary. Performance and crawl health on the money pages matter more than another root file, and a shortlist of URLs worth defending helps when a client asks why a chatbot ignored them. We outline that path in AI Search Optimization: What to Monitor without a subscription.

FAQ

Does Google Search use llms.txt?

Google does not document llms.txt as a requirement or ranking input for AI Overviews or AI Mode. The current AI features guidance says you do not need new machine-readable AI text files or markup to appear in those experiences, and ordinary Search eligibility is enough for those features.

Is llms.txt a replacement for robots.txt?

No. robots.txt expresses crawl preferences; llms.txt orients agents that fetch it. Mixing the two in one “AI robots” story confuses access control with a documentation index.

Should we add cats.txt?

Only if you want the joke on purpose. It is useful in training decks to show why crawl-plus-index-plus-chat-echo is a weak proof, but it is not a production SEO control.

Does Lighthouse requiring llms.txt mean Google Search requires it?

No. Lighthouse’s Agentic Browsing checks are a separate product surface, so an audit row about discoverability for agents is not a Search ranking requirement.

CTA

Before the next AI SEO sprint, a useful ticket line is one sentence naming the job: crawl preference, agent orientation, or citation proof. The artefact should match that sentence. Fetch success, Core Web Vitals, and whether money pages stay fast enough for humans and bots still move the outcomes clients notice.

If you want that measurement on a multi-site portfolio without a separate GEO product, start a Watcher trial or run a free domain scan.

References

• Robots.txt Introduction and Guide (Google Search Central)
• AI features and your website (Google Search Central)
• The /llms.txt file, v2 (Jeremy Howard / Answer.AI)
• How cats.txt showed llms.txt evidence is GEO astrology (Mark Williams-Cook)
• The cats.txt standard (satirical companion site)
• Lighthouse agentic browsing scoring (Chrome Developers)
• Why AI Crawlers Need Fast, Crawlable Pages (Apogee Watcher)
• Lighthouse Agentic Browsing: How to rank in chatbots (Apogee Watcher)
• AI Search Optimization: What to Monitor without a subscription (Apogee Watcher)