One robots.txt Line That Speaks to Every AI Lab: Implementing the Shared Training Opt-Out

Shivam Kumar · VisionQuantech · October 31, 2026

On October 30, 2026, OpenAI joined Google, Microsoft, and Anthropic in backing a shared robots.txt signal for AI training preferences. If you run a website, this is the rare standards announcement you can act on in ten minutes. Here is what it is, how to implement it, and what not to break while you do.

The problem, in one paragraph

Until now, opting out of AI training meant writing a separate robots.txt block for every lab's crawler: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and more, with a new block every time a new lab appeared. Miss one name and your content was in. Worse, training rules and search rules lived in the same file, so a mistake could hide you from search results while you were only trying to opt out of training. The new signal replaces that pile of per-crawler rules with one cross-vendor policy token.

How the signal works

The signal is defined in IETF individual Internet-Draft draft-kelley-aipref-training-preference-signal-00, authored by Kevin Kelley of Anthropic and dated October 30, 2026, in the IETF AI Preferences working group. Its design is deliberately minimal: it introduces no new directives and no new grammar. It reuses RFC 9309, the robots.txt standard, unchanged — same path language, same longest-match precedence, same group structure.

The only new piece is the token __usage_ai-training, used as a User-Agent value:

User-Agent: __usage_ai-training
Disallow: /

That says: no generative AI training on any path of this site. Want to opt out of everything except your press kit? The draft's own example does exactly that:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Path-specific preferences work the way robots.txt authors already expect, because nothing about path matching changed. And because the token looks like an unfamiliar user-agent name, existing parsers that have not implemented the signal simply ignore the group — the same way they ignore any group naming a crawler they do not recognise. Deploying it breaks nothing for anyone.

The crucial separation: usage vs. crawling

The draft is explicit on the point developers most need to internalise: the policy token is not a crawling directive. Content-usage preferences and crawler traffic management are independent and orthogonal. Blocking a bot from visiting your site and telling a lab not to train on your content are two different statements.

In practice that means your file should carry both kinds of rules, separately:

# Search visibility: unchanged
User-agent: *
Allow: /

# Training preference: the new shared signal
User-agent: __usage_ai-training
Disallow: /

# Legacy per-crawler training blocks: KEEP these for now
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /

Do not delete the legacy blocks yet. Adoption is staggered: Anthropic has already adopted the signal, OpenAI says it will implement it in the coming months, and Google and Microsoft plan to support it next year. Until every lab you care about has switched it on, the per-crawler rules are the only protection that works everywhere. OpenAI itself says its existing crawler controls remain in effect alongside the new signal. Treat the token as an addition, not a replacement, and re-audit in six months.

Also note the wildcard rule: a User-agent: * group must not be interpreted as matching the policy token. If no group in your file contains the token, no preference has been expressed. Silence is not consent and it is not refusal — it is simply no signal. If you want a preference recorded, write the group explicitly.

What the token actually means

The meaning comes from the working group's vocabulary draft (draft-ietf-aipref-vocab-08, Keller/Thomson, September 2026). "AI Training" means using an asset to modify the learned parameters of a generative AI model, excluding models used for exclusively non-generative tasks. Separate categories cover AI Use (content fed to a model as input) and Search (selecting assets and directing users to them). Where statements conflict, the most restrictive preference applies.

Two honest limits to keep in your mental model. First, the signal is voluntary: the vocabulary draft says it does not ensure preferences are followed, and the choice is left to the recipient. It is a request, not a lock — the same as every robots.txt rule since 1994. Its power comes from adoption by the major labs. Second, it is forward-looking only. It says nothing about content already crawled or models already trained, and it is not a licence or a deletion request.

If you build crawlers or datasets

Implement the token now. Your existing robots.txt parser already handles the syntax — there is nothing new to parse — so the work is in the policy layer: when a group keyed by __usage_ai-training disallows a path, exclude that path from training corpora, and propagate the preference with the data if you redistribute it. Honouring it early is cheap, and it puts you on the right side of the publishers whose content your product depends on.

The checklist

1. Add the __usage_ai-training group to your robots.txt today.
2. Keep all existing per-crawler training blocks until adoption is complete.
3. Verify you have not changed any search-crawler rule — search visibility and training preference are separate decisions.
4. Test with a robots.txt validator, and confirm the file is served at your site root with the right content type.
5. Calendar a re-check: OpenAI's implementation update in the coming months, Google and Microsoft next year, and the draft's expiry on May 3, 2027.

The open web runs on agreements like this one: unenforceable, widely honoured, stronger than they look. One line, one meaning, every lab — that is a genuine improvement over a dozen dialects, and it is ten minutes of work to adopt.

Sources: OpenAI announcement (October 30, 2026) as reported by Unite.AI; IETF draft-kelley-aipref-training-preference-signal-00 and draft-ietf-aipref-vocab-08; adoption timelines as reported October 30, 2026. The draft is an individual Internet-Draft with no formal IETF standing.

Shivam Kumar is the founder of VisionQuantech, building AI research and sovereign AI systems.