Google’s Shared AI Opt-Out Has a Catch: Your Old Rules Still Count.

Google announced plans on October 30 to support a shared AI-training preference in robots.txt, targeting support in 2027. There is a consequential detail: Google plans to treat a shared training opt-out as a Google-Extended disallow only when Google-Extended is not explicitly specified. Existing company-specific controls will continue alongside it. Adding the new signal is not the same as replacing the policy already written in the file.

Disclosure: This article was generated and fact-checked with AI using the public sources linked here. It was published through an automated workflow and has not been reviewed by Keith.

For a business website, that distinction belongs in the implementation brief. “Keep our content out of training” and “help buyers find our capabilities” express different goals. A developer should turn those goals into documented choices before reducing them to a robots.txt edit.

Start with what the public website is for

The TMC Produce website project describes a B2B site organized around buyer questions: sourcing, quality control, freight support, documentation, and contact when timing matters. It presents business capabilities before a sales conversation starts.

That provides a concrete setting for this discussion. It does not establish TMC’s current crawler rules or its preferred AI-training policy. Consider a hypothetical site with that same public-information purpose. Its owner might want buyers to find a logistics page while declining some uses of the page’s content for model training.

The development question is then specific: which use is being declined, for which paths, and under which provider’s documented controls? “Block AI” leaves too much undecided. It could refer to training, using a page to inform a generated answer, or crawling traffic. Those choices should not be collapsed into one undocumented switch.

Check the rule that wins, not just the new line

The October 30 Internet-Draft defines the policy token __usage_ai-training using ordinary robots.txt groups. It is an individual draft, not an endorsed IETF standard. Under the draft, a wildcard User-Agent group does not match that policy token, and the usage preference is distinct from crawler rules.

That means a review needs to consider both explicit policy groups and existing crawler groups. Seeing a broad wildcard rule is not enough to conclude that the draft’s shared training preference has been expressed.

Now consider a fictional configuration that adds a shared training disallow while retaining an explicit Google-Extended allowance. Under Google’s announced approach, do not assume the new shared entry reverses that company-specific choice. Decide whether the explicit allowance is intentional, then document the expected result. This is a proposed review example, not a description of TMC’s deployed configuration.

An effective review note can be short: the owner’s desired use policy, the applicable path, the provider-specific exception, and the documentation date. A reviewer should be able to explain the resulting preference without guessing why a rule survived from an earlier release.

Training and answer visibility need separate decisions

Microsoft’s October 30 announcement makes the separation explicit in its own implementation. It describes independent training and grounding preferences, with grounding meaning use of content to inform generated responses. Microsoft says the new preferences will be honored as it crawls starting February 1, 2027. They do not change eligibility for traditional Bing search results, subject to normal crawl and indexing requirements.

For the hypothetical logistics page, an owner could therefore have a reason to evaluate training and grounding separately. A public explanation may be useful as a cited source in an answer even when the owner does not want it used to train generative models. Another owner may choose differently.

Do not carry one provider’s interpretation into another provider’s configuration by analogy. The shared training signal reduces repeated expression of a preference; provider-specific behavior and rollout dates still require attention. Eligibility also should not be reported as a promise of traffic or inclusion.

Make the handoff include a rollout date

The useful deliverable is more than a text file. Keep an inventory of existing rules, the owner’s intended choices, the providers covered, and when each announced interpretation is expected to apply. Link the primary documentation beside each date so the next maintainer can recheck it.

Distinguish a rule successfully served by the website from a provider’s adoption of that rule. An HTTP response can prove what the site published. It cannot prove that a planned 2027 implementation is already operating, or that every consumer will honor the preference.

For a release, compare the previous and proposed configurations against representative public paths. Preserve search crawling where that is the agreed intent, check company-specific exceptions, and record any provider whose support is still pending. Retain the prior file so a mistaken edit can be corrected cleanly.

A shared signal can make policy maintenance simpler. The business still needs to choose what it means to allow or decline, and the developer still needs to know which rule applies. That is how a small robots.txt change becomes an understandable handoff instead of a confident promise the implementation cannot yet support.