Google, OpenAI, Microsoft and Anthropic plan to honor shared AI-training opt-out

Originally published on The Daily Flare.

Google, OpenAI, Microsoft and Anthropic said on Friday, October 31, they plan to honor a shared robots.txt signal letting websites set AI-training preferences.

The companies’ plans are aimed at giving publishers one instruction they can use with participating AI providers, rather than separate rules for each company’s crawler. The proposal is being discussed at the Internet Engineering Task Force (IETF), which maintains internet standards. Its individual draft was submitted by Anthropic technical lead Kevin Kelley, but the IETF document tracker says it is not an official standards-track specification.

The draft uses a new user-agent token, __usage_ai-training, in a website’s robots.txt file. A publisher could use the usual “Disallow” and “Allow” rules under that token to state which parts of a site may be used for training. For example, a site could block training across the domain while allowing it for a press-kit directory.

The draft defines AI training as using content to change a generative AI model’s learned parameters. Models used only for non-generative tasks, such as classification or ranking, are outside that definition. A standard wildcard robots.txt group would not apply to the new token, and an existing file without a specific training-preference group would express no preference under the draft.

The signal concerns training, not whether a crawler may visit a site. It would not, by itself, stop other crawling or prevent material from appearing in Google Search. It also cannot reverse scraping that took place before a website sets its preference. The protocol is intended to guide future use of content for training.

Rollout dates vary by company

Anthropic said it has already started rolling out support. OpenAI said implementation would come in the coming months. Google is targeting 2027, with its documentation pointing to early in the year, while Microsoft plans to begin honoring the signal on February 1, 2027. As of Monday, November 2, the three companies named in the original announcement, Google, OpenAI and Microsoft, had not completed their planned support.

Google says it will treat a shared opt-out as if a publisher had blocked its Google-Extended crawler when the website has not set a separate Google-Extended preference. Google’s documentation describes that control as covering use of content to train future Gemini models and to ground responses in Gemini Apps and Vertex AI. Grounding means supplying material from the search index to a model while it answers.

Google also says its existing custom controls will remain in place after it adopts the shared signal. OpenAI’s existing settings will continue as well, so the common instruction is not intended to replace those company-specific options.

Microsoft plans to separate preferences for AI training from preferences for grounding, or using website content as reference material in generated responses. It says website owners will be able to configure whether their content can be used as reference information, with that setting available from February 1, 2027. Microsoft has also told publishers that scraping remains allowed by default until they change the setting.

The companies’ announcements come as publishers face increased scraping of articles for AI models and pursue legal challenges over the use of their work. The New York Times sued OpenAI and Microsoft in December 2023, alleging the companies used millions of its articles without permission or payment to train ChatGPT. The companies have invoked the fair-use exception to copyright law.

Microsoft and Google have cautioned that opting out of training could make a publisher’s material less likely to appear in AI applications and experiences. The shared signal is therefore a way to communicate a preference across providers, but each company’s implementation and related product controls will still matter to publishers.