An AI training opt-out is becoming easier for website owners to express, but the new industry proposal does not mean a publisher can switch off every AI use of its work. On October 30, 2026, OpenAI, Google and Microsoft announced support for a shared signal in robots.txt, while a newly published Internet-Draft described how that signal would work. Their commitments differ in timing and in the treatment of AI-generated answers.
The practical question for publishers is therefore not simply whether to opt out. It is which use they want to restrict: model training, live retrieval for AI answers, ordinary crawling or search visibility. Those are different activities, and confusing them could lead to settings that undermine a site's own distribution goals.
What the AI training opt-out actually adds
The October 30 Internet-Draft proposes a shared policy token named __usage_ai-training. A website can use that token in the familiar robots.txt file, with ordinary Allow and Disallow rules applying to the paths it selects. It introduces neither a new robots.txt directive nor a new method for preventing access to a server.
For example, a site could place a group beginning with User-Agent: __usage_ai-training and a Disallow: / rule to express a preference against using the whole site for generative-model training. A narrower path rule could express a different preference for part of the site. This is an explanation of the proposal, not a recommendation to deploy it unchanged: a publisher should first examine existing crawler rules and its business priorities.
OpenAI says it intends to honor the shared signal in the coming months while retaining its existing crawler controls. Google says it aims to support the mechanism in 2027. Microsoft says its new preferences will begin taking effect as Bing recrawls pages from February 1, 2027. A proposed rule and a company's future implementation pledge are not proof that every crawler already honors it today.
Training is not grounding, crawling or search indexing
Training changes a generative model's learned parameters. Grounding is different: an AI product may retrieve current webpages to help compose an answer, sometimes showing a citation or link. Crawling describes the act of fetching content. Search indexing makes pages eligible to appear in conventional search results. The draft explicitly treats content-use preferences as separate from crawler access rules.
A publisher can want to restrict training while still permitting search engines to index stories and AI services to cite them. Another publisher may prefer stricter controls on generated answers. Neither choice is inherently correct for every newsroom, and neither is guaranteed to produce or prevent traffic.
Google and Microsoft show why one setting is not enough
Google says that when a site uses the shared training opt-out without an explicit Google-Extended preference, it plans to treat the site as though Google-Extended were disallowed. That matters because Google's announcement also connects Google-Extended choices to eligibility for grounding in Gemini Apps. A publisher that wants one outcome for training and another for Gemini grounding needs to review the specific Google rules rather than assume the shared signal has no additional effect.
Microsoft's plan separates the two decisions more explicitly. Its shared training preference can apply to a site or path; it also plans page-level training controls. Grounding controls are page-level settings, with options to disallow grounding or limit it to Microsoft-operated experiences rather than third-party grounding services. Microsoft says these choices do not change eligibility for ordinary Bing search results, although restricting grounding can reduce visibility inside AI-generated experiences.
Microsoft also announced a transition for websites using older NoArchive or NoCache preferences. Its new rules begin applying on recrawl from February 1, 2027, and affected publishers should review the March 1, 2027 transition deadline. A page-level training preference may override the sitewide robots.txt training preference in Microsoft's implementation. Those details make a technical audit more useful than copying a generic opt-out snippet.
What the new signal does not settle
The IETF document is an Internet-Draft, not a final internet standard. Participation is voluntary, and different AI operators can implement or interpret company-specific controls differently. A robots.txt preference is also not an access-control wall: it does not stop a noncompliant actor from downloading a public page, erase material already used in model development or decide whether a particular use is lawful.
That last distinction connects to Unhyd's reporting on the Meta publisher-rights dispute, where contractual permissions and technical crawler settings were separate questions. A new common preference signal may simplify one part of the problem without settling licensing, copyright, compensation or enforcement.
A practical decision process for publishers
First, document the desired outcome for each use: future training, retrieval into AI answers, search indexing and crawler access. Second, inspect the actual robots.txt file and any page-level meta tags or HTTP headers. Third, compare each provider's current documentation with its announced rollout dates; do not assume a future policy is already active. Fourth, preserve a dated copy of the configuration and review traffic, citations and search performance after changes are discovered by crawlers.
Publishers should also distinguish training from the retrieval methods described in Unhyd's guide to retrieval-augmented generation. An AI answer that draws on a current webpage is not necessarily evidence that the page trained the underlying model. Conversely, an AI answer with no citation does not establish that the page was never used in training.
The October 30 announcement is meaningful because several major providers are converging on a common expression of publisher preference. Its real-world value will depend on deployment, consistent interpretation and whether newsrooms can preserve the visibility they want while restricting uses they do not. The immediate task is to make that distinction deliberately, not to treat a proposed robots.txt token as a universal AI off switch.
