Google and Microsoft back a shared AI training opt-out in robots.txt

What changed

Google said on 30 October 2026 that it plans to support a shared AI training opt-out in robots.txt in 2027, written as User-Agent: __usage_ai-training. When a site opts out with this token and does not specify Google-Extended, Google plans to read the line as a Google-Extended disallow.

A policy token is a robots.txt User-Agent name that states a use of content instead of naming a crawler. The token comes from an Internet-Draft filed on 30 October 2026 by Kevin Kelley of Anthropic, an individual draft discussed on the list of the IETF AI Preferences (AIPREF) working group, and it uses that group’s definition of AI training.

Microsoft announced on the same day that Bing will honor the shared token from 1 February 2027 and adds a separate AI Grounding preference, set page by page with a Bingbot meta tag or an X-Robots-Tag header. The grounding preference covers web-grounded responses in Copilot, Microsoft 365 and Bing, and Microsoft sets training and grounding as independent choices:

Bing’s grounding table gives each option a goal. Allow, the default, lets a page inform generated responses. DisallowExtended keeps the page in Microsoft-operated experiences such as Copilot and Microsoft 365 and takes it out of grounding services Microsoft provides to third parties, such as Web IQ. Disallow takes the page out of generated responses. “If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences,” wrote Krishna Madhavan, Principal Product Manager at Microsoft Bing.

Why it happened

Google and Microsoft each built their own AI switch first. Microsoft announced NOCACHE and NOARCHIVE controls on 22 September 2023, and a page tagged NOARCHIVE stayed in Bing search results while it left Bing Chat answers. Microsoft stops reading either tag as a training or grounding preference on 1 March 2027.

Google has offered Google-Extended since 2023, and Google’s crawler documentation says the token governs Gemini training and grounding in Gemini Apps without affecting inclusion in Google Search. A Search Console toggle Google added on 3 June 2026 decides whether a site appears in and helps ground AI Overviews and AI Mode, and has covered all websites worldwide since 31 August 2026.

Google says the shared token answers publishers, civil society groups and regulators who asked for industry-wide training controls, and that it fits the EU Code of Practice for general-purpose AI model providers, published on 10 July 2025.

What it changes for GEO

The shared token does three different things across Google and Microsoft, so every AI switch has to be judged engine by engine. At Microsoft the token stops training only. At Google it counts as a Google-Extended disallow, which also ends grounding in Gemini Apps, while Google AI Overviews and AI Mode follow the Search Console toggle and keep choosing the pages they show as before.

Brands that paste the token into robots.txt with no Google-Extended rule lose Gemini Apps answers once Google reads it, often without the owner knowing. Google-Extended is one token for training and Gemini Apps grounding, so at Google a brand can’t stay in Gemini Apps answers while refusing Gemini training. A company that sells something should accept training there, because the AI recommendation is the most valuable moment in a purchase.

At Microsoft, sites with a forgotten NOARCHIVE tag gain. That tag has kept pages out of Microsoft’s AI answers since 2023, and from 1 March 2027 those pages default to Allow for grounding unless the owner says otherwise. Brands that choose DisallowExtended lose the grounding services Microsoft provides to third parties, and Microsoft names none of the outside products built on them. Their Bing search eligibility, the route by which ChatGPT search finds its candidate pages, stays as it was.

Each of these switches decides only whether an engine may use a page. Microsoft says neither choice guarantees inclusion, traffic or payment, and more frequent recommendations come from content and from mentions on other sites, the work that GEO for brands in AI answers covers.

What to do

The head of marketing should ask the web team, before 1 February 2027, for a list of every AI switch on the site and what each one does. Where robots.txt carries the shared token, or will, the team adds an explicit Google-Extended allow so the site stays a source for Gemini Apps answers. The team keeps OAI-SearchBot allowed, since OpenAI’s crawler documentation says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, while GPTBot covers training.

Next, the team searches page templates for leftover noarchive and nocache tags, sets an explicit Bing AI Grounding preference before 1 March 2027 and checks the AI Overviews toggle in Search Console. Microsoft has to recrawl a page before a new setting takes effect, so the check is the page source plus Bing Webmaster Tools, and IndexNow raises the chance of a recrawl without fixing a date.