Microsoft is replacing the NoArchive and NoCache tags it has read as AI controls since 2023 with a shared robots.txt training token and separate page level grounding tags. This tutorial audits a site for the old tags, adds the new token by path, adds the Microsoft page level overrides, and sets up the Webmaster Tools transition before the March 1, 2027 deadline.
On October 30, 2026, Google, Microsoft, and OpenAI each published a post backing one shared way for a site to state its AI training preference in robots.txt. The mechanism comes from an Internet Draft written by Kevin Kelley of Anthropic for the AI Preferences working group at the Internet Engineering Task Force. Microsoft went furthest in its post. It set firm dates, added page level controls of its own, and named the date when its old tags stop counting.
What You Will Build or Learn
You will end with three things in place. First, a robots.txt group that states your training preference to every participating provider at once, for the whole site or for the paths you choose. Second, Microsoft page level tags on any page that needs a different training choice, plus a grounding choice if you also want to control AI answers. Third, a transition record in Bing Webmaster Tools if your site now uses NoArchive or NoCache and you want a grounding restriction to survive the changeover.
One limit runs through the whole job. The shared token covers training only. Training, in the working group definition, means using an asset to modify the learned parameters of a generative AI model. It does not cover search indexing, and it does not cover grounding, where a system reads a page live to help inform a generated response.
Before You Begin
Back up the current robots.txt file and any template that prints robots meta tags before you change a line. The old Microsoft tags do not live in robots.txt. NoArchive and NoCache sit in a robots meta tag in the page head or in an X-Robots-Tag response header, so the audit below checks both places.
Note the rollout dates before you expect results. Microsoft starts honoring the new preferences on February 1, 2027, as it crawls. Google says it aims to support the control in 2027. OpenAI says it aims to implement the signal in the coming months. Publishing the group today puts your preference on record early. It does not switch anything on today at most providers.
| Date | What happens |
|---|---|
| December 15, 2026 | Temporary grounding transition preferences open in Bing Webmaster Tools, only for sites using NoArchive or NoCache as of that date |
| February 1, 2027 | Microsoft starts honoring the new training and grounding preferences as it crawls |
| March 1, 2027 | After this date, Microsoft no longer reads NoArchive and NoCache as a training or grounding preference |
| May 3, 2027 | The Internet Draft expires unless it is revised or adopted |
| September 30, 2027 | Temporary grounding preferences in Bing Webmaster Tools expire |
Step 1: Audit the Current File and the Old Tags
Action: fetch the live robots.txt file for your site and save a copy. Replace the example domain with your own.
curl https://example.com/robots.txt
Expected result: the live file prints in full. Look for groups that name AI crawlers or provider tokens, such as Google-Extended or GPTBot, and write down what each group allows or blocks. Those rules keep working after this migration, so the audit is your record of what must stay.
Action: check a sample page for the old Microsoft tags in both forms.
curl -s https://example.com/example-page/ | grep -i "noarchive\|nocache"
curl -sI https://example.com/example-page/ | grep -i "x-robots-tag"
Expected result: each match shows where an old tag lives, in the meta tag or in the header. Check the home page, one article page, and one page from each template your site uses. If nothing matches anywhere, your site never used the old tags, and you can skip Step 7.
Step 2: Decide the Training Choice for Each Path
Action: decide, path by path, which of the three states you want. A path covered by a Disallow rule states a preference against training. A path covered by an Allow rule states a preference in favour of training. A path with no group naming the token states no preference at all.
Expected result: a short list you can turn into rules in the next step. Two traps belong in this decision. A group for the wildcard token does not count, since the draft says wildcard groups must not be read as matching the policy token. And the token must match exactly, apart from letter case, so a spelling error turns the whole group into silence.
Step 3: Add the Shared Token Group to robots.txt
Action: add a new group to robots.txt. For a whole site opt out, the group is two lines.
User-Agent: __usage_ai-training
Disallow: /
Expected result: the group states a preference against generative AI training for every path on the site. The token has two leading underscores. Check that your editor or content system did not strip them or turn the name into a link.
Action: for a partial opt out, name the paths instead, and carve out any path you want to leave open with an Allow rule. The draft uses this shape, with a press kit left open.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Expected result: path rules follow the normal robots.txt rules, including longest match precedence, because the draft adds no new grammar and no new directive names. A single section works the same way, with a line such as Disallow: /archive/ in place of the whole site rule.
Step 4: Keep the Provider Rules You Already Have
Action: leave every existing provider group in the file, exactly as the audit found it. Add the shared group beside them instead of replacing them.
Expected result: the controls that work today keep working while support for the shared token arrives. Google states the point for its own token. When a site uses the shared token to opt out and Google-Extended is not explicitly specified, Google plans to read that as a Google-Extended disallow. A site that names Google-Extended keeps its own stated choice for Google, and Google says sites that allow Google-Extended stay eligible to appear in responses in Gemini Apps. OpenAI says the same in general terms. The new signal will not replace the crawler controls it already provides, and those controls will continue to apply.
Step 5: Add Microsoft Page Level Training Tags Where a Page Differs
Action: where one page needs a different training choice from the site rule, add the Microsoft page level form to that page. Use the meta tag in the page head, or the header form if the page has no HTML head to edit.
<meta name="Bingbot" content="ai-training:disallow">
X-Robots-Tag: Bingbot: ai-training:disallow
Expected result: for Microsoft, the page level setting overrides the robots.txt setting for that page. This layer is a Microsoft control. It is not part of the shared signal, and other providers are not bound by it. Microsoft starts honoring it on February 1, 2027, as it crawls the page.
Step 6: Set Grounding Separately If AI Answers Matter to You
Action: decide the grounding choice on its own. Grounding is the use of a page as input to help inform a generated response, in products Microsoft lists as AI Mode, AI Overview, Copilot, and Microsoft 365. The shared token says nothing about grounding, and there is no site wide robots.txt grounding rule. Microsoft sets grounding page by page only.
<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-grounding:disallow
Expected result: a Disallow value keeps the page out of grounding for Microsoft. The middle option, written as ai-grounding:disallowextended in the same tag or header pattern, allows grounding in Microsoft operated experiences only and shuts out grounding services Microsoft provides to third parties, which its post names as Web IQ and grounding through API access in Foundry. Allow is the default, so a page with no grounding tag may be used to inform responses.
Step 7: Record the Transition in Bing Webmaster Tools
Action: this step applies only if Step 1 found NoArchive or NoCache on your site and you want to keep a grounding restriction. From December 15, 2026, open Bing Webmaster Tools and record a temporary grounding preference. There is no temporary training setting, so the training side rests on the robots.txt group and page tags you already published.
Expected result: the temporary record holds your grounding choice for pages Microsoft has not yet recrawled. Microsoft says it might not recrawl every page between February 1 and March 1, 2027, which is why both halves are required before March 1. A temporary preference lasts until the page is recrawled and its new on page preference is found, or until September 30, 2027, whichever comes first. Skip the record, and pages with old tags default to Allow for grounding until a recrawl finds the new tag.
Step 8: Publish and Leave the Old Tags in Place for Now
Action: publish the updated robots.txt file at the root of the domain, and publish any template changes that carry the new meta tags. Leave the NoArchive and NoCache tags where they are.
Expected result: the old tags do no harm during the transition. Microsoft says that if it detects new training or grounding preferences on a site, it will disregard any NoArchive and NoCache preferences that remain. Removing the old tags can wait until after March 1, 2027, when they stop counting as AI preferences in any case. Note that NoArchive still has its older search meaning about cached copies for engines that read it that way, which is a separate decision from this migration.
Verification
Fetch the live file and the changed pages from outside your editor, and read what the server actually serves.
curl https://example.com/robots.txt
curl -s https://example.com/example-page/ | grep -i "ai-training\|ai-grounding"
curl -sI https://example.com/example-page/ | grep -i "x-robots-tag"
The first output shows your new group with the token spelled exactly as __usage_ai-training. The second and third outputs show the page level tags on the pages where you set them. Then check the one thing syntax cannot prove. A published preference takes effect only when a provider crawls and reads it, and Microsoft states that limit for its own rollout. IndexNow can signal that a page changed and raise the chance of a recrawl, but Microsoft does not guarantee a recrawl date, so treat any early test result as a reading of the calendar, not a failure of the file.
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| No provider reacts to the new group | The group sits under the wildcard token, or no group names the token | Add a group that names __usage_ai-training on its own User-Agent line |
| The group looks right and does nothing | The token is misspelled, often with one leading underscore lost | Retype the name with two leading underscores and publish the file again |
| Training is blocked and the page still appears in AI answers | Grounding was never set, since the token covers training only | Add the grounding tag from Step 6 to the page |
| A page follows the site rule and ignores its own tag | The tag names the wrong bot, or the header form is malformed | Use the Bingbot name and the exact forms in Steps 5 and 6 |
| Old restrictions vanish in March 2027 | No new preference was published, or the Webmaster Tools transition record was skipped | Publish the new preferences at once and accept that they take effect on recrawl |
| Pages drop out of Bing search after the edit | A crawler group was blocked while editing the file | Restore the crawler groups from your backup and state use preferences with the token instead |
| Google treats the site as opted out and Gemini Apps traffic stops | The shared opt out was read as a Google-Extended disallow, because Google-Extended was not named | Name Google-Extended in its own group with the choice you intend for Google |
Next Steps
Put two dates in your calendar. The first is March 1, 2027, the deadline in this tutorial. The second is a check in the second half of 2027, when Google support is due, the draft will have been revised or will have expired, and the temporary grounding records will have run out. Recheck the draft status at the IETF datatracker before that visit, since an individual draft has no formal standing and the token form could still change.
Sites with no old Microsoft tags have the easy version of this job. Steps 3 and 4 are the whole migration, and Steps 5 and 6 are there when a single page or the grounding question needs a finer answer. Either way, the file now says what you mean in one place, and it says it before the deadline forces the issue.
Steps, syntax, and dates checked against the IETF Internet Draft for the training preference signal, the Microsoft Bing post on training and grounding preferences, the Google Search Central post on the shared AI training control, and the OpenAI post on AI training preferences, November 2026. This procedure was verified against those documents. It was not run as a live migration on a production site in this run.
Sources
- Internet Draft, Machine Readable Training Preference Signal, IETF datatracker: https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00
- Microsoft Bing, Your content, your choice, clearer preferences for training and grounding: https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
- Google Search Central, Supporting a shared AI training control for websites: https://developers.google.com/search/blog/2026/10/ai-training-control
- OpenAI, Making it easier for websites to express AI training preferences: https://openai.com/index/helping-websites-express-ai-training-preferences/
- Body illustrations: Dirty Static, generated in house style
END OF TRANSMISSION
Stay curious. Question everything.
