AI Training Opt-Out in robots.txt: What the New Token Changes

Q: What exactly do I add to robots.txt to opt out of AI training?

Add a group with User-Agent: __usage_ai-training followed by Disallow: / to cover the whole site. You can use Allow and Disallow lines for specific folders, exactly as with any crawler. The token must be named explicitly. A User-Agent: * group does not count, so an old blanket rule does not express a training preference under the draft.

By Jaspal Singh October 31, 2026 Updated: October 31, 2026

Google, Microsoft, OpenAI and Anthropic said on October 30, 2026 that they will honor one shared robots.txt entry, User-Agent: __usage_ai-training, as a site's answer on whether its pages may train generative AI models. A single group can replace a stack of per-company rules. It does not affect search ranking, and support only starts in 2027.

What Happened

On Friday, October 30, 2026, the four companies published coordinated announcements backing what Google calls "a shared AI training control." The mechanism is set out in an Internet-Draft filed with the Internet Engineering Task Force the same day: draft-kelley-aipref-training-preference-signal-00, "Machine-Readable Training Preference Signal", written by Kevin Kelley of Anthropic. The draft is an individual submission. It has no IETF endorsement yet, and it expires on May 3, 2027 unless it is revised or adopted.

Each company made a different promise. In the Google Search Central post by Search Advocate John Mueller, Google says it aims to support the control "along with other AI providers in 2027." Its crawler documentation narrows that to early 2027. Microsoft gave the firmest date. Its Bing Webmaster blog post says Bingbot will start honoring the new preferences as it crawls from February 1, 2027. SiliconANGLE reported that OpenAI plans similar controls "in the coming months." Anthropic employs the draft's author, but no implementation date had been published at the time of writing. The Verge reported that Meta supports the idea but has not committed to implementing it.

Microsoft also announced a separate page-level control it calls AI Grounding. Its schedule runs from December 15, 2026, when temporary transition settings open in Bing Webmaster Tools, through September 30, 2027, when those temporary settings expire.

Why It Matters

A site that wants to stay out of AI training today has to know every crawler by name. GPTBot belongs to OpenAI, ClaudeBot to Anthropic, Google-Extended is Google's training token, and other companies run their own. Each needs its own robots.txt group, and that group only works if the site owner knows the bot exists. A new AI company that launches next month is not covered by last month's file.

The shared token turns that list into one statement of intent. A participating company reads the same entry, so a publisher writes it once and newcomers inherit it as long as they adopt the convention. That is the main gain for small sites. Few independent bloggers or shop owners track crawler announcements, and many of their robots.txt files were last edited years ago.

It also separates two decisions that robots.txt has long blurred together. Blocking a crawler stops it from fetching pages at all. Blocking a use says something about what may be done with pages that are fetched. Google's post spells out the consequence: Google-Extended "does not impact a site's inclusion in Google Search," and the new token is read the same way. A publisher can refuse training without disappearing from search results. Before Google-Extended existed, that trade-off kept many sites from blocking anything.

There is a regulatory reason as well. Google's post notes that the proposal fits the EU's Code of Practice for general-purpose AI model providers under the AI Act, which treats robots.txt as a primary way for publishers to state machine-readable preferences. A shared, documented token gives AI companies something concrete to point to when they are asked how they respect opt-outs.

How It Works

The draft adds no new syntax to robots.txt. It reuses the format standardized in RFC 9309, the Robots Exclusion Protocol, and puts a reserved name where a crawler's name would normally go. A site that wants to opt its whole domain out of AI training adds this group:

User-Agent: __usage_ai-training
Disallow: /

Paths work exactly as they do for crawlers, including RFC 9309's longest-match rule. The draft's own example disallows the whole site except one folder, so a publisher can keep a press kit open to training while the rest stays closed:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Several details in the draft matter in practice:

The wildcard does not count. A User-Agent: * group must not be read as applying to the token. An old file that blocks everything for * therefore says nothing about training under this scheme.
Silence is not a no. If no group names the token, the site has expressed no preference. Microsoft documents allow as the default.
The double underscore is deliberate. The draft notes that some deployed parsers match bot names more loosely than the standard says. An unusual prefix makes it unlikely that a real crawler's name will collide with the token by accident.
It is a usage signal, not a crawl rule. The token does not tell any bot to stop fetching pages. Crawler-specific groups keep doing that job, and the two operate independently.
The rules apply as of the fetch. Preferences are judged against the robots.txt in effect when a page was retrieved. Changing the file later does not reach back to copies already collected.

What counts as "AI training" comes from the IETF AI Preferences (aipref) working group's vocabulary draft. It means using content to modify the learned parameters of a generative AI model. Models used only for non-generative work such as classification, ranking or scoring are excluded, even if they could technically produce output. A search engine training its ranking system is therefore outside the definition. A chatbot's underlying model is inside it.

Each company maps the shared token onto its own controls. Google says that if a site opts out with the token and does not mention Google-Extended, it will act as if Google-Extended had been disallowed. A site that explicitly allows Google-Extended keeps that permission, and Google says such sites remain eligible to appear and receive clicks through grounding in Gemini Apps. Company-specific rules continue to work, so a publisher can opt out broadly and then allow one partner it has a licensing deal with.

Microsoft adds page-level options. A meta tag or HTTP header naming Bingbot can carry ai-training:disallow, and a page-level setting overrides the robots.txt one. Its separate grounding preference covers something different: a page being fetched to answer a user's question in Copilot or Bing's AI features, rather than being used for training. Grounding has three settings. Allow is the default. Disallow keeps the page out of AI answers. DisallowExtended permits Microsoft's own products but not third-party grounding services built on its technology. Microsoft says neither preference changes eligibility for ordinary Bing results. It is also retiring an old workaround: after March 1, 2027, Bing will stop reading NoArchive and NoCache as training or grounding signals.

What's Still Unknown

The biggest gap is enforcement. Robots.txt has always been voluntary, and the token binds only companies that choose to read it. Scrapers that ignore robots.txt today will ignore this line too. The announcement does not change the legal position of training on content that was fetched anyway.

Coverage of past data is also unclear. Because the draft judges preferences at fetch time, models already trained on a site's pages are not affected. None of the four companies said whether opting out triggers removal from existing datasets.

Timelines are uneven. Microsoft named a date and Google named a quarter. OpenAI's "coming months" and Anthropic's lack of a stated date leave publishers guessing when those crawlers will start reading the line. Meta, one of the largest crawlers, has not committed at all.

The standard itself may still change. The draft is an individual submission. The aipref working group could revise the token, the definition or the matching rules before anything becomes a formal standard. Google says the wider aipref vocabulary, which covers uses beyond training, is still being negotiated.

Finally, the token says nothing about money. Microsoft states plainly that the preferences "do not establish licensing or payment arrangements." Publishers looking for compensation still have to negotiate separately.

Frequently Asked Questions

What exactly do I add to robots.txt to opt out of AI training?

Add a group with User-Agent: __usage_ai-training followed by Disallow: / to cover the whole site. You can use Allow and Disallow lines for specific folders, exactly as with any crawler. The token must be named explicitly. A User-Agent: * group does not count, so an old blanket rule does not express a training preference under the draft.

Will opting out hurt my Google or Bing search rankings?

Both companies say no. Google says Google-Extended, which the new token maps to, does not affect inclusion in Google Search and is not a ranking signal. Microsoft says neither its training nor its grounding preference changes eligibility for traditional Bing results. Normal crawling still applies, so do not block Googlebot or Bingbot themselves unless you want to leave search entirely.

When do the companies start honoring the new token?

Microsoft has the firmest date and says Bingbot will start honoring the preferences from February 1, 2027 as it recrawls pages. Google says early 2027. OpenAI said it plans to add controls in the coming months. Anthropic employs the author of the draft but had not published an implementation date. Meta supports the idea but has not committed to it.

Should I remove my GPTBot, ClaudeBot or Google-Extended rules?

No, keep them for now. Company-specific rules keep working alongside the shared token, and they protect you until each provider actually starts reading the new line. They also let you make exceptions. For example, you could opt out of training broadly but explicitly allow one company you have a content licensing deal with, and that specific rule takes precedence for that company.

Does this stop AI chatbots from quoting or summarizing my pages?

Not by itself. The token covers training, meaning changes to a model's learned parameters. Fetching a page to answer a user's question in real time is a separate use. Microsoft handles that with its own page-level AI Grounding setting. Google says sites that allow Google-Extended can appear in Gemini Apps answers, which implies an opt-out reduces that exposure there.

Does opting out remove my content from models that are already trained?

No. The draft judges preferences against the robots.txt that was in effect when a page was fetched, so it does not reach back to copies collected earlier. None of the companies announced any process for removing past data from existing training sets. The signal only affects pages retrieved after a provider starts reading the token.

Is this an official internet standard yet?

Not yet. It is an individual Internet-Draft filed on October 30, 2026, which carries no formal IETF standing and expires on May 3, 2027 unless renewed. The companies say they will work toward a standard through the IETF's AI Preferences working group. The details, including the token's spelling, could change before it is finalized.