__usage_ai-training is a shared robots.txt token that tells AI companies not to use your pages to train their models. On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they would all honor it, which means one rule can replace a growing list of per-company bot names. You add it as a User-Agent group in robots.txt and use Disallow lines to say what is off limits for training.
This tutorial covers what the token does and does not do, three ready-to-paste examples, how to test it, and how to decide whether you want it at all.
Where does the token come from?
The token is defined in an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, published October 30, 2026 by Kevin Kelley of Anthropic. Like all Internet-Drafts it has an expiry date, May 3, 2027, by which time it will be revised, adopted or dropped. The four companies’ commitment is reported by Dev.to, Business Standard, Emergent and CCLeaks.
Rollout is not simultaneous:
| Company | Honors the token from |
|---|---|
| Anthropic | Reportedly at announcement, October 30, 2026 |
| Microsoft | February 1, 2027 |
| OpenAI | “In the coming months” |
| Supported, but interpretation disputed across reports |
The Google row needs care. Some reports say Google treats the token like a Google-Extended disallow, which affects training only. Others say it may also disallow Gemini grounding, which would affect whether Gemini cites you in answers. Check Google’s own documentation before relying on either reading.
Training versus search and answers: two different decisions
The most common mistake publishers make is treating “AI” as one thing. It is at least two.
Training means your pages become part of the data a model learns from. Once trained, the model may reproduce your facts, style or structure without ever visiting your site again. The new token targets this.
Crawling for answers means a bot fetches your page at query time so an assistant can summarize it and link to it. This is closer to search indexing. It can send traffic. It uses different bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) and different rules.
You can say no to training and yes to answers. That is a reasonable position for a publisher who wants to be cited but not absorbed, and the examples below show how.
Example 1: opt out of training entirely
# Opt out of AI training for the whole site
User-Agent: __usage_ai-training
Disallow: /This is the simplest form. Every participating company, once its rollout is live, should stop using your pages for training. It says nothing about search or answer crawling, so existing per-bot rules still apply.
Example 2: opt out only a paid section
# Allow training on free content, block it on paid content
User-Agent: __usage_ai-training
Disallow: /premium/
Disallow: /members/Path-level rules work the same way they do for any crawler. Free articles remain available for training; the content people pay for does not.
Example 3: allow training, but keep per-bot rules so answer engines can cite you
# No training restriction (token omitted or explicitly allowed)
User-Agent: __usage_ai-training
Allow: /
# Per-bot rules for answer and search crawlers
User-Agent: GPTBot
Allow: /
User-Agent: ClaudeBot
Allow: /
User-Agent: PerplexityBot
Allow: /
Disallow: /drafts/
# Everyone else
User-Agent: *
Allow: /
Disallow: /admin/This is close to what Agentic Weekly HQ runs today. We currently allow AI crawlers so assistants can cite us, because a new publication benefits from being found. That is a choice, not a recommendation; each publisher weighs it differently, and we may revisit it.
How do the token and per-bot rules interact?
They coexist. The token is just another User-Agent value, so a crawler that supports it checks two groups: the one matching its own bot name and the one matching __usage_ai-training. The intent is that the training token governs training use and the bot-name group governs fetching.
Two practical consequences:
- Keep existing
GPTBot,ClaudeBot,Google-Extendedand similar rules in place. Companies that have not finished rolling out the token still read them, and they cover uses the token does not. - The draft does not fully specify what happens when a per-bot rule and the token disagree. Avoid contradictions rather than relying on precedence.
How do I test it?
- Fetch your file directly:
curl -s https://example.com/robots.txtand read it with your own eyes. A surprising number of sites serve a stale cached copy or a 404. - Check the token character by character. It is two underscores, the word
usage, one underscore, thenai-trainingwith a hyphen. Treat it as case-sensitive as written in the draft; do not capitalize it. - Confirm the
User-Agentline and itsDisalloworAllowlines are in one contiguous block with no blank line between them. A blank line ends a group. - Run the file through a robots.txt validator, such as the one in Google Search Console, to catch syntax errors.
- Watch your server logs over the following weeks. A drop in training-bot fetches is the only confirmation you will get.
What the token cannot do
- It does not remove anything already in a dataset or already trained into model weights.
- It does not stop AI answers from citing your site. That is governed by other bots and other rules.
- It does not bind companies that have not committed to it, or crawlers that ignore robots.txt altogether.
- Google and Microsoft have both warned that opting out of AI uses can reduce how often a site appears in their AI experiences. Read that as a tradeoff, not a threat, but read it.
Decision guide for a publisher who wants citations but not training
Ask three questions.
Do you earn from traffic? If yes, you want answer engines to fetch and link you. Keep per-bot Allow rules for search and answer crawlers.
Is your content distinctive enough that training on it costs you? Original reporting, proprietary data and paid analysis usually qualify. Generic how-to content often does not. If yes, add the token with Disallow: / or scope it to the valuable paths as in Example 2.
Can you tolerate ambiguity on Google? Until Google’s interpretation is clear, adding the token might reduce Gemini grounding. If Google traffic matters most to you, wait for Google’s documentation, or test on a low-traffic section first.
A typical answer is a full training opt-out plus explicit per-bot allows for the answer crawlers you want. Recheck in February 2027 when Microsoft’s support goes live and the draft is likely to have a revision.
Our take
A single shared token is a real improvement over tracking a dozen bot names, and four competing companies agreeing on one string is itself notable. But it is a preference signal, not a lock. The limits on removal, citation and Google’s handling make it one tool in a policy, not the policy.
If you run agents that crawl, respect the token now, even before your competitors do. For more on the agent side, see what an AI agent is and our tools directory. If you publish and want to follow how rollout plays out, the newsletter will cover it.
Questions, answered
How do I block AI training in robots.txt?
Add a group with User-Agent: __usage_ai-training followed by Disallow: / to your robots.txt. OpenAI, Google, Microsoft and Anthropic have said they will honor this shared token, though rollout dates differ by company.
Does the AI training opt-out token stop ChatGPT or Gemini from citing my site?
No. The token covers training only. Answer engines that crawl for search or grounding use separate bots, and Google's handling of the token is disputed across reports, so check Google's own documentation.
Does opting out remove my content from existing AI models?
No. The token applies to future crawls. It does not remove data already collected into datasets or already trained into model weights.
