Website owners may soon be able to block AI training across several AI companies with a single robots.txt rule. OpenAI and Google said on 30 October that they plan to honor a shared “policy token” for AI training, based on work from the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF). Neither company supports it yet: OpenAI says it will implement the signal “in the coming months,” and Google says it aims to support it alongside other AI providers in 2027.
Why a shared way to block AI training matters
Today, a site owner who wants to keep content out of AI training usually has to add separate rules for each company’s crawler or control, such as Google-Extended, which Google has offered since 2023. Google says the new approach replaces that patchwork with a single step that participating AI providers can read.
According to Google, the proposal reflects feedback from publishers, civil society groups, regulators and other stakeholders who have asked for industry-wide controls. It is also consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act, which recognizes robots.txt as a primary way for publishers to express machine-readable preferences.
How the robots.txt signal works
The technical details are set out in an IETF Internet-Draft, “draft-kelley-aipref-training-preference-signal-00,” written by K. Kelley of Anthropic and dated 30 October. It defines a reserved policy token, __usage_ai-training, that is used in place of a crawler name in a robots.txt User-Agent line.
The draft gives this example, which opts a whole site out of AI training except for a press kit folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/A few rules from the draft are worth knowing:
- No new syntax: the token uses existing robots.txt grammar and adds no new directives.
- The wildcard does not count: a
User-Agent: *group must not be treated as matching the token, so existing catch-all rules do not automatically express a training preference. - Silence means no preference: if a site has no group for the token, it has not expressed a preference either way.
- It is not a crawl rule: the token states a usage preference for AI training. Crawling rules for individual bots stay independent.
- Specific rules can win: a company-specific token may take precedence, and private agreements can override the signal.
The draft uses the IETF definition of AI training: modifying the learned parameters of a generative AI model. Models used only for non-generative tasks are excluded.
How OpenAI and Google will block AI training requests
OpenAI says it will honor the common signal for training its foundational generative AI models. It describes the effort as voluntary, says it will not replace existing crawler controls and encourages other AI providers to honor the signal as well.
Google says that if a site uses the policy token to opt out and does not specifically mention Google-Extended, it plans to treat that as if the site had disallowed Google-Extended. Company-specific controls will keep working alongside the shared signal. Google also notes that sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
For site owners, my takeaway is that there is nothing to switch on today, but the existing per-company rules are still the ones that work now. For more on search and site controls, see our guides on how Google Search works, mastering technical SEO and keeping Claude chats and Google files private.
FAQ
How do I block AI training with robots.txt?
Under the IETF draft, you add a group for the __usage_ai-training token with Disallow rules. OpenAI and Google have not implemented it yet, so current company-specific controls still apply.
Does blocking AI training stop Google from indexing my site?
The draft says the token is not a crawling directive, and Google says it will treat an opt-out as disallowing Google-Extended, its AI training control.
When will the shared AI training signal work?
OpenAI says it will implement it in the coming months. Google says it aims to support it with other AI providers in 2027.