What the new AI-training opt-out will actually do
A shared preference could simplify publishers’ choices. Rollout is still ahead, and training, crawling and search need different controls.
Website owners may soon have a simpler way to tell several AI companies not to use their work for training. On October 30, OpenAI and Google announced support for a shared preference in robots.txt, the file websites use to communicate rules to automated visitors. The Verge reports that Microsoft and Anthropic also support the initiative.
The useful change is coordination. Publishers currently maintain different controls for different providers. A common signal could reduce that work. But support announcements, implementation dates and the scope of the preference are three separate things.
OpenAI says it aims to implement the signal in the coming months and will retain its existing crawler controls. Google targets 2027. Neither announcement means that adding the proposed signal today makes every provider honor it.
A preference with a specific scope
The underlying proposal, written by Anthropic’s Kevin Kelley, is an individual Internet-Draft. The IETF’s own status page explicitly says it is not endorsed by the organization and has no formal standing in its standards process. Calling it a completed internet standard would overstate what has happened.
The draft introduces the policy token __usage_ai-training. Its definition concerns using content to modify a generative model’s learned parameters. It separates that preference from crawler traffic rules: the token itself is not a command to stop visiting a website. A general wildcard group does not count as expressing this particular preference, and provider-specific controls or private agreements can affect how preferences are interpreted.
Google adds a practical wrinkle. Where Google-Extended is not explicitly specified, it plans to interpret the new opt-out as disallowing Google-Extended. Publishers should read the provider’s documentation about grounding in Gemini as well as training before deciding what they want.
What website owners can do now
Keep existing provider-specific controls while checking implementation updates. OpenAI’s current documentation distinguishes GPTBot, used for content that may train foundation models, from OAI-SearchBot, used for ChatGPT search. A website can disallow the former while allowing the latter. ChatGPT-User handles user-initiated visits separately, and OpenAI says robots.txt rules may not apply to those requests.
Before changing a site, write down the intended outcome: limiting training, limiting automated retrieval, or remaining discoverable in search. Review the corresponding controls, preserve a copy of the existing configuration, and check the deployed file. Do not assume a single switch covers all three goals. We have not changed this publication’s crawler policy as part of this report.
Our view: a shared vocabulary is worthwhile if it makes publishers’ choices easier to express and audit. The next evidence to look for is implementation, clear precedence rules and demonstrated compliance. The announcement alone cannot establish that content use is under its creator’s control.