Google, OpenAI, Microsoft and Anthropic Agree on One Way to Say "Don't Train on My Site"

If you run a website and don't want your work feeding AI models, the current setup is a mess. Every company has its own crawler name, so you end up writing separate rules for each one and hoping you didn't miss anybody. That might finally get simpler.

On Friday, according to TNW, Anthropic's Kevin Kelley published a proposal at the IETF, the group that sets many internet standards, for a single opt-out that lives in your robots.txt file. It uses a new token, "__usage_ai-training," and lets you list which parts of your site you don't want used for training. The example in the draft blocks a whole site except for its press kit. The same day, Google, OpenAI and Microsoft said they'd support it.

Don't expect it overnight. OpenAI is aiming for the coming months. Anthropic says adding it across its services will take some time. Google is targeting early 2027, and Microsoft says it will honor the signal starting February 1, 2027. Meta's spokesperson said he liked the idea of a common signal but didn't commit to following it.

There are some important limits. This is a preference about training, not a block on crawling. If you don't add the rule, you haven't said anything, and older catch-all rules don't count. Private deals between a publisher and an AI company can override it, and Microsoft is clear it doesn't create any licensing or payment arrangement. It's also just an individual draft for now, not an official IETF standard. Staff from the Financial Times, the BBC and the European Publishers Council helped shape it.

It's a real step for creators, but it doesn't touch the bigger fight over getting paid. Cloudflare, for example, has already started telling AI bots to pay up or get blocked.