The companies backed a shared signal that would let publishers tell participating AI providers not to use their content for model training. *(Reported Nov. 10; announcement was Oct. 31)
On Oct. 30, OpenAI, Google, Microsoft and Anthropic backed a shared robots.txt signal letting publishers tell participating providers not to use specified pages for future AI training. The proposed IETF token, `__usage_ai-training`, can be paired with `Disallow: /` to exclude an entire site or narrower path rules to preserve selected material, such as a press kit. It concerns training use, not whether crawlers may fetch pages for search or other purposes; the preference is voluntary and does not undo use of content already collected. Rollout is staggered: Anthropic says it has begun implementation; OpenAI aims to add support âin the coming monthsâ; Microsoft will honor the signal from Feb. 1, 2027; Google targets 2027. A common instruction could spare publishersâespecially news and other licensing-dependent sitesâmaintaining separate controls for each crawler, but it works only as providers adopt and honor it. The proposal was drafted by Anthropic standards lead Kevin Kelley and submitted to the IETF.