The biggest names in AI are backing a new way to opt out of scraping
Summary
- Major AI companies including OpenAI, Google, Microsoft, and Anthropic are backing a new standardized way for publishers to opt out of AI scraping.
- Proposed by Anthropic technical lead Kevin Kelley via the IETF, the shared policy token functions within a website's robots.txt file.
- Publishers can use this single signal to apply AI training preferences across multiple AI crawlers instead of configuring them individually.
- While Anthropic has adopted the signal, OpenAI plans implementation in the coming months, and Google and Microsoft plan support next year.
Why it matters
This new standardized opt-out mechanism simplifies how publishers manage web scraping preferences across major AI developers, replacing fragmented, individual configurations with a unified policy token.