AI giants promise publishers new way to block some scraping

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Credit: Unsplash/CC0 Public Domain OpenAI, Google, Microsoft and Anthropic pledged on Friday to give media outlets and other websites a single way to prevent their content from being scraped to train artificial intelligence (AI) models, amid growing criticisms and legal battles from news publishers.

Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data, but the new proposed solution will allow them to enter a single, standardized instruction that's recognized and honored by any AI company that adopts the protocol.

The solution will not fully take effect for months, though.

Microsoft will implement it beginning Feb. 1, 2027, and Google sometime in 2027. ChatGPT maker OpenAI said it will roll it out "in the coming months," while Claude maker Anthropic said it has "already started" rolling it out.

The announcement comes as publishers are facing a surge in bots that scrape their articles to feed AI models. Some, like The New York Times, have taken the matter to court.

In December 2023, the Times sued OpenAI and Microsoft for allegedly using millions of its articles without authorization or payment to train ChatGPT. The two tech companies are invoking an exception to copyright law known as fair use.

Meanwhile, internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a standard called Really Simple Licensing already allows publishers to charge bots that visit their pages.

The new setting will apply only to future training of models but does not prevent bots from crawling otherwise, and content scraping will be allowed by default until the setting is changed by a publisher, Microsoft said in a blog post.

Opting out of AI training could reduce the chances that a publisher's content will surface in AI apps and experiences, Microsoft and Google said in separate blog posts.

"We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry," Meta spokesperson Andy Stone wrote on X, conveying his support for the effort.

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →