Major AI firms agree to a shared standard letting publishers opt out of scraping

OpenAI, Google, Microsoft, and Anthropic have said they will adopt a new standard that lets publishers block AI scraping. Rather than configuring preferences separately for each company, publishers could apply one set of AI training choices across several providers. The proposed shared "policy token" signal was put forward by Kevin Kelley, a standards technical lead at Anthropic, through the Internet Engineering Task Force. The signal would be placed in a website's robots.txt file, which already indicates which crawlers may access a site for purposes such as search indexing or archiving.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in