Major Publishers Block AI Crawlers, Pushing for Licensing and Content Control

Leading news organizations, including The Guardian, The New York Times, CNN, and ABC, have increasingly blocked AI crawlers from accessing their websites since mid-2023, signaling a broader shift in how publishers manage digital content rights. The Guardian was among the first high-profile cases, blocking OpenAI's GPTBot in September 2023, while The New York Times updated its terms of service in August 2023 to restrict AI training on its content and later pursued legal action. Reuters Institute research from that period confirmed that a significant share of top publishers had restricted access for AI crawlers from OpenAI and Google. Publishers are not uniformly rejecting AI, however — some are simultaneously engaging in licensing negotiations, with The Guardian exploring licensing deals and The New York Times reportedly in discussions with Amazon as recently as 2025. The trend reflects a deliberate move by publishers to establish formal terms for AI access to their content, rather than allowing automated collection to continue by default.