Cloudflare's AI Crawler Payments Signal a Shift From Attention to Data Monetization

Cloudflare recently introduced a feature allowing websites to charge AI crawlers for accessing their content, a move that signals a deeper structural shift in how the internet generates revenue. For two decades, websites monetized human attention by trading free content access to search engines in exchange for visitor traffic, which funded ads, subscriptions, and affiliate links. AI agents have disrupted this cycle by extracting answers directly for users, bypassing the website visit entirely and eliminating those revenue streams for publishers. Unlike humans who resist micro-payments, AI agents can automatically pay fractions of a cent per request using the HTTP 402 status code, making micropayments viable for the first time. This dynamic could transform datasets, APIs, research archives, and product catalogs into metered, transaction-based assets, potentially spawning an entirely new layer of internet infrastructure built around machine-readable, paid resources.