Pay-to-Crawl Proposals Emerge to Tackle AI Bots, But Critics See Flaws

The rapid growth of AI training has intensified web scraping by bots, imposing infrastructure and electricity costs on website owners who receive no benefit from non-human traffic. In response, several companies and non-profits are exploring pay-to-crawl frameworks that would require AI bots to pay for website access. Critics argue these voluntary licensing systems are legally weak, since US courts have repeatedly ruled AI training qualifies as fair use, limiting their practical enforceability. Some proposals, including Cloudflare's bot-exclusion and payment experiments, raise concerns about concentrating too much control over internet access in a single company. One alternative view suggests charging a small micropayment from all users — human or bot — to cover hosting costs, potentially reviving the original vision of a micropayment-based internet over the ad-driven model.