Public APIs outpaced bot-protected marketplaces: a scraper team's hard lesson

A web scraping team spent an entire workday last week failing to publish a single listing after nine scrapers targeting consumer marketplaces and job boards were blocked by commercial anti-bot systems, despite using both datacenter and residential proxies. The same afternoon, the team switched targets to keyless public APIs — including OpenAlex, Crossref, and openFDA — and successfully built and published three scrapers in roughly an hour each on the first attempt. The core finding was that throughput was limited not by engineering skill or tooling, but by the anti-bot defenses of the target platforms. Public research and regulatory APIs carry no incentive to fingerprint or block scrapers, eliminating an entire category of failure common in marketplace scraping. The team has since restructured its backlog to prioritize targets by type, ensuring walled platforms only enter a build cycle when a tested plan for bypassing their defenses is already in place.