Audit of 18 Web Scrapers Uncovers Three Common HTTP Retry Logic Bugs

A code audit across a fleet of 18 Apify web-scraping actors revealed three recurring bugs in HTTP retry logic. The most critical flaw excluded HTTP 429 (Too Many Requests) responses from retry handling entirely, meaning rate-limit errors were never retried when they should have been. Two other actors did the opposite, pointlessly retrying permanent errors like 401 and 404 responses that can never succeed regardless of how many attempts are made. A third widespread issue involved ignoring the server-provided Retry-After header, causing scrapers to wait arbitrary durations instead of the server-specified delay. None of the fixes have been fully rolled out across all 18 actors yet, highlighting how retry-logic bugs can persist silently in production systems.