API rate limiter shielded heavy abuser while blocking 112 innocent customers

A single shared token bucket rate limiter at one company's API edge caused 112 customers to face roughly 30% availability during a Tuesday early-morning incident, while the one customer running a large historical backfill retained most of its throughput. Because the global limit was not partitioned per API key, clients making the most requests naturally consumed the most tokens, penalising well-behaved users disproportionately. The engineering team overhauled the system by assigning each key its own bucket sized on its 30-day peak usage, adding request priority classes to favour interactive calls over batch jobs, and retaining the global bucket only as a backstop. Throttling is now tracked per individual key so engineers can identify exactly which tenant is being shed. Reporting was also updated to surface the worst-performing tenant's success rate alongside aggregate figures, since the aggregate metric had masked the severity of the outage for affected customers.