Misconfigured Retry Loop Turned a Single API Rejection Into a 2 AM Outage

A background summarization worker began failing at 1:47 AM, with roughly one in six API requests returning HTTP 429 errors that initially appeared random. The root cause was not a rate limit but a token-count mismatch: the client-side tokenizer estimated 8,400 tokens while the server counted 10,300, triggering a contract violation. Because the retry logic treated this rejection as a transient error, it re-sent the same oversized payload eleven times, and synchronized backoff caused all workers to hammer the endpoint simultaneously. The fix involved parsing the server's response body to capture the actual rejection reason, enforcing a client-side token cap before requests reached the network, and replacing synchronized backoff with a jittered, three-attempt retry limit. The incident highlights how passing all pre-production evaluations can create false confidence, masking failure modes that only surface under real production conditions.