Why Per-Attempt Timeouts Break Retry Loops and How to Fix Them

A retry loop with a 10-second timeout and three retries can run far longer than a caller expects, because per-attempt timeouts do not limit the total operation time. Backoff delays and pre-request work such as DNS lookups or token refreshes add further latency on top of the attempts themselves. The article recommends starting from an overall time budget and dividing it across attempts, using a monotonic clock to track remaining time. It also suggests propagating a shared deadline through the call stack, as gRPC does, so downstream calls draw from one budget. Tradeoffs to weigh include starved final calls, budget sharing in parallel requests, and server-side work that continues after a client-side timeout, which makes idempotent retried writes important.