Zero-downtime deploys may hide double-executed requests, p99 latency reveals the truth

A software engineer tested rolling restarts on a two-replica Node/Express setup behind nginx and found that zero HTTP errors were reported even when one replica was killed mid-request, because nginx's proxy_next_upstream feature silently retried failed requests on the surviving replica. While this masked errors entirely, p99 latency nearly doubled — from roughly 3 seconds to nearly 6 seconds — because affected requests were executed twice without the client's knowledge. This silent retry is harmless for idempotent operations like search queries but can cause serious issues for non-idempotent ones such as payment authorisations or outbound emails. Adding a SIGTERM handler to drain in-flight requests before shutdown reduced failures significantly but did not eliminate them, as a race condition persists between the app closing and the load balancer removing it from the upstream pool. The findings highlight that error rate alone is an unreliable signal for deploy health, and p99 latency monitoring is essential to detect these hidden issues.