Seven-Month Uptime Exposed a Memory Leak Hidden by Frequent Deployments

An internal reconciliation service crashed repeatedly at 4 a.m. on a Sunday after its pods were killed due to memory exhaustion, revealing a slow leak of roughly 40 MB per pod per day. The root cause was a bug introduced in 2019: an SDK client created per request registered listeners on a static registry that were never unregistered, accumulating 1.2 million listener objects over seven months. The same faulty library was found running across every service in the fleet, but the leak had gone undetected because frequent deployments kept restarting pods before memory limits were reached. Following the incident, the team introduced age-normalised memory growth alerts, a 30-day soak environment for each service, and a pod-age distribution dashboard to surface long-running behaviour. The episode highlighted how high deployment frequency had inadvertently become a reliability mechanism, masking underlying code defects rather than resolving them.