Connection Pool Exhaustion, Not Slow Queries, Was Behind Mysterious API Lag

A backend engineer recounts diagnosing a recurring production slowdown where API response times spiked from under 120ms to nearly 10 seconds every few hours, with no crashes or obvious errors in logs. Initial investigations into database queries, caching, and recent code commits yielded no answers, as the team was largely guessing rather than systematically tracing the problem. The breakthrough came from tracing a single slow request end-to-end, which revealed the application and database were both performing normally, pointing to an issue upstream of the application layer. The root cause turned out to be database connection pool exhaustion under certain traffic patterns, forcing new requests to queue for an available connection rather than execute a slow query. The incident highlights a broader lesson: slow systems are often waiting — for connections, locks, or resources — rather than performing expensive computation, making CPU and memory metrics alone unreliable indicators of latency problems.