Single faulty server caused hours of checkout failures while going undetected

A degraded network interface on one of twenty-four server instances caused checkout error rates between 3–8% for most of a Thursday, going unnoticed because fleet-wide monitoring diluted the signal. The instance continued passing health checks by successfully responding to lightweight synthetic requests, even as roughly one in ten real connections stalled and timed out. It took three hours to isolate the culprit, despite the cloud provider having flagged a hardware degradation notice ninety minutes before the first customer complaint. In response, the team introduced per-instance error rate alerts, passive load-balancer health checks based on real traffic, and a pipeline for hardware health events. The key operational change was to remove a suspected instance immediately and investigate afterward, rather than spending time confirming the obvious.