Silent retries masked a degrading pipeline until it finally broke

A software team discovered that their automated agent pipeline had been silently deteriorating for months, even as every run continued to report a passing status. The root cause was that successful retries were indistinguishable from first-attempt successes in summary logs, effectively hiding a growing failure rate. Over roughly 90 days, the share of runs requiring a second attempt climbed from zero to nearly 50 percent before the system finally produced a visible failure. The team's fix was to log attempt counts as a separate metric alongside pass/fail outcomes, enabling trend analysis rather than relying solely on binary alerts. The episode highlighted a broader instrumentation gap: systems built only to answer 'did it work' lack the vocabulary to warn that a failure is quietly being assembled.