Engineer Explains How a Settlement Engine Handles Failures It Cannot Control

The author argues that some events, like a gateway timeout or a database deadlock, arrive regardless of what a system wants, so the only real control lies in the state the system records next. Their settlement engine marks a timed-out payment as UNKNOWN rather than CONFIRMED or FAILED, because leaving it PENDING would block all future retries under the same idempotency key. A load test later exposed a flaw where retries ran out with no fallback, leaving settlements stuck forever. The fix adds a second retry budget that writes UNKNOWN, a smaller but truthful claim that later reconciliation can resolve. The author draws a parallel between this design choice and their own habit of recording what is true and leaving the rest marked as unknown.