How silent failures in an async video pipeline were diagnosed and fixed

A developer running a small image-to-video service found that some paid jobs never reached users, with no error logged and no refund requested. Initially the system tracked only pending and done states, so jobs whose provider webhooks never arrived stayed pending indefinitely. The fix involved persisting five explicit job states before calling the rendering provider, so crashes could be recovered by a sweeper. Timeouts were reclassified as unknown outcomes that trigger a status check rather than a failure, and the timeout threshold was set from measured render-time data. The article also describes a weekend incident where a webhook endpoint change left dozens of finished renders unrecorded, which exposed further gaps in the pipeline.