How a Content Pipeline's Batch Runner Silently Misreported Completion for Months

A batch runner built for a content pipeline on 16 March 2026 included resumability features from day one, yet the system remained functionally broken for three more months. The core problem involved four distinct bugs that each caused the runner to falsely believe work was already finished, including a quota error that returned an empty result instead of raising an exception, leading the pipeline to record missing content as successfully produced. Three separate parts of the stack independently handled rate-limit errors using different string-matching logic, meaning the same error could be caught or missed depending on where it surfaced. The team also had to manage stale 'running' states left behind by killed runs, as well as separate ownership of entity state and lock files — a distinction discovered only after a failure in production. The fixes for all four bugs landed in a single day, underscoring how resumability in batch pipelines differs fundamentally from retry logic and requires explicit state design.