The Reconciler Pattern: Fixing Silent Queue Job Failures That Never Show Up in Logs

When a queued job is dispatched only once at row-creation time, it can silently disappear due to worker downtime, redeployments, or queue clears — leaving database rows permanently stuck in a pending state with no error or failed-job entry. Unlike standard job retries, which only apply to jobs that actually ran, these lost dispatches are invisible to queue-level monitoring. The reconciler pattern addresses this by running a scheduled database sweep — typically every 15 minutes — that identifies stale, unclaimed pending rows and re-dispatches their associated jobs. Key implementation safeguards include touching the row before dispatch to prevent duplicate re-queuing, applying a batch limit to avoid flooding the queue after prolonged outages, and ensuring the job handler is idempotent so accidental re-runs are harmless no-ops. The core principle is that the database is the durable source of truth, while the queue is merely a delivery mechanism — and domain-level reconciliation must complement, not replace, queue-level retries.