How a Race Condition and Non-Idempotent Workers Silently Broke a Production System

A software team discovered that background jobs were getting stuck in a permanent 'processing' state after deploying their system to production, with roughly one in several hundred jobs never completing. Investigation revealed a race condition where the job queue worker would pick up tasks before the database transaction had fully committed, exhausting retries and leaving records in limbo. After fixing the commit-then-publish order, duplicate job processing emerged, which was resolved by making workers idempotent — checking for existing results before acting. A later database migration adding a non-null column caused table locks on the larger production dataset, triggering cascading latency and health check failures that staging had not caught. The incidents collectively highlighted how distributed systems require explicit coordination between components, and that staging environments rarely replicate the scale and timing pressures of real production traffic.