Three Production Race Conditions That Revealed the Limits of Distributed Locks

A backend engineer encountered three distinct race conditions while building a system where synchronous APIs, async webhooks, and background workers concurrently read and wrote the same MongoDB documents. The first issue arose when a Redis lock expired before a slow API transaction completed, causing a WriteConflict that was resolved by aborting the stale transaction and re-reading the document before writing. A second problem emerged when workers locked on child record IDs but inadvertently collided while writing to shared parent-level state, requiring the lock scope to be elevated to the parent ID. The engineer concluded that lock scope must account for all resources a transaction touches, not just the immediate record being processed. Key takeaways include preferring async queues where real-time processing is unnecessary, scoping locks from a system-wide perspective, and explicitly documenting any trade-offs or residual edge cases accepted in each fix.