Clock Drift Broke a Redis Lock, Letting Two Workers Run the Same Billing Job

A nightly billing-reconciliation system designed to run once on one of three workers accidentally executed twice simultaneously, doubling entries for the same customer batch. The root cause was an NTP daemon that had silently stopped syncing after a container restart, leaving one worker's clock running roughly 9 seconds behind the Redis server's clock. Because Redis measures lock expiry using its own clock, the 30-second lock expired sooner than the worker expected, allowing a second worker to acquire it while the first was still running. The engineering team resolved the issue with three fixes: adding a background thread to auto-renew the lock, introducing monotonically increasing fencing tokens to reject stale writes, and adding explicit monitoring alerts on NTP synchronisation status. The incident highlights a fundamental risk in lease-based distributed locks — the lock server and client clocks must stay in close agreement, or the safety margin can silently disappear.