Eventually consistent DynamoDB index let mobile retries create 1,400 duplicate orders

At 14:07 UTC, PagerDuty alerted on a fulfillment queue running at 340% of baseline, and investigation showed many orders had been placed twice. Engineers first suspected the mobile app's retry logic, but client request logs confirmed each user tap produced a single request with a correctly reused idempotency key. Checks on API Gateway and Lambda also ruled out retries at those layers, since every invocation completed quickly with no duplicate requests. The root cause traced to the checkout service's idempotency check, which queried a DynamoDB Global Secondary Index that only supports eventually consistent reads and lagged during a propagation spike. Staging reproduction confirmed that closely spaced requests carrying the same key could each miss the earlier write and create separate orders.