Race Condition Bug Caused Duplicate Charges on Single Orders, Post-Mortem Reveals

A software engineering post-mortem identified a race condition as the cause of six customers being charged twice for a single order within a 48-hour period. Two near-simultaneous payment requests for the same order both read a 'pending' status from the database before either could update it, allowing both to proceed to charge the customer. The second request initially timed out but successfully retried, resulting in two separate successful charges against one order. The root cause was the absence of a database-level lock or atomic check to prevent concurrent requests from passing the status guard simultaneously. Refunds were issued manually while engineers traced the bug through application logs and database records.