Race condition in billing webhooks randomly wiped customer data on failed payments
A software engineer discovered a bug where two webhooks triggered by the same failed payment produced opposite outcomes depending on which arrived first. The root cause was a single payment status string appearing in both an access-control list and a destructive-cleanup list, causing cleanup logic to sometimes erase scheduled work and revoke API keys. Because webhook delivery order is not guaranteed, the bug was non-deterministic and never reproduced consistently enough to be investigated. A passing test suite masked the problem by asserting the incorrect behavior as expected. The engineer concluded that destructive operations should only trigger at terminal subscription states, not during retry grace periods, and that handlers relying on prior record state must account for unpredictable event ordering.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in