Retry queue design for e-commerce backlogs: admitting delayed jobs without overloading workers
When a large backlog of delayed e-commerce catalog jobs becomes due at once, releasing all of them could recreate the overload that caused the delay. The proposed approach stores long-horizon retry dates in a durable ledger, uses a periodic dispatcher to claim only as many eligible jobs as workers can absorb, and reserves the native queue delay for short waits. Each job keeps a stable ID so repeated offers are harmless, and completion is checked before costly dependencies are called. The author notes this design does not guarantee exactly-once execution across a database and queue, but makes duplicate delivery safe. Generated output must also pass the same schema and content checks as first-attempt output before counting as success.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in