How separating queues and workers fixed silent failures in a headless AI server setup

A developer running always-on AI agents discovered their system appeared healthy while webhook jobs silently piled up unprocessed, with Docker reporting containers as live despite workers being effectively frozen. The root cause was mixing webhook intake, cron scheduling, and heavy LLM execution inside a single process — a setup that works locally but breaks under production load. The fix involved splitting ingress from execution using a Redis queue, routing all LLM calls through a single OpenAI-compatible endpoint, and replacing local storage with PostgreSQL and S3-compatible solutions. A critical additional requirement was making all jobs idempotent, since queue systems like Celery can redeliver tasks if a worker crashes before acknowledging completion, risking duplicate side effects such as double charges or repeated emails. Stripe's webhook system, which retries undelivered events for up to three days, was cited as a real-world example of why systems must track whether a given event has already been processed.