Hybrid Outbox Pattern Uses In-Memory Queue for Speed, PostgreSQL for Recovery

A developer has proposed a hybrid implementation of the Transactional Outbox Pattern that combines an in-memory queue with PostgreSQL to balance low latency and durability in distributed systems. In the standard approach, a worker polls a PostgreSQL outbox table at regular intervals to forward events to Kafka, but this creates a trade-off between polling frequency, database load, and delivery latency. The new design commits only event IDs to an in-memory queue immediately after a database transaction, while PostgreSQL retains the full event payload as the durable source of truth. A Batch Publisher reads these IDs, fetches the corresponding events from PostgreSQL, and forwards them to Kafka, keeping the fast path lean. Database polling is not eliminated but is repurposed as a background recovery mechanism that detects unprocessed events and reintroduces their IDs into the same memory queue pipeline.