Kafka's Exactly-Once Semantics Does Not Cover External Systems, Engineers Warn

Kafka's exactly-once semantics (EOS), introduced in version 0.11.0, guarantees duplicate-free message delivery only within the Kafka cluster itself. Once a consumer writes to an external system — such as a database or REST API — that operation falls outside Kafka's transaction boundary, leaving room for duplicates. A common crash scenario, where a database write succeeds but the Kafka offset commit does not, causes the same message to be reprocessed and double-written. Engineers are advised to design idempotent consumers, use the transactional outbox pattern, or manually manage offsets alongside database writes to achieve true end-to-end safety. Enabling EOS also carries a measurable performance cost, with real-world pipelines reporting 15–20% lower throughput and higher latencies compared to at-least-once delivery.