Why Kafka's Exactly-Once Delivery Is Rarely Worth the Complexity

Many engineering teams configure Kafka's exactly-once delivery guarantee believing it prevents duplicate message processing, but the feature carries significant performance and complexity costs. The setting relies on transactional producers and consumers working in coordination, adding latency and throughput overhead on every message. Critically, Kafka's exactly-once guarantee only applies within Kafka itself — any external action such as a payment charge or database write remains unprotected. This means teams must still implement their own deduplication logic for real-world side effects, negating the primary reason for adopting the feature. A simpler and more practical approach is to design consumers to be idempotent from the start, handling duplicates at the application layer rather than the transport layer.