Why treating Kafka like RabbitMQ can silently break your system at scale

A software team discovered critical failures after scaling their event pipeline from roughly 200 to 40,000 events per hour, tracing the root cause to a fundamental misunderstanding of how Apache Kafka works. Unlike traditional message queues such as RabbitMQ or Celery, Kafka is a distributed append-only log where messages are read rather than consumed, meaning they remain in the log after processing and do not disappear automatically. Each consumer group tracks its position via an offset, so a crashed consumer that never advances its offset will silently retry the same events indefinitely, causing duplicate side effects with no built-in dead-letter queue to catch failures. The team also encountered rebalance death spirals, where slow processing exceeded default timeout settings, causing Kafka to repeatedly kick consumers out of the group and halt consumption entirely, leading to mounting lag. The key lessons highlighted are to monitor consumer lag as a primary metric, handle offset commits and failure logic explicitly, and tune timeout and polling configurations to reflect real-world processing times rather than relying on defaults.