Engineering Team Uses KEDA to Cut Kafka Idle Costs and Tame Burst Lag Spikes

A development team running Kafka consumer services faced two conflicting problems: wasted resources during long idle periods and slow spike absorption during nightly data bursts. Traditional Kubernetes Horizontal Pod Autoscalers proved ineffective because sink workloads are I/O-bound, meaning CPU stays flat even as consumer lag climbs. The team adopted KEDA (Kubernetes Event-Driven Autoscaling), which scales workloads based on Kafka consumer-group lag rather than CPU or memory metrics, and supports scaling all the way down to zero replicas. A ScaledObject configuration was used to wake idle consumers the moment lag appeared and cap replicas at the topic's partition count during peak load. All benchmarks were conducted in a reproducible local lab environment, with full source code and setup commands published on GitHub.