Designing Kafka Consumers on EKS for Frequent Pod Restarts and Rebalances

Kafka consumers are stateful clients whose group membership must remain stable, but Kubernetes routinely restarts, throttles, and reschedules pods, which the group protocol treats as membership changes. The author argues most consumer incidents on EKS stem from this clash rather than from Kafka defects. A baseline configuration with a generous session timeout, a poll interval with headroom, manual offset commits, and cooperative-sticky assignment prevents many problems outright. The post then examines eight recurring failure modes, each explained by its symptom, underlying mechanism, and usually a simple config change or lifecycle hook. The overarching guidance is to treat restarts as a permanent condition to design around, using stable identities and a graceful shutdown path.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in