Why rebuilding Kafka from scratch closes the gap between using and understanding it

Apache Kafka is widely used for data streaming, but most engineers rely on its abstractions without understanding the internals that govern its behavior. Common production issues — such as consumer group stalls, rebalance-induced latency spikes, and unexpected disk usage — are direct consequences of how Kafka's log structure actually works. Rebuilding Kafka from scratch, even as a toy implementation, forces developers to make design decisions the official documentation never surfaces, such as how segment files are managed and what 'committed' truly means under different acknowledgment settings. Key concepts like offset tracking, the consumer fetch loop, and partition-count constraints become intuitive only when implemented hands-on rather than read about. The exercise is not about replacing production Kafka but about eliminating the mental model debt that accumulates when engineers cannot identify which layer is responsible during an incident.