What rebuilding Kafka from scratch reveals about its core internals

·1 views

A software engineer shares insights gained from following a ground-up reimplementation of Apache Kafka to deepen understanding of how the system works internally. Key revelations include how producer-side partition logic determines message routing before the broker is ever involved, directly affecting ordering guarantees in event streams. The exercise also clarified how Kafka's storage layer uses append-only segment files paired with index files to enable fast offset-based reads via binary search. Rebuilding the consumer group rebalance protocol exposed why session timeouts and heartbeat intervals are critical for detecting failed consumers in a distributed setting. Similarly, reconstructing leader-follower replication made clear how the in-sync replicas list dynamically affects durability guarantees when followers fall behind.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer manually compiled 100 SaaS pricing pages to find real market patterns

A developer grew frustrated with gut-feel pricing and the common habit of simply undercutting one competitor, so they manually reviewed around 100 public SaaS pricing pages across categories like analytics, email tools, and form builders. Rather than copying exact numbers, they focused on structural patterns such as where free tiers cut off, what mid-tier plans actually sell, and which pricing units — seats, usage, or projects — different products use. Key recurring patterns included annual discounts of roughly 15–20%, a heavy clustering of indie entry prices in the $9–25 range, and free tiers designed around one core job rather than time-limited full-feature trials. The researcher packaged all findings into an Excel workbook with a patterns tab and a blank experiments sheet, offering a free 12-row sample and a full ~100-company version for $19. The project was shared on DEV Community to spark discussion about how founders actually conduct pricing research when launching a product.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why AI Development Is Shifting From Prompt Engineering to Full AI Engineering

Building AI features once meant iterating on prompts until outputs were good enough, but production software demands far more than that. Real-world AI systems must handle questions around data access, tool selection, failure recovery, and how much autonomy to grant a probabilistic system. A well-crafted prompt alone cannot address missing context, permission constraints, invalid business logic in valid outputs, or the need for state and validation. This has driven a broader shift toward AI engineering — a discipline drawing from software engineering, MLOps, LLMOps, security, and distributed systems. The AI model is now just one component within a larger, carefully engineered system rather than the feature itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Set Up AWS IAM Credentials and Infrastructure Before Using Terraform

Before running any Terraform commands against a real AWS account, developers need a dedicated IAM user with programmatic access rather than using the root account, which carries significant security risks. The IAM user should be granted AdministratorAccess for testing purposes, though a least-privilege policy is recommended for production environments. AWS credentials must be exported as environment variables — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION — and should never be stored inside Terraform configuration or variable files. The planned infrastructure includes a VPC spanning two AWS Availability Zones, with public and private subnets, a NAT Gateway, an EKS control plane, and EC2 worker nodes running application pods. A Network Load Balancer handles incoming public traffic, while the NAT Gateway manages outbound-only access from private subnets to the internet.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why database indexes can fail: the B-tree mechanics every developer should know

A database index is a sorted B-tree structure that maps a specific column to row pointers, allowing queries to find data in just a few hops instead of scanning every row in a table. Without an index, a query on a million-row table performs a full table scan, checking each row individually — a process that scales linearly with table size. While indexes can cut query times from seconds to milliseconds, they come at a cost: every insert, update, or delete requires the index tree to be updated, slowing down writes. Composite indexes follow a strict left-to-right rule, meaning a two-column index on last name and first name cannot speed up searches by first name alone. Common pitfalls like leading wildcards in LIKE queries or wrapping indexed columns in functions also prevent the database engine from using the index at all.

0 comments Read more at DEV Community