Developer stress-tests home Kubernetes cluster with chaos engineering, documents failures

·1 views

A software engineer running a 4-node bare-metal Kubernetes cluster on Talos Linux deliberately introduced failures using Chaos Mesh to test real-world resilience. The homelab setup included a Dell OptiPlex control plane, three Raspberry Pi worker nodes, and tools such as ArgoCD, Longhorn, Cilium, Prometheus, and Grafana — all built on roughly $220 of hardware. Despite appearing stable on paper, with etcd snapshots every six hours and replicated storage volumes, the engineer had never conducted a live failure test. Using Chaos Mesh, a CNCF sandbox tool that injects real failures into Kubernetes clusters, pod-kill experiments were run against a production namespace hosting a PostgreSQL database and several APIs. Initial results showed that while StatefulSets and Deployments recovered within seconds, the exercise was designed to surface deeper weaknesses around storage failover, network partitioning, and actual mean time to recovery.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why AWS S3 Egress Fees Can Turn a $2 Storage Bill Into $180+

Cloud storage on platforms like AWS S3 appears inexpensive at roughly $0.023 per GB per month, but a separate charge called egress — applied every time data leaves the provider's network — can dwarf storage costs. At approximately $0.09 per GB, egress fees accumulate rapidly because they are triggered by every user download, CDN cache miss, cross-region replication, backup export, and even failed transfer retry. A single popular file served to millions of users can generate terabytes of billable outbound traffic from one asset alone. Unlike storage costs, egress is not capped by how much data you hold but by how much you serve, making it difficult to predict from a pricing page. Developers and businesses are advised to model both storage and egress costs separately before choosing a provider, as the latter is often the dominant line item in real-world bills.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

LighthouseReckoning Brings Lightweight LoRa Mesh Networking to Arduino, ESP32 and RP2040

A developer has released LighthouseReckoning, an open-source LoRa mesh networking library designed for Arduino-compatible microcontrollers, currently tested on ESP32 and RP2040 with SX126x radios. The library enables multi-hop sensor data routing toward a single Home node without requiring manual route configuration or full network topology awareness. It uses a distance-vector approach where each node tracks neighbors and their hop count to Home, selecting the best next-hop dynamically. Reliability is handled through hop-by-hop confirmations and local retries, rather than a single end-to-end acknowledgment, reducing the coordination burden on the Home node. The full source code, protocol documentation, and field-test data are publicly available on GitHub, with future plans covering security improvements and broader hardware support.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds AI Tool to Auto-Document Undocumented APIs from Source Code

A developer released an open-source tool called API Archaeologist after spending days manually tracing undocumented and outdated API routes on a new team. The tool is a skill file for Claude Code and Codex CLI that scans backend source code directly, without relying on existing annotations or OpenAPI specs. It automatically generates a full API catalog in Markdown with Mermaid diagrams and a draft OpenAPI YAML file. The tool is designed for legacy codebases and projects where documentation has fallen behind the actual code. It is available on GitHub and works by running a single command from within a backend repository.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Five-Step Guide to Transactional Email Warmup for Safe Deliverability Scaling

A structured approach to transactional email warmup recommends using a dedicated sending domain and letting real business demand drive volume increases rather than following arbitrary calendar schedules. Engineers are advised to authenticate all sender identities, envelope paths, and tracking behavior before the first production send, prioritizing payment receipts over lower-urgency mail. Volume should be scaled in cohorts, with each batch fully observed and its bounce, complaint, and deferral signals reconciled before the next increase begins. To manage storage costs, teams should retain compact audit records — template versions, render inputs, and delivery events — while expiring full rendered message bodies on a defined schedule set by compliance requirements. The fallback sending lane should only be retired after peak traffic, retries, and at least one template change have successfully passed through the new domain without gaps in the delivery ledger.

0 comments Read more at DEV Community