How a Stuck Kubernetes Volume Took Down a Payments Service at 2 a.m.

·1 views

A payments-adjacent service on a Kubernetes cluster went completely offline in the early hours, triggering an on-call incident for a veteran engineer. Investigation revealed that a third pod replica was stuck in ContainerCreating status due to a Multi-Attach error on a ReadWriteOnce PersistentVolumeClaim. The volume remained logically attached to a node marked NotReady, preventing it from being mounted on the node where the new pod was scheduled. Kubernetes' attach-detach controller was behaving as designed, refusing to forcibly detach the volume from an unresponsive node to avoid potential data corruption. The incident highlights a common but poorly understood limitation of RWO persistent volumes, where exclusive node-level attachment can block rescheduling during node failures.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How a Misleading Kubernetes Readiness Probe Silently Routed Users to Broken Pods

A software engineer discovered that two of five Kubernetes pod replicas were returning HTTP 500 errors on real endpoints while their health checks continued reporting a healthy 200 OK status. The incident began on a Tuesday morning when the checkout API's error rate quietly climbed to around 18% over twenty minutes, with no alerts or pod failures visible on any dashboard. Because Kubernetes relies solely on the configured readiness probe to determine whether a pod should receive traffic, both faulty pods remained in active rotation and kept serving real customer requests. Bypassing the load balancer and querying each pod directly revealed the mismatch: the /healthz endpoint was functioning normally while the actual business logic was failing. The root cause pointed to a common readiness probe configuration that checks only a lightweight health path rather than verifying the pod's ability to handle real application traffic.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why HSL Makes Color Palettes Look Uneven and How OKLCH Fixes It

Developer and color tool builder explains why HSL, a widely used color model, produces visually uneven palettes despite its mathematically equal lightness values. The problem lies in HSL's lightness channel being a geometric rather than perceptual measure, causing yellows to appear far brighter than blues at identical settings. OKLCH, a cylindrical form of the OKLab color space, addresses this by modeling human vision, ensuring colors at the same lightness value genuinely appear equally bright across all hues. Modern browsers now natively support the oklch() CSS function, allowing developers to use OKLCH values directly in stylesheets without conversion. The author has built a free palette generator at irrationaltools.com that leverages OKLCH for distinct, sequential, and diverging color schemes suited to data visualization and UI design.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer releases envault, a zero-dependency TypeScript env variable validator

A developer named Gavin Cettolo has released envault, an open-source Node.js library that validates, coerces, and types environment variables at application startup. The tool addresses a common vulnerability where missing or malformed environment variables go undetected until a runtime error surfaces deep in the application. Unlike using dotenv with Zod, envault has zero runtime dependencies and provides full TypeScript type inference, meaning variables like PORT are typed as number rather than string or undefined. Version 0.2.0 introduces array type support, per-environment enforcement via requiredIn, and a custom validate option for user-defined logic. The package also includes secret masking and auto-generation of a .env.example file, and is available via npm or pnpm.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer shares pattern to let Claude Code agents find files without hardcoded paths

A developer building a Claude Code skill for a Mac screenshot app called SlimSnap ran into a common integration problem: the skill needed to locate a captures folder that differs across user machines. The solution uses a config file written by the app itself to a fixed home-directory path, ~/.slimsnap/config.json, which the skill reads fresh on every invocation. This approach means users can change the folder in app preferences and the agent automatically picks up the new location without any manual reconfiguration. The developer notes that similar patterns already exist in tools like Cursor and Raycast, but are rare in the app-plus-agent space because most apps do not yet consider agent integration. The full skill file and capture schema are available as open-source repositories on GitHub under the MIT license.

0 comments Read more at DEV Community