How a single dead DNS pod silently broke one-third of API gateway traffic

·1 views

An API gateway began returning 'Upstream Service Unavailable' errors for roughly a third of all requests, with no code changes, deployments, or backend failures to explain it. The root cause was an nginx worker holding a long-lived UDP socket pointed at a CoreDNS pod that Karpenter had already replaced during routine node cycling. Because nginx only reopens a DNS socket on an explicit error — not on a timeout — the worker kept querying a pod that no longer existed, causing silent DNS failures and downstream 502s. A misconfigured nginx setting allowed one worker to monopolize incoming connections, meaning a single broken socket effectively crippled the entire pod. Engineers confirmed the diagnosis by predicting failure patterns and deliberately deleting a live CoreDNS pod, ruling out an initially suspected port-exhaustion theory.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Linux fork() Uses Copy-on-Write and Why It Can Exhaust Memory

When a Linux process calls fork(), the kernel does not immediately duplicate physical memory; instead, it copies the page table hierarchy and marks all entries as read-only. Any subsequent write by either the parent or child process triggers a hardware page fault, prompting the kernel to allocate a new physical page and copy the data. This Copy-on-Write mechanism keeps fork() fast under low-write conditions but becomes costly when workloads involve heavy, sustained writes to large memory regions. Systems running in-memory databases like Redis can experience write amplification, TLB thrashing, and uncontrolled memory growth during background snapshot operations. In extreme cases, the cascade of micro-allocations exhausts available RAM and activates the kernel's Out-of-Memory killer, terminating processes unexpectedly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

NPU, DPU, QPU Explained: Which Specialist Chips Are Production-Ready Now

Hardware vendors are aggressively promoting three chip types — NPUs, DPUs, and QPUs — but their real-world utility varies significantly. Neural Processing Units (NPUs) are already proving their worth in production for low-power, on-device AI inference tasks like keyword detection and camera segmentation on mobile and edge devices. Data Processing Units (DPUs), such as NVIDIA BlueField and AMD Pensando, are similarly mature, helping large-scale infrastructure operators offload high-throughput networking, storage, and security tasks from general-purpose CPUs. Quantum Processing Units (QPUs), by contrast, remain largely confined to research settings and are not yet viable for mainstream production workloads. Engineers are advised to evaluate each chip against specific workload requirements rather than adopting them based on vendor marketing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

EduVetta: New SaaS Tool Automates Repetitive Admin Tasks for Teachers

A developer has launched EduVetta, a SaaS platform designed to reduce the administrative workload teachers face outside of actual classroom instruction. The tool offers four core features: automatic lecture note summaries, AI-drafted question papers with answer keys, a simplified assignment submission system, and interactive quiz generation. Teachers can generate content from multiple input types including YouTube links, PDFs, or spoken requests, and students can access assignments without creating an account. Objective grading such as multiple-choice questions is automated, while subjective answers are intentionally left for human review to avoid errors that could harm students. The product is currently available with a free tier at eduvetta.com, though the developer notes that diagram- or equation-heavy subjects still require additional manual editing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Structured Data Contracts Can Make Backlink Audits Easier to Debug

A technical design note proposes a small, structured data contract to improve how backlink audit tools record and interpret results. Rather than returning a simple pass-or-fail flag, the proposed format captures fetch state, document scope, link presence, and index or referral status separately. The distinction matters because a page can be reachable while a specific link is missing, or a link can exist without any evidence of indexing. The article also outlines best practices such as using URL parsers instead of substring matching, resolving relative links before comparison, and preserving prior successful observations during outages. The author notes the piece was prepared with AI assistance and is a design proposal, not a production benchmark or description of Google's internal systems.

0 comments Read more at DEV Community