Split ConfigMap values caused duplicate dispatch emails for two months

·10 views

A software team discovered that editing a Kubernetes ConfigMap does not trigger a pod restart, meaning environment variables are only updated when pods are recycled. A lease duration change made in June — cutting job timeout from ten minutes to two — never propagated to all running pods until a node pool upgrade in August partially applied it. This left 6 of 24 pods running the new two-minute lease while 18 retained the old ten-minute value, creating inconsistent job-handling behavior across the fleet. Any email send taking over two minutes would be reassigned by the queue and delivered twice, affecting roughly one in every nine thousand sends. The team resolved the issue by adding a configuration hash annotation to deployments and setting up alerts to detect when pods in a fleet report differing config values.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Healthtech Teams Can Log API Key Identity Without Exposing Secrets

A software architecture guide outlines a method for healthtech platforms to record API key usage at service startup without storing the key itself or any sensitive data. The approach uses an HMAC-SHA-256 derived fingerprint tied to an immutable build identifier, workload identity, and deployment attempt, enabling incident attribution without credential exposure. Seven bounded fields — including schema version, environment, build ID, and key fingerprint — are recommended to keep audit records minimal and privacy-safe. A uniqueness constraint and 90-day retention policy are proposed to prevent audit logs from generating unbounded telemetry costs, with one event per deployment attempt estimated at roughly 2,880 records per day across 240 workloads. The design ensures that losing the audit log sink does not disrupt clinical services, maintaining a clear boundary between optional correlation signals and critical operations.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Startup Founder Gets Unexpected Job Lead Through His Own Marketing Co-Founder

A startup founder shared how his marketing co-founder, just weeks into her role, surfaced an unsolicited job opportunity for him while doing outreach for their company. The role's requirement was unusual — not a specific language or framework, but someone who 'ships fast with Claude but still owns the architecture.' The founder interpreted this as a signal that hiring managers are increasingly worried about developers who use AI tools to produce output without truly understanding what they've built. He argues that the real skill now is not just building quickly, but being able to explain, defend, and take accountability for the systems produced. He ended up referring himself for the role, reflecting that opportunities often come through the networks others build on your behalf.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Docloom Uses AST Parser to Ground AI-Generated API Docs in Real Code

A solo developer has built Docloom, a tool that addresses the common problem of outdated or inaccurate API documentation. Instead of letting an AI freely interpret source code, Docloom first uses an AST parser to extract routes, function signatures, and parameter types, then passes only those verified facts to an LLM for writing. A built-in fact-checking step cross-references every claim in the generated text against the source before publication. When documentation is regenerated after a code merge, users see a diff and must manually approve changes before anything goes live. The tool connects via a read-only GitHub App, does not store source code, and is currently free to use without a credit card.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Feature Flag Services Must Validate Config Values Before They Go Live

Feature flag and remote config services let teams change production behavior without new deployments, but this convenience introduces risk since config changes bypass the automated testing that code deployments undergo. Common outages trace back to small input errors such as wrong units, missing URL schemes, typos in JSON, or misscaled numeric ranges that go unnoticed until they reach users. Most services store values in loose types like string, number, or boolean, which is insufficient to enforce the specific constraints an application actually requires. ConfigDirector, a remote config platform, addresses this by assigning fine-grained types to each config — including bounded integers, timespans, HTTPS-only URLs, enums, and JSON Schema-validated documents — so invalid values are rejected at save time across all environments. The approach ensures a bad value never reaches the SDK or end users, regardless of whether it was entered via dashboard, Terraform, or an AI assistant.

0 comments Read more at DEV Community