How a Green Canary Deployment Missed a Bug That Corrupted Nine Hours of Data

·2 views

A software team's automated canary deployment system approved a release that ultimately corrupted nine hours of nightly settlement processing, despite showing healthy metrics. The flaw stemmed from canary pods being excluded from the Kafka consumer group, meaning the asynchronous batch workload — which actually triggered the bug — was never tested. The faulty code path involved a nullable decimal handled by a shared serialiser, a route rarely touched by HTTP traffic but hit constantly by a batch job running at 1 a.m. The team also discovered that analysis relied solely on HTTP status codes and latency, so a service returning incorrect data with a 200 status would still pass. In response, they added consumer group participation for canary pods, a shadow batch replay mode, business-level metrics, and synthetic requests mimicking edge-case tenant payloads.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Builds Fan-Made Interactive Daily Map Tool for PEAK Game

An independent developer has created Peak Map Today, a fan-made interactive map utility for the game PEAK, built around a design principle of immediate usability. Unlike typical game guide pages that lead with marketing content, the tool places a fully functional map at the top of the page so players can access the current biome, route, and markers without scrolling. The map draws on data from the open PeakMap project and supports multiple layers including Luggage, Belltowers, Animals, Amulets, and Tombs, each of which can be toggled or inspected individually. Both top-down and side-angle views are available where source data permits, helping players assess lateral distances and elevation changes from a single workspace. The implementation also handles touch and desktop zoom interactions carefully, using pointer events and browser gesture guards to keep map controls local and prevent unintended page-level zooming.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Is Erasing the Grunt Work That Trained Junior Developers

A widely shared essay by developer Asael Shinder argues that AI tools are automating the repetitive, low-level tasks that historically taught junior engineers core problem-solving instincts. Tasks like debugging obscure errors, writing test suites, and fixing configurations by trial and error formed an informal apprenticeship that new developers no longer experience by default. Shinder warns that juniors today are instead handed AI-generated output to review — a skill that itself requires the instincts only hands-on struggle can build. He calls on senior engineers to deliberately assign challenging work, encourage independent debugging, and make clear that the difficulty is intentional and instructive. Without such intervention, he cautions, the industry risks producing a generation that can generate code at scale but lacks the judgment to verify it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How VPNs Work: Encryption, DNS Leaks, and What They Cannot Do

A VPN routes your internet traffic through an encrypted tunnel via a VPN server, masking your public IP address from websites and making it harder for others on your network to read your data. The website you visit sees the VPN server's IP address rather than your own, while your device-to-server connection remains encrypted. However, a VPN does not make users fully anonymous, as websites can still track individuals through cookies, browser settings, and account information. DNS traffic must also be routed through the VPN to avoid DNS leaks, where name resolution requests bypass the encrypted tunnel entirely. Connection speeds can be affected by factors such as server distance, server load, and network routing quality, so switching servers may help improve performance.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why You Should Keep Feedback You Disagreed With and Revisit It Later

Career advice from developer Asael Shinder urges professionals to record feedback they initially reject, rather than dismissing it outright. Some critical feedback may not be wrong but simply early, meaning the recipient lacks enough context or examples to recognise its validity at the time. Revisiting saved feedback after six months often reveals patterns that were invisible in the moment, allowing for more honest self-assessment. Feedback that still seems incorrect after multiple reviews also becomes useful, giving a person evidence-based grounds to refute it rather than a reflexive denial. Shinder also warns that the same criticism from multiple unrelated sources over time is less a matter of opinion and more a reliable signal worth taking seriously.

0 comments Read more at DEV Community