SShortSingh.
Back to feed

Phantom Friday checkout crashes traced to forgotten canary deployment instance

0
·16 views

A checkout service began throwing intermittent server errors most Friday afternoons, with all symptoms disappearing by Monday, making the bug extremely difficult to reproduce or diagnose. Initial investigations ruled out long-running queries, connection leaks in a newly shipped reporting endpoint, and rogue scheduled jobs as potential causes. The real culprit turned out to be a forgotten third service instance left over from a canary deployment experiment months earlier, which was no longer receiving live traffic and therefore invisible in standard metrics. This dormant instance had a faulty health-check routine that opened a fresh database connection every few seconds without reusing the pool, steadily exhausting the connection limit. An auto-scaling policy restarted the instance every Monday, resetting the leak and explaining the precise weekly pattern engineers had struggled to pin down.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

53ON: Open-source OpenAI-compatible AI server for AMD Ryzen AI Max mini PCs

A developer has released 53ON, an MIT-licensed AI server built for AMD Ryzen AI Max+ 395 mini PCs with 128 GB of unified memory. It runs as a dependency-free Node.js process on top of llama.cpp and supports both OpenAI and Anthropic API formats. The software includes a control panel for loading models, issuing per-client API keys with token quotas, and forwarding requests to cloud providers when local models fall short. The author also publishes a custom llama.cpp build whose KV cache allocates memory as tokens arrive, not all at load time, which reduces initial memory use substantially on Vulkan. So far it has been tested only on Strix Halo hardware with Vulkan (RADV) on Ubuntu 24.04, and the author invites feedback and bug reports.

0
ProgrammingDEV Community ·

Per-Account Timezone Scheduling for SaaS Deadline Emails Using UTC Dispatch

The article argues that SaaS systems should store each customer's timezone and local business deadline as data rather than encoding them in cron expressions. A single UTC scheduler resolves that local intent into an eligible instant and claims due records for delivery. Recalculation and rule versioning let pending work adapt to changes without altering already delivered emails. Because email side effects cannot realistically be exactly-once, the author recommends at-least-once execution paired with idempotent operations and a stable delivery identity. Ambiguous local times during clock transitions need an explicit, versioned, and tested resolution policy.

0
ProgrammingHacker News ·

Internet Archive Moves to Preserve Three Decades of Salon Journalism

The Internet Archive has announced an effort to preserve about 30 years of content published by Salon, the online magazine. The announcement appeared on the Archive's official blog in November 2026. The source material provided is limited to the blog post link and a Hacker News discussion thread, so specific details of the preservation method and scope are not available. The initiative aims to keep the long-running publication's archive accessible for future readers and researchers. Further details would be found in the original Internet Archive blog post.

0
ProgrammingDEV Community ·

How to safely unsave Reddit posts and export your saved data first

Reddit does not appear to offer an official bulk unsave or multi-select option, so the documented method is unsaving items one at a time. Because unsaving has no documented undo, the article recommends requesting an official data export first, which Reddit says can take up to 30 days. Unsaving only removes an item from your saved list and does not delete the post or comment itself. The Saved page also appears to show a limited number of items, so older saves may not be visible there. The author advises against third-party scripts or automation, which Reddit's rules restrict, and promotes their own browser extension for organizing saves.