A single unparseable message silently halted shipment dispatches for 4.5 hours

·1 views

A logistics platform lost nearly 4.75 hours of shipment dispatches one Saturday morning after a single malformed message caused a Kafka consumer to loop indefinitely on one partition. The problem began at 4:12 AM but went undetected until a depot called at 9 AM asking about a missing manifest, by which point 9,400 events had backed up. The root cause was an upstream team quietly changing a weight field to accept free text, allowing an operator override value of 'N/A' that the deserialiser could not handle. Rather than skip or quarantine the record, the consumer caught the exception, logged a warning, and retried the same message repeatedly for hours, generating 14,000 identical log lines that no alert monitored. The team subsequently introduced a dead-letter topic for unrecoverable errors, per-partition lag alerting on the oldest uncommitted offset, and a replay tool to reprocess fixed records.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Ditches Paid Snippet App, Rebuilds It Locally With Just 6 Commands

A developer abandoned a paid cloud-based code snippet manager after concluding that its sync convenience did not justify its cost or trust model. The core problem it solved — scattered snippets across bookmarks, notes apps, and scratch files — is real and measurable, costing developers time each time they re-derive code they have already written. The author argues that for working developers, the terminal is the most natural place to capture and retrieve snippets, making a local, CLI-based tool more practical than browser or cloud alternatives. A self-built snippet manager using six commands offers local-first storage, zero dependencies, and compatibility with version control. The author advises that such a tool only earns its place once a developer's snippet collection outgrows their editor's built-in system, typically signalled by repeated failure to locate code they know they once wrote.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Visa Raises Card Retry Limit to 20 in 30 Days, Catching Many Merchants Off Guard

Visa updated its Core Rules on 18 April 2026 to allow merchants up to 20 retry attempts on soft-declined cards within 30 days, raising the previous limit of 15. Major payment processor Adyen's documentation still references the old figure, suggesting many merchants may be operating on outdated retry logic. A card decline carries three distinct signals — an ISO 8583 response code, a scheme-level retry category, and a PSP-normalised code — and retry rules are tied to the first two, not the third. Visa classifies declines into four categories, with Category 1 codes barring retries entirely and Categories 2, 3, and 4 each permitting up to 20 attempts in 30 days. Adding complexity, the same response code can carry different meanings across networks — for example, code 65 signals a withdrawal frequency limit on Visa but triggers a strong customer authentication requirement on Mastercard in the UK and EEA.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Long Build Times Are Quietly Stealing Developer Focus and Productivity

Software developer Serguey Asael Shinder argues that slow build times have a hidden cost beyond the seconds they consume. When developers wait for builds to finish, they habitually fill the gap with distractions like email and chat, fragmenting their concentration. Repeated dozens of times a day, these micro-interruptions compound into significant lost focus. Shinder urges developers to measure their actual build loop duration and treat excessive wait times as a defect worth fixing. Where waits are unavoidable, he recommends intentional breaks rather than reflexive context-switching.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Silent Data Truncation Can Make AI Models Confidently Wrong

AI language models can produce confident but flawed responses when they only receive a portion of the input data, with no error or warning raised. Common causes include retrievers returning limited passages, file readers hitting line limits, and context windows silently dropping older content. Because the model only processes what it receives, it has no way of knowing that information is missing. This makes it difficult for users to identify the root cause, as the problem lies in the pipeline rather than the model itself. Experts recommend verifying inputs before questioning outputs — for example, by asking the model to quote the last line received or counting characters sent — to catch truncation early.

0 comments Read more at DEV Community