A single unparseable message silently halted shipment dispatches for 4.5 hours

A logistics platform lost nearly 4.75 hours of shipment dispatches one Saturday morning after a single malformed message caused a Kafka consumer to loop indefinitely on one partition. The problem began at 4:12 AM but went undetected until a depot called at 9 AM asking about a missing manifest, by which point 9,400 events had backed up. The root cause was an upstream team quietly changing a weight field to accept free text, allowing an operator override value of 'N/A' that the deserialiser could not handle. Rather than skip or quarantine the record, the consumer caught the exception, logged a warning, and retried the same message repeatedly for hours, generating 14,000 identical log lines that no alert monitored. The team subsequently introduced a dead-letter topic for unrecoverable errors, per-partition lag alerting on the oldest uncommitted offset, and a replay tool to reprocess fixed records.