How one team built a real-time corruption detector for a self-hosted 300B LLM

An engineering team running a self-hosted ~300B reasoning model in production found that it occasionally produced garbled, off-topic output mid-sentence — a problem they term 'decoding corruption,' distinct from hallucination. Unlike hallucination, where the model gives wrong information, corruption means the model stops generating a coherent answer entirely, sometimes outputting repeated names, spreadsheet errors, or unrelated foreign-language articles. Because the model streams output token by token to live users, post-generation detection was deemed too late and too costly, with one corrupt run wasting over four minutes of GPU time. The team built a lightweight, single-pass streaming detector using five methods — rule-based thresholds, character n-gram surprise scoring, Count-Min Sketch repetition tracking, rolling SimHash fingerprinting, and character entropy — all running off one shared state object at ~50,000 characters per second on CPU alone. Testing showed a true positive rate of 1.000 across all corrupt streams, though an unexpected bad model merge temporarily caused 43% of healthy outputs to be falsely flagged, highlighting that detector tuning is as critical as detection itself.