58-Day LLM Agent Study Finds 43% of Failures Were Silent Output Shape Errors

A team running AI agents for real back-office work — including bookkeeping — logged 6,768 failures over 58 days and published the records as an open dataset. Contrary to expectations, 43% of failures were not hallucinations or refusals but 'shape errors,' where the model returned a correctly formatted HTTP 200 response with fluent text that downstream systems could not parse. The single most common failure, occurring 2,535 times, was a missing verdict line that a review agent's contract required, silently breaking the pipeline while human spot-checks showed nothing wrong. A secondary cluster of 1,048 failures involved language errors — including Chinese characters in Japanese output and mixed-language gibberish — often passing length and fluency checks undetected. The team recommends machine-checking output structure before any downstream stage reads it, logging failure reasons rather than just failure events, and measuring produced artifacts rather than agent invocations.