Sentry Reveals Scoreboard Bugs Were Hiding True AI Model Eval Results

A developer building an evaluation harness for AI models discovered that a seemingly catastrophic scoreboard — showing nearly all outputs as malformed — was the result of two capture-layer bugs, not actual model failures. Both bugs resided in a single file and caused valid responses from Anthropic Sonnet and a local llama3.2 model to be incorrectly labelled as malformed. One bug involved terminal control bytes corrupting CLI-captured output from the local model, while another caused the parser to reject valid JSON wrapped in markdown fences returned by Sonnet. A third issue was also uncovered: one failure entry on the scoreboard represented an API credit exhaustion event where no model was ever called. Using Sentry instrumentation, the developer was able to trace each failure to its specific layer — transport, parsing, or infrastructure — rather than incorrectly attributing them to model behavior.