Flawed output parsing, not model ability, caused a 0.31 vs 0.70 benchmark gap

A developer benchmarking a vision-language model initially scored it at 0.31 accuracy, only to discover the true score was 0.70 after fixing how the model's output was read. The root cause was a token limit of 1024: on harder questions requiring lengthy reasoning, the model hit the cap before printing its final answer line, causing the scoring script to mark those responses as wrong. Of 100 questions, 64 returned no parseable answer — all due to truncation, not model errors — and the affected questions were disproportionately the most complex ones. The fix was to enforce structured output at decode time using a JSON schema, ensuring the model always returned a valid answer letter regardless of reasoning length. The author warns that scraping letters from truncated reasoning traces introduces position bias, making garbage outputs appear statistically valid to a scorer.