Benchmark of 7 Python JSON parsers on 300 malformed LLM outputs reveals key tradeoffs

A developer who maintains a Python tool called jsonshim built a 300-case test suite, MALFORMED-300, to evaluate how well seven JSON parsers handle malformed outputs from large language models. The benchmark, run on Python 3.12.3, split results across 275 recoverable cases and 25 unrecoverable ones to fairly account for differing design goals among the libraries. jsonshim led the overall 300-case ranking with 94% accuracy, largely due to its refusal policy on unrecoverable inputs, while json-repair edged ahead on the 275 recoverable cases at 96% versus 94.3%. Libraries like dirtyjson, json5, pyjson5, and demjson3 scored poorly overall because they are dialect parsers focused on looser JSON grammar, not on extracting JSON embedded in prose or code fences. The author advises developers to first determine what refusal behavior their pipeline requires — whether silent failures are acceptable — before choosing a parser based on recovery rate.