Developer Builds Open-Source QA Tool to Catch TTS Mispronunciations Standard Metrics Miss

A developer created ttsproof, a text-to-speech quality assurance framework, after finding that the widely used Word Error Rate metric produced both false positives and false negatives when evaluating TTS output. The tool separates structural audio defects from pronunciation errors and uses equivalence-aware scoring to avoid penalising correct speech rendered in a different written form. In a blind study of 390 audio samples across 130 edge cases and three voices, no structural defects were found, but 19 genuine mispronunciations were identified — all involving short acronyms or isolated letters. A human-review quarantine layer prevented both the wrongful rejection of 23 correctly spoken clips and the silent shipment of the 19 real errors. The study has been published as a citable technical report on Zenodo under a CC-BY-4.0 licence.