Engineer Discovers AI Voice Selection System Was Penalizing Candidates for a Non-Issue

A developer built an automated system to score and rank AI voice model candidates using metrics like Whisper transcription match rate, speech rate, and jitter. The system rejected candidates whose generated audio contained hallucinations — fabricated words appended to short sentences by the diffusion TTS engine used during corpus creation. Further investigation revealed that these hallucinations were a quirk of the voice-generation pipeline and did not appear in the final trained models that users actually hear. When probe sentences were lengthened to eliminate hallucinations, top-ranked candidates changed, exposing that the metric was measuring process behavior rather than actual voice quality. The engineer concluded that the scoring system had been filtering out candidates based on a manufacturing defect that was both fixable and absent from the shipped product.