Engineer shares 18 lessons from building 12 AI voice models using diffusion TTS

A software engineer documented roughly one month of work designing and deploying 12 role-specific AI voice models — covering narrator, counselor, sales, presenter, operator, and MC roles for both men and women. The project used diffusion TTS for voice design, where each voice is defined by a text caption and a random seed, making results fully reproducible. A key finding was that diffusion TTS ran 2.5 times slower than real time on the same GPU, making it unsuitable for live conversation and prompting a two-stage approach using lighter pre-trained models for actual speech synthesis. The engineer identified corpus quality gates as a central problem, noting that six of the 18 documented failures involved gates that existed but did not work as intended — for instance, strict gates inadvertently filtered out expressive takes, leaving only monotonous audio. Additional pitfalls included undetectable long-vowel defects caused by text normalization, a hallucination guard that only failed during actual hallucinations, and symbol-stripping rules that silently dropped valid Japanese characters from input text.