How to Build an AI Video Dubbing Pipeline Using Whisper, XTTS-v2 and FFmpeg
A developer tutorial on DEV Community outlines how to build a minimal AI dubbing pipeline that converts English-narrated videos into Spanish using cloned versions of the original speaker's voice. The stack combines faster-whisper for timestamped transcription, a translation engine for segment-level text conversion, XTTS-v2 for voice synthesis, and FFmpeg for audio-video processing. A key challenge addressed is sync drift, where translated audio runs longer or shorter than its allotted time slot — tackled by passing length constraints per segment to the translation step. The guide notes that the original Coqui TTS company shut down in early 2024, and the maintained fork is now published by Idiap as coqui-tts. Readers are cautioned that XTTS-v2 model weights carry a non-commercial license, and that voice cloning legally requires written consent from the original speaker.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in