Practical Tips to Boost Faster-Whisper Accuracy in Real-Time Voicebot Systems

A developer running faster-whisper as the speech-to-text engine in a real-time avatar/voicebot system has shared practical techniques that improved recognition accuracy in production. The most impactful finding is that passing recent avatar speech output as the initial_prompt — rather than a static dictionary of proper nouns — provides more relevant, self-updating vocabulary hints with zero maintenance overhead. The author warns against including user-side STT results in the prompt, as misrecognitions can cascade and corrupt subsequent recognition. On the audio pipeline side, raising Opus bitrate to 128 kbps with in-band FEC enabled in WebRTC setups reduced edge-case errors, while acoustic echo cancellation was flagged as a source of word-onset distortion that cannot be corrected at the STT stage. Confidence-score thresholding based on avg_logprob was also found unreliable in practice, with the recommendation to log scores for diagnostics and delegate noise rejection to downstream LLM or application logic instead.