How one developer cut 1.5 seconds from a real-time speech-to-LLM desktop pipeline

A developer building a desktop overlay that transcribes video call audio and streams AI responses identified several bottlenecks adding up to roughly three seconds of latency. Running voice activity detection on raw audio before normalization prevented the speech-to-text server from misreading ambient noise as speech, sharpening its segment timing. A question-classifier model intended to filter utterances was removed entirely after it repeatedly failed on conversational follow-ups, saving 200ms and improving accuracy. A generation counter was introduced to prevent stale WebSocket connections from mixing transcript data with live ones during reconnects. Prompt cache TTLs were also extended to one hour to avoid costly cache misses during the long pauses typical in interview-style conversations.