Developer finds LLM is fastest link in real-time AI pipeline, not the bottleneck

A developer building LiveSuggest, a browser-based real-time meeting assistant, instrumented his entire AI pipeline in production to find where latency was actually coming from. After a week of metrics, he discovered that the language model — long assumed to be the slow part — delivered a first token in roughly 1.2 seconds at the median, using both GPT-4o-mini and Claude Haiku 4.5. Speech-to-text transcription was faster at the median (around 75ms) but spiked to nearly nine seconds at the 95th percentile due to how silence-based sentence-ending detection handles non-stop speakers. A gating layer that decides whether enough new content exists to warrant a suggestion turned out to be the most impactful hidden cost in the pipeline. The findings challenged the developer's assumption that fighting LLM token latency would dominate optimization efforts, pointing instead to tail-end STT behavior and prompt-construction logic as the real culprits.