SaaS automotive AI assistant cut response time from 30 seconds to under 3 via pipeline redesign

A US-based automotive SaaS company was struggling with an AI customer assistant that took up to 30 seconds to respond while searching a library of roughly 40,000 PDF documents totalling over 120 GB. Engineers found the bottleneck was not the language model itself but four upstream inefficiencies: retrieving too many passages, sending oversized prompts, checking user permissions after retrieval, and performing no caching between sessions. The team rebuilt the system as a custom retrieval-augmented generation pipeline on Google Vertex AI with Gemini, adding agentic orchestration and voice capabilities. The key architectural change was placing intent detection first, so the system identified the type of request before deciding what to retrieve, which documents the user was authorised to see, and how much context to pass to the model. The redesign brought response times down to under three seconds while also reducing costs and preventing potential data-permission leaks.