How One Team Built Automated LLM Evaluation That Catches 92% of Hallucinations

A development team discovered critical gaps in their AI quality process after deploying a RAG-based customer support assistant that served hallucinated responses to over 500 users before any issues were detected. The assistant had fabricated a billing policy and pulled rate-limit figures from a competitor's documentation, exposing the dangers of relying solely on informal manual testing. In response, the team built a production-grade evaluation pipeline that integrates automated judges — covering faithfulness, instruction-following, JSON schema validation, and safety — directly into their CI/CD workflow. The system uses a versioned 'golden dataset' of test cases and an ensemble of LLM-based and deterministic judges to block code merges that degrade output quality. The pipeline now catches 92% of hallucinations before deployment, replacing ad-hoc review with measurable, domain-specific metrics.