How One Team Built Automated LLM Evals After Hallucinations Reached 500 Users

A development team discovered the cost of informal AI testing after their RAG-based customer support assistant served hallucinated responses — including fabricated policies and competitor data — to over 500 users before anyone caught the errors. The incidents exposed a complete absence of automated evaluation, with the team's only testing process being manual review of a handful of questions. In response, they designed a production-grade LLM evaluation pipeline integrating domain-specific judges, CI/CD blocking, and a versioned golden dataset to catch regressions before deployment. The system uses a ensemble of judges assessing faithfulness, instruction-following, JSON schema validity, safety, and domain accuracy, with thresholds ranging from 0.8 to 1.0 depending on the criteria. The team reports the pipeline now catches 92% of hallucinations prior to deployment, replacing subjective gut-checks with measurable, automated quality gates.