How One Team Built Automated LLM Evals After 500 Users Saw Hallucinated Responses

A development team discovered the cost of informal AI testing after their RAG-based customer support assistant delivered hallucinated responses — including fabricated policies and competitor data — to over 500 users in production. The incident exposed a complete absence of automated evaluation, with testing limited to manually reading a handful of answers and giving a thumbs up. In response, the team built a production-grade evaluation pipeline integrating a suite of judges covering faithfulness, instruction-following, JSON schema validation, and safety checks. The system runs within CI/CD workflows, blocking code merges that degrade output quality and enabling rapid regression detection. According to the team, the pipeline now catches 92% of hallucinations before deployment, replacing subjective gut-checks with measurable, domain-specific metrics.