How One Team Built Automated LLM Evals After Hallucinations Reached 500 Users

A development team discovered the cost of informal AI testing after their RAG-based customer support assistant served hallucinated responses—including fabricated policies and competitor data—to over 500 users in production. The incident revealed their entire evaluation process consisted of manually reviewing five sample questions before deployment, with no automated checks in place. In response, the team engineered a production-grade evaluation pipeline integrating a suite of judges covering faithfulness, instruction-following, JSON schema validation, and domain accuracy into their CI/CD workflow. The system uses an LLM judge ensemble alongside deterministic checks, run against a versioned golden dataset, and is designed to block pull request merges that degrade quality. The team reports the pipeline now catches 92% of hallucinations before deployment, replacing subjective human review with measurable, repeatable metrics.