How One Team Built Automated LLM Evals After Hallucinations Reached 500 Users

A development team discovered the cost of informal AI testing after their RAG-based customer support assistant served hallucinated responses—including fabricated policies and competitor data—to over 500 users in production. The failures stemmed from a manual review process of just five questions before deployment, with no automated evaluation in place. In response, the team built a production-grade LLM evaluation pipeline incorporating a suite of judges covering faithfulness, instruction-following, JSON schema validation, and domain-specific accuracy. The system integrates with CI/CD workflows to block code merges that degrade response quality, and relies on versioned golden datasets for regression detection. The team reports the pipeline now catches 92% of hallucinations before deployment, replacing subjective gut-checks with measurable, automated metrics.