How One Team Built Automated LLM Evaluation That Catches 92% of Hallucinations

A software team discovered critical gaps in their AI quality process after deploying a RAG-based customer support assistant that served hallucinated responses to over 500 users before any issues were detected. The failures included fabricated billing policies and rate-limit figures pulled from a competitor's documentation, exposing a near-total absence of automated evaluation. In response, the team built a production-grade evaluation pipeline that runs domain-specific judges — covering faithfulness, instruction-following, JSON schema validation, and safety — directly within their CI/CD workflow. The system uses a versioned golden dataset and an ensemble of LLM-based and deterministic judges to score responses at each code merge, blocking deployments that fall below defined quality thresholds. The approach reportedly catches 92% of hallucinations before any changes reach production, replacing informal manual review with measurable, repeatable metrics.