How One Team Built Automated LLM Evaluation That Catches 92% of Hallucinations

A software team discovered critical gaps in their AI quality process after deploying a RAG-based customer support assistant that served hallucinated responses to over 500 users before being caught. The system had fabricated billing policies and pulled rate-limit figures from a competitor's documentation, exposing the danger of relying solely on informal, manual review. In response, the team built a production-grade evaluation pipeline integrating automated judges for faithfulness, instruction-following, JSON schema validation, and safety checks directly into their CI/CD workflow. The pipeline uses an ensemble of LLM-based and deterministic judges running against a versioned golden dataset, with results surfaced as pull request comments and regression alerts. The new approach reportedly detects 92% of hallucinations before deployment, replacing what the team described as a process of asking five questions and giving a thumbs up.