How One Team Built Automated LLM Evals After Hallucinations Reached 500 Users

A development team discovered the cost of informal AI testing after their RAG-based customer support assistant served hallucinated responses—including fabricated policies and competitor data—to over 500 users in production. The failures exposed a complete absence of automated evaluation, with their only testing method being a manual review of five sample questions. In response, the team built a production-grade LLM evaluation pipeline integrating multiple judge types—covering faithfulness, instruction-following, JSON schema validation, and safety—into their CI/CD workflow. The system uses a versioned 'golden dataset' of test cases and a judge ensemble to catch regressions before code merges, reportedly identifying 92% of hallucinations pre-deployment. The team shared their architecture and code publicly, arguing that academic benchmarks like MMLU are insufficient substitutes for domain-specific, automated evaluation in real-world applications.