How to Evaluate LLM Response Quality and Accuracy in Production

This interview-style piece argues that judging an LLM's answers requires a repeatable evaluation pipeline rather than a single accuracy figure. It recommends building a golden dataset with reference answers or rubrics, plus adversarial cases such as stale facts, conflicting memories, and queries that should be refused. Retrieval, generation, and memory are to be scored separately, using metrics like context precision and recall, faithfulness, and answer relevance. LLM-as-judge scoring is suggested for scale, but only when calibrated against human labels and paired with deterministic checks where possible. The author also advises running offline evals in CI, monitoring live traffic for drift, and tying metrics to business outcomes like task success rate.