How to properly evaluate LLM agents using evals, golden sets, and LLM-as-judge

Unlike deterministic functions, LLM agents can produce different outputs for the same input due to temperature and model sampling, making traditional unit tests ineffective for quality assurance. Evaluations, or 'evals', replace pass/fail assertions with a measurement-based approach that scores outputs across multiple runs and tracks performance over time. A curated 'golden set' of real input-to-expected-outcome examples — ideally grown from production failures — forms the backbone of this testing framework. Offline evals run against this fixed set in CI pipelines before deployment, while online evals monitor live traffic to catch distribution shifts and unanticipated inputs. For subjective output qualities, LLM-as-judge uses a separate model to score responses against a rubric, though the judge model itself must be evaluated for biases and failure modes.