Judge the Judge: Using GEPA to Calibrate LLM Evaluators Against Human Annotations

A talk by Mahmoud Mabrouk, co-founder and CEO of Agenta AI, addresses why off-the-shelf LLM-as-a-judge evaluators often disagree with human assessments and mislead teams about agent quality. He argues that calibrating evaluators against human annotation is essential for both faster offline iteration and reliable online monitoring. The method centers on GEPA, a genetic-style prompt optimization algorithm that refines an evaluator's prompt using reflection on failures and Pareto-front selection. The example uses the Tau-bench airline customer-support dataset, with 599 labeled conversations judged on policy compliance, with reasoning included in each label. The speaker also stresses that metrics should derive from domain experts' error analysis, and that binary judgments with reasoning are easier to calibrate than fine-grained scales.