Building Trustworthy LLM-as-a-Judge Graders Requires Calibrated Rubrics

An LLM judge is only reliable when it has been calibrated against human labels, since otherwise it produces a consistent number that poorly reflects real quality. The article argues that the rubric, not the judge prompt, does the real work, and recommends writing concrete, checkable conditions instead of vague adjectives. It identifies three common problems: position bias in pairwise comparisons, a preference for verbose answers, and lenient grading when a model evaluates its own outputs. To counter these, it advocates placing hard-failure conditions first so clear errors are scored before any subjective interpretation. The piece closes with an example judge implementation built on the Anthropic API.