Swapping AI Judges Mid-Experiment Makes Quality Scores Unreliable, Study Warns

A technical analysis warns that replacing one LLM-based evaluation judge with another mid-cycle makes it impossible to determine whether a score change reflects genuine product improvement or simply a different measuring instrument. Drawing on the 1986 Bland-Altman clinical measurement framework, the author argues that correlating two judges is the wrong approach and that plotting their differences against their averages reveals more meaningful information. Using a 60-item anchor set, the team found their offset correction carried an uncertainty of ±0.030, roughly 60% of the 0.05 improvement they were trying to detect. This means subtracting a calculated offset merely relabels the uncertainty rather than resolving it, while creating a false impression of precision. The analysis concludes that anchor set size, score distribution, and the distinction between aggregate and per-item confidence intervals are all critical factors when switching evaluation instruments.