Massive AI Judge Audit Finds Consistency Mistaken for Accuracy in Benchmarks

A large-scale study analyzing over half a million AI-generated judgments, published June 19, 2026 (arXiv 2506.19544), found that AI judges are consistently repeatable but not actually correct in their evaluations. Researchers identified a critical flaw: the AI evaluation field has been treating consistency as a proxy for trustworthiness, an assumption the audit proves is unfounded. A judge that blindly selects the same answer every time would score perfectly on consistency metrics while being entirely useless. When the researchers adjusted scores to account for chance agreement, previously meaningful performance gaps between models shrank considerably. The paper also offers a short practical checklist for developers to verify whether their AI judges are genuinely valid before relying on them in real-world applications.