Benchmark study finds AI model rankings shift based on which model grades answers

A developer built a testing framework to compare five AI language models — including local Llama, GPT, DeepSeek, and two Claude variants — across cost, speed, and answer quality using the same ten questions run twice each. The quality scores across all paid models clustered tightly between 0.92 and 0.97, with statistical analysis showing the differences were essentially noise rather than meaningful gaps. The developer discovered a critical flaw: the grading model was the free local Llama, which was simultaneously one of the five models being evaluated. When answers were re-graded using a paid model as judge, the pass rates shifted noticeably, revealing that the choice of grader significantly influenced the benchmark results. The experiment concluded that among the paid models tested, quality alone cannot determine a winner, and cost and speed become the more reliable decision factors.