Developer Builds Tool to Evaluate Reliability and Bias in LLM Judge Models

A developer has created a lightweight open-source evaluator designed to assess the performance of large language models when used as automated judges. The tool tests LLM judges across several dimensions, including consistency across repeated runs, position bias, sensitivity to response length, and accuracy in preferring higher-quality answers. Each test case in the dataset includes a task rubric, an ideal response, and a negative response — where the latter represents a less preferred rather than necessarily incorrect answer. The project, currently around 200 lines of code, is publicly available on GitHub under the name JudgeDjudge. The developer is actively seeking community feedback on additional failure modes or alternative approaches to evaluating LLM judges.