LLM judges give inconsistent scores across runs, undermining AI eval pipelines

A software developer discovered that an LLM-based quality gate was producing different scores on identical inputs across repeated runs, with results fluctuating between 0.79 and 0.82 without any code changes. The inconsistency stemmed from four main causes: non-zero sampling temperature, floating model version aliases, vague scoring rubrics, and random tie-breaking at borderline scores. This unreliability eroded trust in the gate, as engineers began re-running failed checks until they passed, effectively turning the safeguard into a lottery. To fix this, the developer recommends setting temperature to zero, pinning exact model snapshots, running multiple judge calls and averaging results, quantizing scores to a coarse grid, and version-controlling rubric prompts in the codebase. The core principle is to treat LLM judgments as samples from a distribution rather than ground truth, and only flag failures when the mean score falls outside the expected noise band.