LLM Judges Give Inconsistent Scores on Identical Inputs, Undermining CI Gates

A software developer discovered that an LLM-based quality gate on their merge pipeline returned different scores — 0.79, 0.82, and 0.80 — across three identical runs with no code or prompt changes. The inconsistency stemmed from four main causes: non-zero sampling temperature, floating model version aliases, vague scoring rubrics, and noise at tie-breaking boundaries. When gates behave this way, developers learn to re-run jobs until they pass, effectively turning a quality check into a lottery. The author resolved the issue by setting temperature to zero, pinning exact model snapshots, averaging scores across multiple judge calls, and quantizing scores to a coarse grid. The broader fix recommended is to treat LLM judge scores as distributions rather than ground truth, only failing a gate when the mean falls below the threshold by more than the measured noise margin.