Why Side-by-Side Prompt Comparisons Are Often Just Coin Flips

A developer discovered that comparing AI prompt versions using only a handful of sample outputs is unreliable, because model outputs vary between runs even when nothing changes. Running the same prompt twice on identical inputs revealed disagreements roughly as large as those between two competing prompt versions, exposing that small-sample comparisons are largely noise. The author argues that teams must first measure their model's run-to-run variability — the 'noise floor' — before any prompt comparison result can be trusted. Once sample sizes grow beyond a handful, human reviewers become inconsistent, making regex-based, countable metrics more reliable than subjective reading. Rather than chasing abstract quality, the approach focuses on naming specific, observed failure modes and tracking whether each one decreases across versions.