Qiskit Benchmark Misreports 5% Slowdown as 10% Regression 1-in-4 Times

An analysis of IBM's Benchpress benchmark suite found that a real 5.37% performance regression in Qiskit's transpiler was incorrectly flagged as a 10% or greater regression about 24% of the time under the suite's standard three-run protocol. The root cause is that Benchpress's Qiskit gym does not fix the compiler's random seed, meaning each gate-count result is a single draw from an unmeasured distribution rather than a reproducible measurement. Increasing runs per version reduces the error rate but does not eliminate it — even 20 runs, representing roughly 40 hours of compute, still yields a 3.74% false-positive rate. The author identifies this as 'finite-sample decision risk' and notes that a single seed_transpiler argument could remove the sampling variance entirely at no additional compute cost. The findings raise broader concerns about benchmark reliability when stochastic compilers are evaluated without proper seed controls, since merge or revert decisions on pull requests depend on these verdicts.