LLM Cost Benchmarks Can Mislead Without Accounting for System-Level Noise

A developer running identical LLM cost checks four consecutive times on an untouched local server observed swings of +27% and -11% between runs, despite no configuration changes. The variation stemmed entirely from timing differences, as cost per million output tokens is directly derived from wall-clock time when token counts remain fixed. Background processes, thermal throttling, OS scheduling, and memory pressure on the host machine were identified as likely culprits, though none were instrumented during the test. The author warns that a single before-and-after measurement cannot distinguish genuine performance changes from transient system noise, even when confidence intervals appear non-overlapping. The recommended fix is to run multiple baseline checks of an unchanged configuration to quantify run-to-run variability before drawing conclusions from any benchmark comparison.