Flawed LLM Eval Harnesses Can Blame Models for Their Own Congestion

A common mistake in LLM evaluation occurs when a test harness opens too many concurrent requests, triggers rate-limiting (HTTP 429 errors), and then retries those requests — effectively doubling the intended load on the endpoint. This causes client-side queueing delays to inflate observed latency figures, meaning the harness is measuring its own congestion rather than the model's true performance. The result is misleading metrics: a reported p99 latency of 8.4 seconds, for example, may include roughly 6 seconds of queue-wait time that has nothing to do with the model. The recommended fix is a closed-loop harness that caps in-flight requests and separates queue-wait time from actual service time, ensuring measured latency reflects endpoint behavior. This approach also enables systematic discovery of an endpoint's concurrency limits by sweeping the concurrency gate upward until throughput plateaus and 429 rates rise.