Why Tokens Per Second Benchmarks Can Be Misleading for Local AI Models

Tokens per second (tok/s) is the standard speed metric for local large language models, but the same model can produce anywhere from 45 to 793 tok/s depending on how it is tested. Most consumer benchmarks measure performance for a single user making one request at a time, which only reflects personal, private use cases. When a model is deployed behind an API or shared across a team, throughput under concurrent requests becomes the more meaningful measure. Technologies like vLLM use continuous batching and PagedAttention to keep GPU cycles occupied with multiple simultaneous requests, dramatically increasing total output. In a Red Hat benchmark using Llama 3.1 8B on an A100 40GB GPU, vLLM achieved up to approximately 793 tok/s under heavy concurrency — roughly 19 times the throughput of single-request tools like Ollama at scale.