Developer builds honest local LLM benchmark tool after finding standard metrics misleading

A developer created an open-source tool called homebench after finding that standard token-per-second metrics failed to reveal whether a local LLM was actually useful or correctly quantized. The tool measures output token speed during generation only, deliberately excluding model load time and prompt processing to reflect what users actually experience during text output. Memory reporting presented a separate challenge, as RAM usage figures vary depending on whether weights, KV cache, or theoretical model-card sizes are counted, so the tool labels each metric transparently rather than blending them into a single figure. The developer also built a concurrency sweep feature to capture how batching servers like vLLM behave under simultaneous requests, since single-stream benchmarks do not reflect real throughput gains from batching. Throughout the project, the author prioritized clearly labelling measurement limitations over producing misleadingly precise numbers.