nano-vLLM Benchmarked on RTX 3090: Key Insights Into LLM Serving Performance

A developer analyzed nano-vLLM, a compact ~1,200-line Python LLM serving engine, by modeling its prefill and decode costs using Qwen3-0.6B BF16 on a single RTX 3090 GPU. During cold prefill, the engine utilized roughly 76–84% of the GPU's measured BF16 compute capacity, while single-request decode aligned closely with a memory-traffic model estimating 531–562 GB/s of weight and KV data movement. Batching proved highly effective, scaling total decode throughput from 334 tokens per second at batch size 1 to 4,794 tokens per second at batch size 128. Longer prompt histories were found to reduce decode speed, as growing key-value cache reads add memory overhead per token. The study highlights that prefill and decode stress fundamentally different hardware resources, with batching amortizing weight costs and longer contexts increasing KV cache traffic.