How many tokens per second one GPU should serve for an LLM depends on the metric

A single NVIDIA H100 serving an 8B model under full load produces roughly 1,300 to 26,000 output tokens per second in total, depending on prompt length, while each individual user sees only a few dozen to a few hundred. The article argues the right target is a per-user speed floor for chat, around 30 to 50 tokens per second, with as many users packed onto the GPU as possible while that floor holds. It distinguishes total system throughput, which determines cost, from per-user output speed, which determines what people experience, citing NVIDIA's metrics guide. Single-user speed is capped by memory bandwidth, since each token requires reading all model weights, so an H200 can stream one user about 16 times faster than an L4. Batching shares those weight reads across requests, which is why total throughput rises as concurrency grows while per-user speed falls.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in