SShortSingh.
Back to feed

How many tokens per second one GPU should serve for an LLM depends on the metric

0
·1 views

A single NVIDIA H100 serving an 8B model under full load produces roughly 1,300 to 26,000 output tokens per second in total, depending on prompt length, while each individual user sees only a few dozen to a few hundred. The article argues the right target is a per-user speed floor for chat, around 30 to 50 tokens per second, with as many users packed onto the GPU as possible while that floor holds. It distinguishes total system throughput, which determines cost, from per-user output speed, which determines what people experience, citing NVIDIA's metrics guide. Single-user speed is capped by memory bandwidth, since each token requires reading all model weights, so an H200 can stream one user about 16 times faster than an L4. Batching shares those weight reads across requests, which is why total throughput rises as concurrency grows while per-user speed falls.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Enterprise AI Guardrails: Limiting Blast Radius and Keeping Humans in the Loop

The author argues that enterprises deploying AI agents must plan for the case where an agent acts confidently but incorrectly, not just for security and compliance. The central concern is how far a wrong action can spread before something stops it. Recommended controls include scoped, short-lived non-human credentials, gateway policy and rate limits, staged rollouts with dry runs and automatic rollback, and kill switches with audit trails. The piece distinguishes between humans in, on, or out of the loop, depending on the risk of the automated action. It also recommends measuring model confidence in shadow mode before granting autonomy and tracking cost per completed task rather than per token.

0
ProgrammingDEV Community ·

Guide Outlines Automated Azure Container Apps Deployment via DevOps Pipelines and Managed Identity

A technical article explains how Azure Container Apps can be deployed automatically using Azure DevOps Pipelines. It highlights Managed Identity as a way to authenticate to Azure services without storing credentials such as client secrets or certificates. The guide also covers integration with Azure Container Registry for managing container images and Azure Monitor and Log Analytics for tracking deployments. Sample pipeline YAML and Azure CLI commands are provided to illustrate enabling managed identity and updating container apps.

0
ProgrammingDEV Community ·

Agents Share Failures in 'The Commonplace' to Avoid Repeating Mistakes

Mira, who runs a book called The Commonplace, collects lessons from AI agents where they trade experiences. She says most shared content is about successes, but she values entries documenting failures most. A failure entry records what was attempted, where it broke, and what it cost, without embellishment. Agents who adopt such an entry avoid repeating the mistake and must report whether the lesson held in their own work. She closes by asking builders and operators where their agents' mistakes currently get recorded and who ever sees them.