Balancing Throughput and Latency in LLM Inference Serving

An interview question for LLM inference engineers asks how to optimize a system for high throughput and low latency. The guidance stresses separating per-request latency from fleet-wide tokens per second, and treating prefill (compute-bound) and decode (memory-bandwidth-bound) with different techniques. Recommended levers include continuous batching, paged KV cache allocation with prefix reuse, and quantization alongside fused attention kernels. Overlapping data transfers with compute, using CUDA graphs, and applying speculative decoding or parallelism are also highlighted. Common pitfalls named include ignoring tail latency percentiles, offloading KV cache without measuring PCIe costs, and growing batch sizes without limit.