Autoscaling AI Inference Pods on Queue Depth Instead of CPU and Memory
Inference servers like vLLM pre-allocate most GPU memory at startup, so memory and CPU readings stay high regardless of load and make poor scaling signals. The article recommends scaling Kubernetes Horizontal Pod Autoscalers on the number of waiting requests per pod, or on time per output token, since these track the delay users actually experience. A sample HPA configuration scales out immediately when the queue grows and scales in cautiously over a five-minute window to avoid wasting slow-to-load model pods. Exposing the custom metric requires an adapter such as Prometheus Adapter or KEDA, with the exact metric name depending on the vLLM version. The approach aims to keep replica counts tied to user experience while avoiding idle expensive capacity.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in