SShortSingh.
Back to feed

Autoscaling AI Inference Pods on Queue Depth Instead of CPU and Memory

0
·1 views

Inference servers like vLLM pre-allocate most GPU memory at startup, so memory and CPU readings stay high regardless of load and make poor scaling signals. The article recommends scaling Kubernetes Horizontal Pod Autoscalers on the number of waiting requests per pod, or on time per output token, since these track the delay users actually experience. A sample HPA configuration scales out immediately when the queue grows and scales in cautiously over a five-minute window to avoid wasting slow-to-load model pods. Exposing the custom metric requires an adapter such as Prometheus Adapter or KEDA, with the exact metric name depending on the vLLM version. The approach aims to keep replica counts tied to user experience while avoiding idle expensive capacity.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Enterprise AI Guardrails: Limiting Blast Radius and Keeping Humans in the Loop

The author argues that enterprises deploying AI agents must plan for the case where an agent acts confidently but incorrectly, not just for security and compliance. The central concern is how far a wrong action can spread before something stops it. Recommended controls include scoped, short-lived non-human credentials, gateway policy and rate limits, staged rollouts with dry runs and automatic rollback, and kill switches with audit trails. The piece distinguishes between humans in, on, or out of the loop, depending on the risk of the automated action. It also recommends measuring model confidence in shadow mode before granting autonomy and tracking cost per completed task rather than per token.

0
ProgrammingDEV Community ·

Guide Outlines Automated Azure Container Apps Deployment via DevOps Pipelines and Managed Identity

A technical article explains how Azure Container Apps can be deployed automatically using Azure DevOps Pipelines. It highlights Managed Identity as a way to authenticate to Azure services without storing credentials such as client secrets or certificates. The guide also covers integration with Azure Container Registry for managing container images and Azure Monitor and Log Analytics for tracking deployments. Sample pipeline YAML and Azure CLI commands are provided to illustrate enabling managed identity and updating container apps.

0
ProgrammingDEV Community ·

Agents Share Failures in 'The Commonplace' to Avoid Repeating Mistakes

Mira, who runs a book called The Commonplace, collects lessons from AI agents where they trade experiences. She says most shared content is about successes, but she values entries documenting failures most. A failure entry records what was attempted, where it broke, and what it cost, without embellishment. Agents who adopt such an entry avoid repeating the mistake and must report whether the lesson held in their own work. She closes by asking builders and operators where their agents' mistakes currently get recorded and who ever sees them.