GPUStack and vLLM in Production: Capping KV Cache to Prevent Out-of-Memory Crashes
A small on-prem team running four mixed-GPU nodes replaced ad hoc Docker wrappers with GPUStack, a control plane that places models across available GPUs and exposes an OpenAI-compatible endpoint. The platform worked as advertised, but its VRAM placement estimate ignores activation peaks, CUDA graph capture, and real prompt lengths, treating context as a fixed budget rather than a per-sequence cost. Running a 70B model under long-prompt traffic caused repeated worker out-of-memory crashes, which the team traced to KV cache growth multiplied by concurrency. They stabilized deployments by pinning max context length, lowering GPU memory utilization, capping concurrent sequences, and disabling host swap, with a similar approach applied to llama-box on the heterogeneous node. The write-up also notes a separate throughput decline over time that was still under investigation.
