LLM Servers Can OOM on Short Prompts If KV Cache Ignores Output Budget

Self-hosted large language model servers risk out-of-memory crashes when admission control accounts only for prompt length, ignoring the memory cost of the full generation budget. A request with a short prompt but a large max_tokens cap can consume KV cache for thousands of tokens as the model generates output, far exceeding what was estimated at admission. These failures tend to be intermittent because most requests stop generating early, masking the flaw until several long-running requests overlap and exhaust the cache mid-decode. The recommended fix is to reserve memory upfront based on the worst-case context size — prompt tokens plus the maximum output cap — and to treat unset output caps as the model's full context-window limit. Servers should also return a Retry-After header when rejecting requests, preventing clients from triggering retry storms that worsen the overload.