Blackwell GPU Inference Failures Traced to Backend, Runtime, and Memory Config Gaps

Developers testing large language models on NVIDIA Blackwell GPUs (sm_120 and sm_121) have documented how a single configuration change — such as switching MoE backends, adjusting memory utilization, or updating a runtime commit — can dramatically alter model behavior, available KV cache, or system stability. In one case, changing the MoE backend from flashinfer_b12x to marlin on a 96 GB RTX PRO 6000 freed 11.47 GiB of additional KV cache, while the b12x path's memory reservation caused outright failures on 16 GB and 32 GB Blackwell cards. A separate test found that non-deterministic logprob outputs in a 30B MoE model were resolved by disabling fused finalize, and an SGLang regression caused degenerate text generation until a specific upstream commit was applied. On a GB10 DGX Spark, setting GPU memory utilization to 0.92 instead of 0.60 led to an OOM condition invisible to the kernel, ultimately requiring a full power cycle after seven hours. A diagnostic tool called blackwell-doctor has been released to help users log the exact configuration fields needed to reproduce and report such issues accurately.