INT4 Quantization Boosts Decode Speed But Fails to Accelerate Prefill

INT4 weight-only quantization (W4A16) reduces memory bandwidth usage rather than floating-point computation, making it highly effective for decode operations but not for prefill. During decode, each weight matrix is read once per token, making the process memory-bandwidth-bound, so cutting weight size fourfold yields roughly a fourfold speedup at low concurrency. Prefill, however, processes many tokens simultaneously against the same weights, pushing arithmetic intensity well past the GPU's roofline ridge point and into compute-bound territory where W4A16 offers no benefit. In fact, the dequantization overhead of unpacking INT4 back to FP16 slightly reduces prefill performance compared to native FP16. Engineers needing compute-bound speedups should consider activation quantization methods such as FP8 (W8A8) or W4A8, while INT4 remains useful primarily as a capacity and memory-efficiency tool.