Why INT8 Quantization Fails for Large Language Models and How FP8 Helps

Quantizing large language models from 16-bit to 8-bit can halve memory usage, but a single extreme activation value can distort the entire scaling grid, degrading precision for the vast majority of normal values. Research by Tim Dettmers and colleagues in 2022 found that in models around 6.7 billion parameters, roughly 150,000 outlier activations per sequence appeared, yet these were concentrated in just about six feature dimensions representing only 0.1% of all values. Despite their small share, removing these outlier dimensions significantly harmed model performance, revealing that the core challenge is not bit-width reduction alone. Developers must treat bit width, number representation format, and scaling granularity as three distinct design choices when quantizing LLMs. This understanding has driven a progression from basic INT8 approaches toward more sophisticated methods, including floating-point 8-bit formats with finer-grained scaling strategies.