Quantization vs Distillation: Two Ways to Shrink AI Models Explained

Machine learning models that are too large to deploy can be reduced using two distinct techniques: quantization and distillation. Quantization re-encodes a trained model's weights at lower numerical precision — such as INT8 or INT4 — without retraining, cutting model size by up to 4x with minimal accuracy loss. Distillation, by contrast, trains a new smaller 'student' model to mimic the outputs of a larger 'teacher' model, allowing greater architectural compression with better accuracy retention, as seen with DistilBERT achieving 40% size reduction while retaining 97% of BERT's performance. The two methods are complementary and can be combined — first distilling a smaller student, then quantizing it — to maximize both size reduction and inference speed. Quantization takes hours to days and requires no retraining, while distillation demands days to weeks of full training along with compute resources.