Int4 Per-Row Quantization Cuts Transformer Size 6x With No Accuracy Loss

A series of experiments on a small transformer model found that 71.7% of its parameters resided in the embedding lookup table, making it the primary target for compression. Quantizing the embedding to int4 precision with per-tensor scaling caused a notable accuracy drop, but switching to per-row scaling — adding just 16 KB of extra data — recovered nearly all of that loss. The int4 per-row configuration matched the full fp32 model's top-1 mixture score of 0.384 while reducing model size from 5.72 MB to 0.93 MB, a 6.2x reduction with no measurable performance cost. The findings highlight how sweeping only one variable at a time — such as bit-width without varying granularity — can produce misleading conclusions, since the benefit of per-row scaling only becomes visible at lower bit-widths. The experiment required no retraining and was conducted across just seven runs, suggesting the cross-product sweep approach is practical even in low-resource settings.