LoRA: Why a Rank-8 Weight Update Can Be Enough for Fine-Tuning

Fine-tuning very large models usually requires storing a full new copy of billions of parameters for each task. LoRA, introduced by Hu et al. in 2021, sidesteps this by freezing the pretrained weights and training two small low-rank matrices whose product approximates the needed update. The approach is motivated by the observation that the weight changes from adaptation tend to lie in a few directions, so a small rank such as 8 can suffice. On GPT-3 175B, the authors report roughly 10,000 times fewer trainable parameters and about 3 times less training GPU memory than full fine-tuning, with comparable or better quality. The low-rank matrices can also be merged back into the base weights after training, so inference incurs no extra latency.