Why LoRA Dominates Fine-Tuning: The Merge Trick That Eliminates Inference Cost

Parameter-efficient fine-tuning methods freeze a pretrained model and train only a small fraction of new weights, avoiding the massive memory and storage costs of full fine-tuning a large model. Early adapter methods (Houlsby et al., 2019) inserted small bottleneck modules sequentially within each layer, adding measurable latency at inference since extra computations were required on every token. LoRA instead applies a low-rank correction directly to existing weight matrices in parallel, allowing the trained update to be mathematically merged back into the frozen weights before deployment. After merging, no additional operations exist at inference, giving LoRA zero latency overhead compared to adapters — the core reason it displaced earlier approaches. However, when serving many fine-tuned variants from a single base model, LoRA adapters must remain unmerged and applied as separate branches per request, deliberately reintroducing a small per-token cost in exchange for hosting hundreds of tunings simultaneously.