LoRA matches full fine-tuning on most tasks, but full training wins on large new-knowledge workloads

Nodus reports that LoRA performs about as well as full fine-tuning on typical instruction and reasoning datasets and in reinforcement learning, provided the adapter covers all layers and uses a higher learning rate. Two studies, Biderman et al. (2024) and Thinking Machines' LoRA Without Regret, show that LoRA trails full fine-tuning once data volume exceeds the adapter's capacity, as in continued pretraining on billions of tokens. The authors attribute the gap to full fine-tuning's ability to learn weight changes of far higher rank. Memory costs also differ sharply: full fine-tuning needs roughly 16 bytes per parameter, about 7.7 times the footprint of LoRA on Qwen3-8B and Qwen3-32B by the article's estimates. By that math, a LoRA run on Qwen3-8B fits on a single 48 GB GPU, while full fine-tuning requires multiple GPUs.