Discriminative Fine-Tuning: Why Different Model Layers Need Different Learning Rates

A developer fine-tuning a DETR-style object detector for single-class face detection applied a technique called discriminative fine-tuning, assigning the backbone a learning rate of 5e-6 and the decoder head 1e-4 — a 20x difference. The approach addresses a core flaw in standard fine-tuning, which assumes all layers should update at the same rate. Using a uniform learning rate risks either degrading the backbone's general-purpose features through catastrophic forgetting or slowing the task-specific head's convergence to a crawl. Discriminative fine-tuning, originally introduced in ULMFiT by Howard and Ruder in 2018, assigns lower learning rates to earlier, more general layers and higher rates to later, task-specific ones. The technique is now widely used across transformer fine-tuning pipelines, including BERT, MAE, and BEiT-style vision models.