Gradient Clipping Lowered Accuracy in ML Experiments, Likely by Amplifying Weight Decay

A machine learning practitioner tested whether gradient clipping, a common default in training loops, provides stability like learning rate warmup. Across five paired seeds, clipping at 1.0 cost accuracy at a tuned learning rate and worsened results at a higher one, with a tighter 0.3 threshold causing the largest drop. Instrumentation showed gradient norms were stable and clipping was active on every step, so it was acting as a constant gradient attenuator rather than a safeguard. The author argues that clipping shrinks the loss gradient but not Adam's weight decay term, effectively intensifying regularization. They propose a follow-up test removing weight decay from the most affected configuration to check this explanation.