Muon Optimizer Treats Neural Network Weights as Matrices, Outpaces AdamW

Muon is a neural network optimizer that differs from the widely used AdamW by treating weight matrices as linear transformations rather than collections of independent scalar values. While AdamW applies element-wise gradient statistics, Muon uses singular value decomposition to preserve gradient directions while equalizing their magnitudes — a process called update orthogonalization. The optimizer emerged in October 2024 from the NanoGPT speedrunning competition, where a Muon-based run improved training speed by roughly 35% over the previous record. It was developed collaboratively by Keller Jordan and contributors including Jeremy Bernstein, Yuchen Jin, and Vlado Boza, each refining different aspects of the method. Muon has since scaled to multi-billion-parameter language models and been integrated into the mainstream PyTorch optimization stack.