MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
MuonGS changes Muon pretraining by projecting out each update’s radial component without fixing weight norms. It reports lower long-run validation perplexity than MuonH, including an estimated 15% token-equivalent speedup in one dense-model comparison.
MuonGS uses a Gram–Schmidt projection to remove the radial part of each Muon update without fixing the weight norm. The paper proves a **√t** upper bound on norm growth under stated conditions and reports up to **15%** estimated token-equivalent speedup at matched validation loss in a dense model.
For teams training language models, MuonGS is an optimizer candidate to compare when long training horizons matter. The reported experiments include dense models at **160M and 400M parameters** and a **135M-active / 1B-total MoE** model.
MuonGS uses a Gram–Schmidt projection to remove the radial part of each Muon update without fixing the weight norm. The paper proves a **√t** upper bound on norm growth under stated conditions and reports up to **15%** estimated token-equivalent speedup at matched validation loss in a dense model. For teams training language models, MuonGS is an optimizer candidate to compare when long training horizons matter. The reported experiments include dense models at **160M and 400M parameters** and a **135M-active / 1B-total MoE** model. The speedup is an estimate from one dense-model comparison, and the theoretical guarantees depend on stated assumptions. The material does not establish that the result transfers to other architectures or training setups.
MuonGS offers a way to limit optimizer-driven norm growth without fixing weight norms, with a reported token-equivalent speedup in a bounded set of model experiments. The result makes optimizer choice worth comparing for long training runs, but does not establish that the gain transfers across architectures or setups.