# MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

Source: [arXiv](https://arxiv.org/abs/2610.37647v1)  
Feed7 permalink: https://feed7.dev/p/2610-37647v1-0r1fv8r  
Published: 2026-10-27T17:56:40.000Z  
Trust: Needs Review (needs_review)

## Why Included

MuonGS changes Muon pretraining by projecting out each update’s radial component without fixing weight norms. It reports lower long-run validation perplexity than MuonH, including an estimated 15% token-equivalent speedup in one dense-model comparison.

## Source Summary

MuonGS uses a Gram–Schmidt projection to remove the radial part of each Muon update without fixing the weight norm. The paper proves a **√t** upper bound on norm growth under stated conditions and reports up to **15%** estimated token-equivalent speedup at matched validation loss in a dense model.

## Practical Implication

For teams training language models, MuonGS is an optimizer candidate to compare when long training horizons matter. The reported experiments include dense models at **160M and 400M parameters** and a **135M-active / 1B-total MoE** model.

## Agent-Ready Context

MuonGS uses a Gram–Schmidt projection to remove the radial part of each Muon update without fixing the weight norm. The paper proves a **√t** upper bound on norm growth under stated conditions and reports up to **15%** estimated token-equivalent speedup at matched validation loss in a dense model.

For teams training language models, MuonGS is an optimizer candidate to compare when long training horizons matter. The reported experiments include dense models at **160M and 400M parameters** and a **135M-active / 1B-total MoE** model.

The speedup is an estimate from one dense-model comparison, and the theoretical guarantees depend on stated assumptions. The material does not establish that the result transfers to other architectures or training setups.

## Connected Context

Feed7 judgment across 944 accumulated Signals:

MuonGS offers a way to limit optimizer-driven norm growth without fixing weight norms, with a reported token-equivalent speedup in a bounded set of model experiments. The result makes optimizer choice worth comparing for long training runs, but does not establish that the gain transfers across architectures or setups.

- [When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning](https://feed7.dev/p/2608-31118v1-1abyqwd) — The ontology study finds model-size rankings depend on the task and architecture; MuonGS likewise has bounded experimental evidence across specific model configurations. Together they support matched, workload-specific comparisons rather than assuming one training or model choice generalizes.

## Context Map

- Layer: model
- Domains: research, data
- Topics: open-models, model-selection

## Uncertainty

- The speedup is an estimate from one dense-model comparison, and the theoretical guarantees depend on stated assumptions. The material does not establish that the result transfers to other architectures or training setups.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.