arXivPaperNeeds Review
Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap
Muon’s pretraining advantage may not carry through fine-tuning: at larger data budgets, AdamW performs better downstream. Optimizer choice may depend on both training scale and model size.
arXiv · Nov 2, 2026
Source Summary
The study finds that **Muon** does better after fine-tuning at smaller pretraining data budgets, while **AdamW** takes the lead beyond a threshold. As model size grows, AdamW takes over at smaller pretraining token-to-parameter ratios.
Practical Implication
If you train models, evaluate optimizers on downstream fine-tuning outcomes as well as pretraining loss, and account for both data budget and model size.
Agent-Ready Context
The study finds that **Muon** does better after fine-tuning at smaller pretraining data budgets, while **AdamW** takes the lead beyond a threshold. As model size grows, AdamW takes over at smaller pretraining token-to-parameter ratios. If you train models, evaluate optimizers on downstream fine-tuning outcomes as well as pretraining loss, and account for both data budget and model size. The paper explains the shift using a margin-to-spread measure and proves a related result in a linear associative memory. The supplied material gives no threshold values for practical model sizes or training setups.
Context Map
modelresearch#model-selection#open-modelsUncertainty
The paper explains the shift using a margin-to-spread measure and proves a related result in a linear associative memory. The supplied material gives no threshold values for practical model sizes or training setups.