# Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

Source: [arXiv](https://arxiv.org/abs/2611.03633v1)  
Feed7 permalink: https://feed7.dev/p/2611-03633v1-05w62ui  
Published: 2026-11-02T18:56:43.000Z  
Trust: Needs Review (needs_review)

## Why Included

Muon’s pretraining advantage may not carry through fine-tuning: at larger data budgets, AdamW performs better downstream. Optimizer choice may depend on both training scale and model size.

## Source Summary

The study finds that **Muon** does better after fine-tuning at smaller pretraining data budgets, while **AdamW** takes the lead beyond a threshold. As model size grows, AdamW takes over at smaller pretraining token-to-parameter ratios.

## Practical Implication

If you train models, evaluate optimizers on downstream fine-tuning outcomes as well as pretraining loss, and account for both data budget and model size.

## Agent-Ready Context

The study finds that **Muon** does better after fine-tuning at smaller pretraining data budgets, while **AdamW** takes the lead beyond a threshold. As model size grows, AdamW takes over at smaller pretraining token-to-parameter ratios.

If you train models, evaluate optimizers on downstream fine-tuning outcomes as well as pretraining loss, and account for both data budget and model size.

The paper explains the shift using a margin-to-spread measure and proves a related result in a linear associative memory. The supplied material gives no threshold values for practical model sizes or training setups.

## Context Map

- Layer: model
- Domains: research
- Topics: model-selection, open-models

## Uncertainty

- The paper explains the shift using a margin-to-spread measure and proves a related result in a linear associative memory. The supplied material gives no threshold values for practical model sizes or training setups.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.