The Base Model Is Dead — Varun Singh, Arcee AI
Base-model data is shifting from broad web imitation toward code, reasoning, and agent-task priors. The unresolved choice is how early to introduce synthetic and instruction-shaped data.
The talk contrasts GPT-3’s roughly **85% web-derived mix** with MAI Thinking 1 at **15% web text**. Newer recipes emphasize code, STEM, reasoning traces, and task-shaped data that better prepare models for downstream RL.
When selecting or training a model for agents, evaluate its pre-RL skill coverage, not just general knowledge. **NeMoTron 3 Ultra** pulls SFT-style data into pre-training, while synthetic rephrasing can expose the same information in several forms.
The talk contrasts GPT-3’s roughly **85% web-derived mix** with MAI Thinking 1 at **15% web text**. Newer recipes emphasize code, STEM, reasoning traces, and task-shaped data that better prepare models for downstream RL. When selecting or training a model for agents, evaluate its pre-RL skill coverage, not just general knowledge. **NeMoTron 3 Ultra** pulls SFT-style data into pre-training, while synthetic rephrasing can expose the same information in several forms. There is no settled recipe: MAI Thinking 1 deliberately avoids synthetic model-generated data, while NeMoTron leans into it. Synthetic data can degrade a model when used indiscriminately, and it remains unclear how far RL can displace supervised learning for language.
This shifts model selection beneath gateway features and headline benchmarks to the capabilities and data composition established before RL. It explains why nominally general models may differ sharply on agent work, while refusing a universal recipe: synthetic data, SFT-like pre-training, and RL remain choices whose value must be established on the intended workload rather than inferred from model branding or context size.