AI EngineerVideoSource Linked

The Base Model Is Dead — Varun Singh, Arcee AI

Base-model data is shifting from broad web imitation toward code, reasoning, and agent-task priors. The unresolved choice is how early to introduce synthetic and instruction-shaped data.

AI Engineer · Jul 31, 2026

Open Source Open Markdown Open JSON

Source Summary

The talk contrasts GPT-3’s roughly **85% web-derived mix** with MAI Thinking 1 at **15% web text**. Newer recipes emphasize code, STEM, reasoning traces, and task-shaped data that better prepare models for downstream RL.

Practical Implication

When selecting or training a model for agents, evaluate its pre-RL skill coverage, not just general knowledge. **NeMoTron 3 Ultra** pulls SFT-style data into pre-training, while synthetic rephrasing can expose the same information in several forms.

Agent-Ready Context

The talk contrasts GPT-3’s roughly **85% web-derived mix** with MAI Thinking 1 at **15% web text**. Newer recipes emphasize code, STEM, reasoning traces, and task-shaped data that better prepare models for downstream RL.

When selecting or training a model for agents, evaluate its pre-RL skill coverage, not just general knowledge. **NeMoTron 3 Ultra** pulls SFT-style data into pre-training, while synthetic rephrasing can expose the same information in several forms.

There is no settled recipe: MAI Thinking 1 deliberately avoids synthetic model-generated data, while NeMoTron leans into it. Synthetic data can degrade a model when used indiscriminately, and it remains unclear how far RL can displace supervised learning for language.

Connected Context · Feed7 Judgment

This shifts model selection beneath post-training results: agent readiness may depend on whether pre-training already covers code, STEM, reasoning, and task-shaped behavior. It makes gateway tiers, context sizes, and reasoning toggles insufficient selection criteria on their own. The conflicting MAI and NeMoTron recipes also prevent a general rule about synthetic data, so provenance and workload evaluation matter more than adopting either synthetic-heavy or synthetic-free training as doctrine.

Introducing Grok 4.5Grok 4.5’s benchmark exclusion because training included an earlier code snapshot reinforces this signal’s demand to inspect pre-training composition and provenance, not interpret downstream scores without qualification.GPT 5.6 Sol, Luna, and Terra now available on AI GatewayGateway tiering offers convenient flagship, balanced, and lower-cost routes, but this signal says those labels must be supplemented by testing whether each model’s pre-RL skill coverage matches the agent workload.DeepSeek V4 Flash now runs updated weights on AI GatewayThe reported score jump after a weight replacement shows that underlying training changes can materially alter a fixed model route, while this signal cautions that one downstream benchmark does not reveal the broader data recipe or skill coverage.What's Next After RLHF? — Diogo Almeida, TypeSafe AIThe RLHF critique supports this signal’s unresolved boundary between supervised learning and RL: post-training for persuasive interaction does not establish the dependable decision-making skills needed for autonomous agents.

Context Map

modelcoding#reasoning#coding-agents#model-selection

Uncertainty

There is no settled recipe: MAI Thinking 1 deliberately avoids synthetic model-generated data, while NeMoTron leans into it. Synthetic data can degrade a model when used indiscriminately, and it remains unclear how far RL can displace supervised learning for language.