The Base Model Is Dead — Varun Singh, Arcee AI
Base-model data is shifting from broad web imitation toward code, reasoning, and agent-task priors. The unresolved choice is how early to introduce synthetic and instruction-shaped data.
The talk contrasts GPT-3’s roughly **85% web-derived mix** with MAI Thinking 1 at **15% web text**. Newer recipes emphasize code, STEM, reasoning traces, and task-shaped data that better prepare models for downstream RL.
When selecting or training a model for agents, evaluate its pre-RL skill coverage, not just general knowledge. **NeMoTron 3 Ultra** pulls SFT-style data into pre-training, while synthetic rephrasing can expose the same information in several forms.
The talk contrasts GPT-3’s roughly **85% web-derived mix** with MAI Thinking 1 at **15% web text**. Newer recipes emphasize code, STEM, reasoning traces, and task-shaped data that better prepare models for downstream RL. When selecting or training a model for agents, evaluate its pre-RL skill coverage, not just general knowledge. **NeMoTron 3 Ultra** pulls SFT-style data into pre-training, while synthetic rephrasing can expose the same information in several forms. There is no settled recipe: MAI Thinking 1 deliberately avoids synthetic model-generated data, while NeMoTron leans into it. Synthetic data can degrade a model when used indiscriminately, and it remains unclear how far RL can displace supervised learning for language.
This shifts model selection upstream from headline capability and post-training performance to the skill distribution created during pre-training. Against the gateway candidates, it implies that context size, reasoning controls, and tier labels are insufficient without workload tests that reveal pre-RL coverage. It also prevents a simple synthetic-data rule: the contrasted recipes support evaluating data composition and outcomes, not assuming synthetic inclusion is inherently beneficial or harmful.