# The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Source: [AI Engineer](https://www.youtube.com/watch?v=KhYifX22yhE)  
Feed7 permalink: https://feed7.dev/p/the-messy-reality-of-scale-synthetic-data-and-pre-training-marah-abdin-r-15307vg  
Published: 2026-07-26T01:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

poolside treats synthetic data and training observability as one scaling problem: diversify scarce code data, decompose hard generation tasks, and verify numerical invariants before failures compound.

## Source Summary

Laguna XS.2 used **13% synthetic data** during pre-training. The team also found that BF16 accumulation near the LM head stalled Laguna M.1; moving it to **FP32 restored convergence**. Laguna S scales the revised recipe to **118B total / 8B active parameters** and **30T tokens on 4,000 GPUs**.

## Practical Implication

For model-training pipelines, treat data mix, generator orchestration, numerical precision, and replica checks as one system. Break generation tasks into smaller agent stages, validate outputs, hash replicated weights, and monitor activation and gradient trends.

## Agent-Ready Context

Laguna XS.2 used **13% synthetic data** during pre-training. The team also found that BF16 accumulation near the LM head stalled Laguna M.1; moving it to **FP32 restored convergence**. Laguna S scales the revised recipe to **118B total / 8B active parameters** and **30T tokens on 4,000 GPUs**.

For model-training pipelines, treat data mix, generator orchestration, numerical precision, and replica checks as one system. Break generation tasks into smaller agent stages, validate outputs, hash replicated weights, and monitor activation and gradient trends.

The Laguna S results shown are base-model evaluations before post-training, and the model was not yet public. Its stronger coding results do not generalize across every benchmark; the speakers report a remaining knowledge-data gap on MMLU Pro.

## Context Map

- Layer: model
- Domains: coding, data
- Topics: open-models, coding-agents

## Uncertainty

- The Laguna S results shown are base-model evaluations before post-training, and the model was not yet public. Its stronger coding results do not generalize across every benchmark; the speakers report a remaining knowledge-data gap on MMLU Pro.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.