The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside
poolside treats synthetic data and training observability as one scaling problem: diversify scarce code data, decompose hard generation tasks, and verify numerical invariants before failures compound.
Laguna XS.2 used **13% synthetic data** during pre-training. The team also found that BF16 accumulation near the LM head stalled Laguna M.1; moving it to **FP32 restored convergence**. Laguna S scales the revised recipe to **118B total / 8B active parameters** and **30T tokens on 4,000 GPUs**.
For model-training pipelines, treat data mix, generator orchestration, numerical precision, and replica checks as one system. Break generation tasks into smaller agent stages, validate outputs, hash replicated weights, and monitor activation and gradient trends.
Laguna XS.2 used **13% synthetic data** during pre-training. The team also found that BF16 accumulation near the LM head stalled Laguna M.1; moving it to **FP32 restored convergence**. Laguna S scales the revised recipe to **118B total / 8B active parameters** and **30T tokens on 4,000 GPUs**. For model-training pipelines, treat data mix, generator orchestration, numerical precision, and replica checks as one system. Break generation tasks into smaller agent stages, validate outputs, hash replicated weights, and monitor activation and gradient trends. The Laguna S results shown are base-model evaluations before post-training, and the model was not yet public. Its stronger coding results do not generalize across every benchmark; the speakers report a remaining knowledge-data gap on MMLU Pro.