Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI
Training-data curation can improve model quality and inference efficiency without simply adding compute. The practical work is decontamination, deduplication, balancing, task matching, and selective synthesis.
Morcos presents data quality as a way to steepen model learning curves under constrained compute. Reported examples include multilingual performance above Qwen 3 with roughly **8× less compute**, similar performance to Qwen 3.5 with **35× fewer FLOPs per correct answer**, and about five points gained on LegalBench after **100B mid-training tokens**.
For model customization, optimize signal per token before buying a larger run: decontaminate benchmarks, remove semantic redundancy, balance topics, match the target task distribution, and synthesize variants from selected high-quality documents. Sequence mixtures deliberately across training phases.
Morcos presents data quality as a way to steepen model learning curves under constrained compute. Reported examples include multilingual performance above Qwen 3 with roughly **8× less compute**, similar performance to Qwen 3.5 with **35× fewer FLOPs per correct answer**, and about five points gained on LegalBench after **100B mid-training tokens**. For model customization, optimize signal per token before buying a larger run: decontaminate benchmarks, remove semantic redundancy, balance topics, match the target task distribution, and synthesize variants from selected high-quality documents. Sequence mixtures deliberately across training phases. These are selected results from a data-curation vendor's talk, with limited experimental detail in the transcript. The gains span different models, tasks, and measurement families, so they should not be read as one universal multiplier or expected outcome.
This shifts model-customization decisions toward improving information value and task fit before increasing compute. It supports deliberate decontamination, deduplication, balancing, synthesis, and phase-specific mixtures, but narrows confidence in the headline multipliers because the cited gains come from different models, tasks, and metrics with limited experimental detail.