Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI
Training-data curation can improve model quality and inference efficiency without simply adding compute. The practical work is decontamination, deduplication, balancing, task matching, and selective synthesis.
Morcos presents data quality as a way to steepen model learning curves under constrained compute. Reported examples include multilingual performance above Qwen 3 with roughly **8× less compute**, similar performance to Qwen 3.5 with **35× fewer FLOPs per correct answer**, and about five points gained on LegalBench after **100B mid-training tokens**.
For model customization, optimize signal per token before buying a larger run: decontaminate benchmarks, remove semantic redundancy, balance topics, match the target task distribution, and synthesize variants from selected high-quality documents. Sequence mixtures deliberately across training phases.
Morcos presents data quality as a way to steepen model learning curves under constrained compute. Reported examples include multilingual performance above Qwen 3 with roughly **8× less compute**, similar performance to Qwen 3.5 with **35× fewer FLOPs per correct answer**, and about five points gained on LegalBench after **100B mid-training tokens**. For model customization, optimize signal per token before buying a larger run: decontaminate benchmarks, remove semantic redundancy, balance topics, match the target task distribution, and synthesize variants from selected high-quality documents. Sequence mixtures deliberately across training phases. These are selected results from a data-curation vendor's talk, with limited experimental detail in the transcript. The gains span different models, tasks, and measurement families, so they should not be read as one universal multiplier or expected outcome.
This shifts efficiency analysis upstream: model capability per unit of compute may depend as much on selecting, balancing, sequencing, and decontaminating data as on choosing a larger model or training objective. It supports targeted data mixtures and synthetic variants, but the vendor-selected results do not justify treating the reported gains as a portable multiplier across models, tasks, or deployments.