Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI
Training-data curation can improve model quality and inference efficiency without simply adding compute. The practical work is decontamination, deduplication, balancing, task matching, and selective synthesis.
Morcos presents data quality as a way to steepen model learning curves under constrained compute. Reported examples include multilingual performance above Qwen 3 with roughly **8× less compute**, similar performance to Qwen 3.5 with **35× fewer FLOPs per correct answer**, and about five points gained on LegalBench after **100B mid-training tokens**.
For model customization, optimize signal per token before buying a larger run: decontaminate benchmarks, remove semantic redundancy, balance topics, match the target task distribution, and synthesize variants from selected high-quality documents. Sequence mixtures deliberately across training phases.
Morcos presents data quality as a way to steepen model learning curves under constrained compute. Reported examples include multilingual performance above Qwen 3 with roughly **8× less compute**, similar performance to Qwen 3.5 with **35× fewer FLOPs per correct answer**, and about five points gained on LegalBench after **100B mid-training tokens**. For model customization, optimize signal per token before buying a larger run: decontaminate benchmarks, remove semantic redundancy, balance topics, match the target task distribution, and synthesize variants from selected high-quality documents. Sequence mixtures deliberately across training phases. These are selected results from a data-curation vendor's talk, with limited experimental detail in the transcript. The gains span different models, tasks, and measurement families, so they should not be read as one universal multiplier or expected outcome.
This moves model-efficiency decisions upstream from routing and model size to the composition and sequencing of training data. It reinforces workload-matched provenance, redundancy removal, and decontamination as selection criteria, while making the reported compute advantages hypotheses to reproduce rather than portable multipliers because the examples span unlike models, tasks, and metrics.