DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

DataOrchestra routes each data chunk to drop, preserve, or clean, then chooses programmatic edits or LLM rewrites and generates rewrite instructions. Models from **0.5B to 7B** trained on its output showed stable average gains across **11 benchmarks**.

Builders preparing training corpora should reconsider corpus-wide cleaning rules. Per-example routing can reserve expensive rewriting for material that needs it, and the reported math continued-pretraining results also exceeded stronger processing baselines.

The abstract gives no gain sizes, processing-cost figures, or operational complexity. It therefore supports the routing principle more clearly than any estimate of whether implementing the orchestrator pays off for a particular dataset.