Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
Controlled pre-training experiments suggest varied reformulations can teach facts more efficiently than repeating documents under the same token budget, though paraphrasing gains depend on batch size.
Controlled experiments found that repetition remains necessary for knowledge acquisition, but shifting a fixed token budget from repeated documents to **auxiliary views** improved learning, including factual recall.
For builders training or adapting models, data diversity may matter at the representation level: supply contextual or foundational reformulations instead of spending every extra token on duplicates. The generating teacher’s strength was not decisive.
Controlled experiments found that repetition remains necessary for knowledge acquisition, but shifting a fixed token budget from repeated documents to **auxiliary views** improved learning, including factual recall. For builders training or adapting models, data diversity may matter at the representation level: supply contextual or foundational reformulations instead of spending every extra token on duplicates. The generating teacher’s strength was not decisive. The effect is conditional: paraphrasing helped only at **smaller batch sizes**, and the abstract does not quantify gains or establish how well the recipe transfers to production-scale training.
This turns the broad data-quality case into a narrower training recipe: repetition still matters, but auxiliary reformulations can use a fixed token budget better than duplicates. It supports representation-level diversity while adding an important batch-size dependency and leaving production-scale transfer unresolved.