Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
Controlled pre-training experiments suggest varied reformulations can teach facts more efficiently than repeating documents under the same token budget, though paraphrasing gains depend on batch size.
Controlled experiments found that repetition remains necessary for knowledge acquisition, but shifting a fixed token budget from repeated documents to **auxiliary views** improved learning, including factual recall.
For builders training or adapting models, data diversity may matter at the representation level: supply contextual or foundational reformulations instead of spending every extra token on duplicates. The generating teacher’s strength was not decisive.
Controlled experiments found that repetition remains necessary for knowledge acquisition, but shifting a fixed token budget from repeated documents to **auxiliary views** improved learning, including factual recall. For builders training or adapting models, data diversity may matter at the representation level: supply contextual or foundational reformulations instead of spending every extra token on duplicates. The generating teacher’s strength was not decisive. The effect is conditional: paraphrasing helped only at **smaller batch sizes**, and the abstract does not quantify gains or establish how well the recipe transfers to production-scale training.
This makes the broad case for higher-information training data more specific: under a fixed token budget, varied representations of the same knowledge can outperform further duplication, while repetition still remains necessary. It also narrows the recipe by showing a batch-size dependency and leaves production-scale transfer unresolved, so auxiliary views are a testable mixture choice rather than a universal replacement for repeated exposure.