Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining
A task-agnostic influence measure tracks which examples steer pretraining toward final parameters without choosing a downstream eval. It reveals a literature-to-STEM shift across training.
The method defines an example’s influence by how much its gradient update reduces squared distance to the run’s final parameters. It estimates that value from intermediate checkpoints **without retraining** or selecting a downstream task.
Across **18 Pythia and PolyPythia configurations**, influential data changed over time: literature-related examples aligned more strongly early, while STEM data aligned more strongly later. Training-pipeline builders can use this trajectory view alongside task-specific attribution when inspecting data mixtures.
The method defines an example’s influence by how much its gradient update reduces squared distance to the run’s final parameters. It estimates that value from intermediate checkpoints **without retraining** or selecting a downstream task. Across **18 Pythia and PolyPythia configurations**, influential data changed over time: literature-related examples aligned more strongly early, while STEM data aligned more strongly later. Training-pipeline builders can use this trajectory view alongside task-specific attribution when inspecting data mixtures. The crossover was qualitative and broadly consistent, but the measure is tied to the final parameters of a particular run. It does not show which examples improve a coding agent, nor replace downstream capability and reliability evaluations.
This adds a task-agnostic, checkpoint-based view of which pretraining examples most shaped a particular run, complementing candidates focused on downstream behavior and benchmark design. The changing literature-to-STEM influence pattern argues against treating data contribution as static, but the run-relative measure does not establish agent capability or reliability and therefore remains a diagnostic alongside, not a substitute for, task and workflow evaluations.