arXivPaperNeeds Review
Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining
A task-agnostic influence measure tracks which examples steer pretraining toward final parameters without choosing a downstream eval. It reveals a literature-to-STEM shift across training.
arXiv
Source Summary
The method defines an example’s influence by how much its gradient update reduces squared distance to the run’s final parameters. It estimates that value from intermediate checkpoints **without retraining** or selecting a downstream task.
Practical Implication
Across **18 Pythia and PolyPythia configurations**, influential data changed over time: literature-related examples aligned more strongly early, while STEM data aligned more strongly later. Training-pipeline builders can use this trajectory view alongside task-specific attribution when inspecting data mixtures.
Agent-Ready Context
The method defines an example’s influence by how much its gradient update reduces squared distance to the run’s final parameters. It estimates that value from intermediate checkpoints **without retraining** or selecting a downstream task. Across **18 Pythia and PolyPythia configurations**, influential data changed over time: literature-related examples aligned more strongly early, while STEM data aligned more strongly later. Training-pipeline builders can use this trajectory view alongside task-specific attribution when inspecting data mixtures. The crossover was qualitative and broadly consistent, but the measure is tied to the final parameters of a particular run. It does not show which examples improve a coding agent, nor replace downstream capability and reliability evaluations.
Context Map
benchmarkresearchdata#benchmark-integrityUncertainty
The crossover was qualitative and broadly consistent, but the measure is tied to the final parameters of a particular run. It does not show which examples improve a coding agent, nor replace downstream capability and reliability evaluations.