What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
A study of six point-in-time models finds that knowledge with more variable representation across web years often improves faster later in pretraining. Its temporal variability score offers a diagnostic beyond aggregate benchmark results.
The authors define a temporal variability score (TVS) from the variation in an item’s negative log-likelihood across point-in-time models. High-TVS items improved faster later in pretraining and ended ahead in **34 of 36 model-benchmark settings**, by an average **7.3 points** of chance-normalized accuracy.
For teams evaluating pretrained models, TVS could help explain which knowledge items change most over training and why aggregate scores miss some learning patterns. The effect was strongest on knowledge-intensive tasks.
The authors define a temporal variability score (TVS) from the variation in an item’s negative log-likelihood across point-in-time models. High-TVS items improved faster later in pretraining and ended ahead in **34 of 36 model-benchmark settings**, by an average **7.3 points** of chance-normalized accuracy. For teams evaluating pretrained models, TVS could help explain which knowledge items change most over training and why aggregate scores miss some learning patterns. The effect was strongest on knowledge-intensive tasks. The study covers **six models** from multiple open model families and six benchmarks. TVS indicates how consistently knowledge appears across web snapshots; the material does not establish that it predicts performance in a particular application.
This introduces temporal variability across point-in-time models as a signal for which knowledge items improve during pretraining, with a reported association across many model-benchmark settings. It adds a training-dynamics lens to evaluation, while leaving open whether the signal predicts performance in a specific application.