PaperlayerView exact revision on arXiv

Source-linked AI summary

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

Chang Su, Bin Wu, Emine Yilmaz, Yao Lu

arXiv:2610.37631v1cs.AI

TL;DR

How different kinds of knowledge are learned across pretraining remains unclear, and exact-match corpus counts can miss knowledge expressed in different words. The paper uses point-in-time models to measure temporal variability in item likelihoods; high-TVS items improve faster later and outperform low-TVS items in most model–benchmark settings at the end of training.

  • Problem

    How different kinds of knowledge are learned at different pretraining stages is unclear, while exact-match counts can miss knowledge expressed with different wording.

    Source: §1 · p. 1

  • Method

    The paper proposes TVS, an item-level coefficient-of-variation measure of question-and-answer likelihood across point-in-time models.

    Source: §2 · p. 2, §1 · p. 2

  • Results

    High-TVS items outperform low-TVS items in 34 of 36 model–benchmark settings at the end of training, with a 7.3-point average advantage in chance-normalized accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TVS reveals knowledge-acquisition dynamics beyond aggregate scores, without requiring training-corpus access or assigning semantic categories to items.

    Source: §6 · p. 5

    The study covers only likelihood-ranked multiple-choice benchmarks, chosen because they are most suitable for monitoring early checkpoints.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

1 Introduction

The paper asks how knowledge types are learned at different pretraining stages, since aggregate task gains reveal little about the knowledge acquired and exact-match counts miss paraphrases. It uses point-in-time models to measure temporal variability and finds that high-TVS items improve faster later in training, especially on knowledge-intensive tasks.

1 Introduction

  • Exact-match corpus counts depend on query wording and can miss passages that express the same knowledge differently.The paper frames this as a challenge for measuring how often facts or concepts appear in pretraining data.Source: §1 · p. 1
  • TVS uses likelihood variation across point-in-time models to characterize how consistently knowledge appears across yearly web snapshots.It is an item-level diagnostic intended to go beyond exact textual matches.Source: §1 · pp. 1–2
  • The study tracks low-, medium-, and high-TVS items from 16,239 benchmark items across checkpoints of six models.The benchmarks span factual knowledge, scientific reasoning, and commonsense reasoning.Source: §1 · p. 1
  • High-TVS items outperform low-TVS items in 34 of 36 model–benchmark settings at pretraining’s end, with a 7.3-point average advantage in chance-normalized accuracy.The pattern is most robust on knowledge-intensive tasks, including when comparing items of similar difficulty.Source: §1 · p. 2

2 Measuring Temporal Variability

TVS measures relative variation in item likelihood across yearly point-in-time models. Its labels describe likelihood stability, not knowledge categories or prevalence directly.

2 Measuring Temporal Variability

  • TVS is the coefficient of variation of per-token negative log-likelihood for an item’s question-and-gold-answer string across yearly PIT models.The PIT-4B models share architecture, tokenizer, and training setup, but use yearly FineWeb partitions from 2017–2024.Source: §2 · p. 2
  • Low TVS marks items whose yearly models agree relative to overall likelihood, while high TVS marks items whose likelihood depends on training year.The paper calls these groups temporally stable and temporally variable, respectively.Source: §2 · p. 2
  • Normalizing by mean likelihood matters because perplexity variation is entangled with mean perplexity and unnormalized NLL variation mostly selects harder items.TVS measures relative cross-year likelihood variation, not prevalence directly or semantic categories.Source: §2 · p. 2
  • Each yearly model scores the same fixed question–answer string, so time indexes the training snapshot rather than a changing evaluation target.The stable and variable labels are likelihood-based, not a knowledge taxonomy.Source: §2 · p. 2

3 Experimental Setup

The experiments compare likelihood-ranked multiple-choice performance across benchmarks, models, and training checkpoints, using chance-normalized accuracy for evaluation.

3 Experimental Setup

  • The evaluation covers HellaSwag, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, and MMLU, with 16,239 items overall.Items are split into low-, medium-, and high-TVS groups within bins of PIT-4B token length.Source: §3 · p. 2
  • Six models from four open families are evaluated: Pythia, TinyLlama, SmolLM3, and OLMo.The Pythia and OLMo families each contribute two model sizes.Source: §3 · p. 2
  • Each item is scored by ranking answer options by length-normalized likelihood, and results are reported as chance-normalized accuracy.This metric makes benchmarks with different numbers of answer options comparable.Source: §3 · pp. 2–3
  • Checkpoints cover most of each training run, from 0.54B to 11.1T tokens, with one run per trajectory.The OLMo models use their 0724 versions.Source: §3 · p. 2

4 Results

At the final checkpoint, high-TVS items outperform low-TVS items in nearly all model–benchmark settings. Their advantage persists after difficulty controls on knowledge-intensive tasks, while answer-specific TVS predicts later margin growth and contamination checks argue against leakage as the dominant explanation.

4.1 Learning Patterns Diverge Across TVS Groups

  • 2.0 points per tenfold increase in training tokens is the faster high-TVS accuracy growth over the second half of checkpoints.The difference is positive in 26 of 36 settings and all six benchmark and model averages, but is not significant across four family averages (p = .13).Source: §4.1 · p. 3
  • 7.3 points of chance-normalized accuracy is the average final high-minus-low TVS gap, with high-TVS items ahead in 34 of 36 model–benchmark settings.The gap is positive on every benchmark, ranging from 1.7 points on WinoGrande to 16.1 points on PIQA; intervals summarize variation across settings or family averages, not run-to-run uncertainty.Source: §4.1 · p. 3, Table 2 · p. 3

4.2 Does TVS Capture More Than Question Difficulty?

  • +6.9 points of chance-normalized accuracy remains on ARC and MMLU after grouping by TVS residuals from mean NLL, positive in all 18 settings.Conditioning additionally on PIT-4B gold-versus-distractor margin and accuracy leaves a 1.4-point gap, positive in 22 of 36 settings; those covariates need not exhaust difficulty for every evaluated model.Source: §4.2 · p. 3, Table 3 · p. 4
  • Splitting by mean NLL or unnormalized NLL standard deviation reverses the final ordering, distinguishing those groupings from the TVS pattern.Unnormalized NLL standard deviation increases with mean NLL and therefore selects harder items.Source: §4.2 · p. 3

4.3 Answer-Specific TVS Predicts Learning in Later Training Stages

  • On ARC-Challenge, ARC-Easy, and MMLU, answer-specific TVS predicts greater late gold-versus-best-distractor margin growth after controls (p < 10^-6 each).The effect is weaker on PIQA and lacks robust late-margin growth on HellaSwag and WinoGrande; group comparisons show a 12.3-point average gap for answer-margin TVS versus 2.1 points for question-only TVS.Source: §4.3 · pp. 4–5, Table 4 · p. 5

4.4 Assessing TVS Robustness to Benchmark Contamination

  • PIT-4B contamination checks find no detectable release-specific gains, and high-TVS accuracy falls slightly after benchmark release (−0.011, p = .004).ARC, PIQA, and MMLU share similar year profiles despite different release dates.Source: §4.4 · p. 5
  • The contamination checks provide evidence against benchmark leakage as the dominant explanation, but sparse yearly sampling cannot establish its absence or exclude contamination in evaluated-model corpora.The FineWeb scan sampled ten files per crawl year, and the checks address PIT-4B rather than the evaluated models’ full pretraining corpora.Source: §4.4 · p. 5

5 Related Work

The related work positions TVS alongside item-analysis and corpus-attribution methods, and distinguishes its use of yearly likelihood variation from approaches tied to evaluated-model behavior or exact textual overlap.

Benchmark analysis

  • Dataset cartography and item response theory characterize benchmark items through training dynamics or difficulty and discrimination, while such behavior-based measures can vary with the evaluated model.TVS instead uses an external suite sharing architecture and training setup but differing in temporal data slice, enabling comparisons across model families.Source: §5 · p. 5

Attributing learning to pretraining data

  • TVS measures sensitivity beyond exact overlap without requiring corpus interventions, complementing exact-match corpus counts and controlled corpus ablations.Exact-match counts make exposure estimates wording-dependent.Source: §5 · p. 5

Point-in-time language models

  • Unlike time-stratified work on temporal generalization, TVS compares fixed benchmark items using yearly likelihood variation, with groups defined independently of evaluated checkpoints.Time indexes the corpus, not an intervention or changing targets.Source: §5 · p. 5

6 Conclusion

TVS tracks item-level learning using point-in-time models. Across six benchmarks and six models, high-TVS items generally improve more later, with the strongest evidence on knowledge-intensive tasks.

6 Conclusion

  • High-TVS items generally improve more later across six benchmarks and six models, with the most robust effects on ARC and MMLU.Difficulty-controlled, answer-specific TVS predicts late margin growth; the diagnostic does not require training-corpus access or semantic item categories.Source: §6 · p. 5

Limitations

The study’s scope is limited by the model-based nature of TVS and its benchmark and model coverage.

Limitations

  • TVS effectiveness depends on the quality of its underlying models; PIT-4B covers crawl years 2017–2024 at one model size, and evaluation uses likelihood-ranked multiple-choice benchmarks.Contamination checks cover PIT-4B training data, not the evaluated models’ full pretraining corpora.Source: Limitations · p. 6

A Additional Results

The final high-minus-low TVS gap is positive across model and benchmark aggregations, while high- and low-TVS items perform similarly at the earliest checkpoints. High-TVS items also gain faster during later training.

A.1 Effect Sizes and Significance

  • +0.073 chance-normalized accuracy is the mean final high-minus-low gap across 36 settings (95% t-CI [0.055, 0.092], p = 1.9×10−9).The gap remains positive when averaged within benchmarks, models, or model families.Source: §A.1 · p. 8
  • 0.020 per tenfold increase in training tokens is the high-group slope advantage over the low group during the second half of checkpoints.The difference is positive in 26 of 36 settings, with a 95% CI of [0.011, 0.029].Source: §A.1 · p. 8

A Additional Results

  • At the earliest Pythia and OLMo checkpoints, the mean high-minus-low gap is −0.008, with the low group ahead in 13 of 24 settings.The appendix compares earliest and final checkpoints; Figure 2 shows five TVS quantiles rather than three.Source: §A.1 · p. 8, A Additional Results · p. 8

B.1 Mean Difficulty

Difficulty conditioning reduces the TVS advantage, though it remains on ARC and MMLU. Results also vary with the TVS target string and dispersion measure, while length stratification controls for string length.

B.2 Length

  • Within length bins, high-TVS items have lower mean NLL in every benchmark.Length-stratified grouping leaves TVS essentially uncorrelated with length (Pearson −0.016; Spearman 0.006).Source: §B.1 · p. 8, §B.2 · p. 8

B.1 Mean Difficulty

  • Conditioning TVS on mean NLL roughly halves the final gap, from +0.073 (34/36 settings) to +0.037 (26/36).The remaining advantage is concentrated in ARC-C, ARC-E, MMLU, and PIQA; additionally conditioning on PIT-4B margin and accuracy leaves +0.014 (22/36).Source: §B.1 · p. 8

B.3 Perplexity-Based CV

  • The unnormalized standard deviation of NLL reverses the pattern (−0.026, 8/36), while rank-based disagreement gives a weak positive gap (+0.011, 22/36).The unnormalized standard deviation rises with mean NLL and selects harder items; perplexity-based CV is also entangled with mean difficulty and length, motivating TVS.Source: §B.1 · p. 8, §B.3 · p. 8, Table 6 · p. 9

C.1 Question-Only TVS

  • Question-only TVS produces a weaker final advantage than TVS computed on the question and gold answer: 0.021 in 26 of 36 settings versus 0.073.The question alone can underspecify the relevant fact or procedure.Source: C Alternative Target Strings · pp. 8–9, §C.1 · p. 9

C.2 Gold Answer versus Best Wrong Answer

  • The gold-answer-versus-best-wrong-answer TVS margin isolates variation specific to the correct answer.The margin split has the largest final-checkpoint separation of the stratifications reported here.Source: §C.2 · p. 9
  • The gold-versus-best-wrong TVS margin yields the largest final separation: its high-margin group leads in all 36 settings, with a mean gap of 0.123.It is the best final-accuracy group in 35 settings; at the earliest Pythia and OLMo checkpoints, the low-margin group leads in 18 of 24.Source: §C.2 · p. 9

D Item-Level Analysis by Benchmark

TVS-related results vary by benchmark and by whether variability is shared with distractors or specific to the gold answer. Synthetic and frequency analyses further examine what the score captures.

D Item-Level Analysis by Benchmark

  • The most variable TVS quintile ends ahead of the least variable one in 32 of 36 settings; all four exceptions are on WinoGrande.Figure 2 presents the main result with five TVS quantiles.Source: Figure 2 · p. 10
  • The gold-specific TVS component strongly predicts late margin growth on ARC-C, ARC-E, and MMLU, weakly on PIQA, and not at all on HellaSwag or WinoGrande.On the latter two benchmarks, it instead predicts a small difference in final correctness; 90–92% of gold TVS variance is shared with distractors on ARC-C, ARC-E, and MMLU, versus 53–59% on HellaSwag and WinoGrande.Source: D Item-Level Analysis by Benchmark · p. 10

E What Does TVS Capture?

  • Making answer support era-dependent raises synthetic-model TVS from 0.229 to 0.351 across five seeds, while stable conflicting evidence does not raise it.Token, entity, and answer counts and total support were held fixed across eras.Source: E What Does TVS Capture? · p. 10
  • Higher mean NLL is associated with lower accuracy at all checkpoints.Difficulty-conditioned TVS results are reported separately in Appendix B.1.Source: Figure 3 · p. 11
  • After controlling for n-gram counts, TVS retains an independent association with late gold-margin growth (β = 0.031 per SD, 95% CI [0.017, 0.045]).The association is concentrated in ARC-C, ARC-E, and MMLU; the analysis uses 25.7B sampled tokens and counts 2- and 3-grams ending in answers.Source: E What Does TVS Capture? · p. 11

F Contamination Checks

The release-date test finds no release-specific improvement in PIT-4B snapshots, and verbatim-overlap checks provide evidence against overlap as the dominant explanation of the TVS pattern.

F Contamination Checks

  • Post-release gold NLL is slightly higher (+0.016), with no detectable gold-versus-distractor margin change (p = .34).PIT-4B accuracy on high-TVS items falls after release (−0.011, p = .004), and the test finds no release-specific improvement predicted by benchmark leakage.Source: F Contamination Checks · p. 11
  • Verbatim overlap occurs for 3.1% of high-TVS versus 2.0% of low-TVS items, but adding overlap indicators leaves the TVS coefficient essentially unchanged (0.0267 versus 0.0268).The scans cover 55.4B FineWeb tokens; the authors report evidence against verbatim overlap as the dominant explanation.Source: F Contamination Checks · pp. 11–12

Related papers

Loading 2610.37631v1…