Source-linked AI summary
Understanding Data Temporality Impact on Large Language Models Pre-training
Hippolyte Pilchen, Romain Fabre, Franck Signe Talla, Patrick Perez, Edouard Grave
TL;DR
The temporal grounding of LLM knowledge and the role of pre-training data order remain poorly understood. This paper compares sequential and shuffled Common Crawl pre-training with a temporal-factuality benchmark, finding that sequential training preserves general understanding while improving factual freshness and temporal precision.
Problem
The role of pre-training data ordering in shaping LLMs’ temporally distributed factual knowledge remains poorly understood, despite models’ frozen and uneven knowledge horizons.
Method
The authors compare 6B-parameter models trained on sequential or shuffled Common Crawl snapshots and evaluate general abilities and year-specific facts with OLMES and KairosQA.
Results
Sequential pre-training matches shuffled baselines on general language understanding while producing more up-to-date and temporally precise knowledge, including a recency peak on contemporary facts.
Takeaways & Limitations
Temporal data ordering is a consequential, under-explored factor for developing more up-to-date base language models and supporting continual-learning research.
Takeaways & Limitations
Sequential pre-training relatively forgets older knowledge as models adapt to newer temporal distributions.
Abstract
from arXiv · showhide
Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.
1. Introduction
This section motivates studying how pre-training data ordering shapes the temporal distribution of knowledge in LLMs. It introduces a controlled comparison of sequential and shuffled pre-training, alongside a time-sensitive benchmark and intermediate checkpoints for evaluating temporal knowledge and sequential-training effects.
- Motivation: Pre-training data ordering remains under-explored, despite evidence that models may know older facts more accurately than facts near the training cutoff.This exposes a gap between the dataset horizon and the model’s effective knowledge horizon.
- Experimental setup: The study compares 6B-parameter models trained at a fixed token budget on filtered sequential Common Crawl snapshots or shuffled versions of the same data.The resulting models are evaluated across language modeling, common knowledge, and temporal knowledge tasks.
- Contributions: The work establishes a controlled yet realistic sequential pre-training setup and plans to release intermediate yearly checkpoints for research on factuality and forgetting.The setup supports analysis of how knowledge changes throughout training.
- Contributions: The authors introduce a time-sensitive question–answering benchmark for evaluating temporal knowledge across diverse tasks.The benchmark is designed to assess whether models associate knowledge with the appropriate time periods.
- Contributions: Intermediate training checkpoints are compared with carefully chosen shuffled-data baselines to study the effects of sequential pre-training.This analysis is part of the paper’s contribution on understanding pre-training dynamics.
2. Pre-training Sequential Models
The study isolates temporal ordering by comparing a globally shuffled 6B-parameter baseline with a chronologically trained model using the same architecture and hyperparameters. The sequential curriculum spans 2018–2025 and produces yearly checkpoints for evaluating converged models at distinct temporal stages.
- Baseline setup: The shuffled baseline isolates data-ordering effects by using a globally shuffled corpus while holding model capacity constant.This design attributes performance differences strictly to the data curriculum rather than model capacity.
- Baseline setup: The baseline uses Common Crawl snapshots from 2020 to 2024, treating temporally mixed content as a timeless information pool.Global shuffling exposes the model to all temporal contexts simultaneously, despite persistent historical content within snapshots.
- Sequential curriculum: The sequential experiment preserves the baseline architecture and hyperparameters while processing Common Crawl snapshots in strict chronological order from early 2018 through 2025.The 2018–2019 period establishes initial linguistic capabilities and world knowledge in their original temporal context.
- Experimental limitation: The experimental comparison has a temporal asymmetry because the pre-existing shuffled baseline covers 2020–2024, whereas the sequential pipeline uses a wider multi-year timeline.This difference is acknowledged as a limitation of the experimental design.
- Sequential curriculum: The sequential design uses five corpora per year, approximately 315B tokens per yearly segment, and 2.5T tokens overall.Checkpoints are generated after each yearly segment, yielding eight models with data cutoffs from 2018 to 2025.
3. Evaluating Temporal Alignment
This section introduces KairosQA, a temporally sensitive question-answering dataset built from time-annotated Wikidata facts whose answers change across years. It evaluates temporal alignment through cloze and generative protocols, using constrained choices and normalized F1 to assess time-grounded knowledge.
- Dataset Construction: KairosQA samples Wikidata subject–relation–object triplets tied to specific years to evaluate facts that evolve over time.Wikidata is selected for its scale, open availability, and explicit temporal annotations.
- Dataset Construction: The dataset filters for relations whose answers change at least twice between 2018 and 2025, covering people, organizations, sports, and events.Popularity based on Wikipedia page views is used to reduce noise from rare entities.
- Dataset Construction: GPT-4o mini generates varied multiple-choice questions, while answer choices draw first from neighboring years and then from added distractors.Questions begin with reference templates and are modified to incorporate the target year.
- Evaluation Protocol: The evaluation adopts cloze formulation because models may not yet master multiple-choice structural constraints.On MMLU, label-based evaluation shows discontinuities around 200B tokens for the shuffled baseline and 400B tokens for the sequential model, reflecting formatting ability rather than only knowledge.
- Evaluation Protocol: A generative evaluation complements cloze scoring with normalized F1, while constrained answer choices help disambiguate ambiguous questions and sharpen temporal assessment.The generative score follows standard question-answering evaluation protocols.
4. Experimental settings
The experiments evaluate general language and knowledge performance alongside temporally grounded factual knowledge, using matched shuffled and sequential checkpoints across yearly training snapshots. Additional open-source models provide validation across different training periods and cutoff dates.
- Evaluation datasets: Evaluation uses OLMES for downstream language modeling and general knowledge, and KairosQA for temporal factual knowledge.OLMES provides a broad, precise, reproducible evaluation setup, while KairosQA is the paper’s temporal factual knowledge benchmark.
- Our checkpoints: Eight shuffled and sequential checkpoints are evaluated with pairwise-matched token counts, spanning 315B to 2.5T tokens.Sequential checkpoints correspond to yearly crawls from 2018 to 2025, with primary comparisons covering 2020–2024.
- Other open-source base models: A range of open-source LLMs validates the temporal evaluation dataset across comparable recent models and earlier training periods.Training cutoffs are reported when available; otherwise, public release dates serve as upper bounds, including Llama 3.1-8B with a December 2023 cutoff.
5. Results
Sequential pre-training preserves final general language understanding while producing more temporally precise and recent factual knowledge than shuffled training. Its benefits come with older-knowledge forgetting, uneven long-tail performance, and limited discrimination from TAQA under the standard protocol.
- General language understanding: Sequential and shuffled models achieve comparable final OLMES performance, although shuffling learns faster during mid-training while sequential learning steadily closes the gap.The sequential model’s slower, near-linear trajectory is attributed to ordering constraints and non-stationary data quality or density.
- Temporal knowledge: Shuffled checkpoints show nearly identical dynamics across training lengths and consistently favor 2024 accuracy, suggesting repeated historical information outweighs data quantity for temporal grounding.The reported interpretation is that shuffled training fails to effectively internalize contemporary knowledge despite access to 2020–2024 data.
- Temporal knowledge: Sequential checkpoints peak in the year immediately preceding their training cutoff, while performance declines after the cutoff, demonstrating a strong recency bias.Sequential models excel on recent years where shuffled baselines experience their largest failures.
- Model comparisons and limitations: Sequential training improves recent factual performance, surpassing larger models from 2023 onward, but trades off some older knowledge through relative forgetting.Later checkpoints’ larger token budgets partially mitigate forgetting in the generative task.
- Evaluation robustness: Cloze accuracy remains about 15 percentage points above random with 12 choices, reaching 22% versus a 7.7% baseline, indicating performance is not merely elimination-based.Accuracy decreases as the number of choices grows but stabilizes above eight choices.
- Evaluation limitations: Generative performance is strongest for popular subjects, with approximately 0.4 F1 for the top 10% but 0.17 for the 30–40% bin and 0.10–0.15 for the bottom 50%.This reveals a substantial long-tail weakness despite capturing up-to-date information.
6. Related Work
Prior research shows that pre-training data composition and scheduling affect downstream performance, while other studies improve temporal awareness through fine-tuning, prompting, architectures, or continual-learning strategies. The authors distinguish their benchmark from existing temporal benchmarks that are too challenging or incompatible with base-model temporal-alignment evaluation.
- Pre-training data impact: Prior work finds that pre-training data composition and scheduling can significantly influence downstream performance.Blakeney et al. (2024) report gains from upsampling domain-specific and code datasets toward the end of pre-training.
- Temporality in LLMs: Zhao et al. (2024) enhance temporal awareness through fine-tuning, whereas Park et al. (2025) target knowledge from specific years through prompting.Park et al. apply their prompting pipeline across multiple domains.
- Temporality in LLMs: The authors find Zhao et al.’s and Park et al.’s temporal benchmarks too challenging or incompatible for measuring base-model temporal alignment.This assessment motivates using a benchmark suited to the paper’s setting.
- Temporality in LLMs: Related approaches include Faro et al.’s time-aware mixture-of-experts architecture and Li et al.’s continual-learning study using month-specific corpora, learning-rate schedules, and replay mechanisms.Li et al. (2025) are identified as the work most closely related to this study.
7. Perspectives & Conclusions
The paper identifies shuffled pre-training as a major source of temporal knowledge gaps and shows that sequential pre-training improves recency on contemporary facts. Its main trade-off is forgetting older knowledge, motivating time-aware architectures that preserve historical information while tracking evolving facts.
- Shuffled pre-training is identified as a primary driver of the knowledge-horizon gap in modern LLMs.
- KairosQA isolates temporal awareness through rigorous Wikidata filtering, enabling evaluation of time-sensitive factual knowledge.
- Sequential pre-training produces a recency peak that surpasses larger models on contemporary facts.
- Sequential pre-training can cause relative forgetting of older knowledge as models adapt to newer temporal distributions.Increasing total token volume partially mitigates this trade-off.
- Future time-aware architectures should model data origins explicitly to track evolving facts without erasing history.The paper notes preliminary experiments highlighting the inherent difficulty of this challenge.
A. Additional experiments … A.3.2. REPLAYING FACTS FROM PREVIOUS YEARS
Additional experiments show that sequential training eventually matches shuffled learning on MMLU while improving temporal knowledge, but mitigating historical forgetting remains difficult. Model merging and replay provide limited recovery of older knowledge, with replay helping only on the cloze task.
- A.1. More results on our checkpoints: The Sequential model steadily closes Shuffle’s mid-training MMLU efficiency advantage without compromising final model capacity.The comparison spans 2.5T tokens, and chronological ordering changes the learning trajectory.
- A.1. More results on our checkpoints: Sequential and shuffled checkpoints show distinct temporal-evaluation dynamics across pre-training lengths on KairosQA.The evaluation covers cloze accuracy and generative performance, with a matched-token shuffled baseline denoted Shuff eq 202*.
- A.2. Open-source base models comparison: KairosQA compares the sequential approach with open-source models across model sizes and cutoff dates using cloze accuracy and generative F1.The figure reports temporal performance separately for cloze accuracy and F1 across evaluation years.
- A.3. Mitigating the forgetting effect: a non-trivial challenge: Attempts to preserve historical knowledge during sequential training were inconclusive, underscoring the difficulty of integrating new information without forgetting past facts.The experiments targeted mitigation of forgetting while training on contemporary data.
- A.3.1. MODEL SOUP: Model merging combines cooled-down yearly checkpoints through linear weight combinations, beginning with 2020 because 2018 and 2019 were under-trained.The goal was to obtain stronger performance across the full timeline.
- A.3.1. MODEL SOUP: Merged models average their components rather than achieving best-of-all-worlds performance, degrading 2025 results while negligibly recovering older knowledge.Simple weight interpolation therefore fails to mitigate catastrophic forgetting in sequential learning.
- A.3.2. REPLAYING FACTS FROM PREVIOUS YEARS: A final 30k-step replay cool-down mixes 2025 crawl data with filtered 2020 data in equal proportions and slightly mitigates historical loss only on cloze.The 2020 data filtering emphasizes educational content, following a strategy inspired by FineWeb.
A.4. TAQA (Zhao et al., 2024) results
The TAQA evaluation uses standard normalized F1 and compares baseline prompting with time-aware prompting that re-aligns models to knowledge from target years. Experiments assess targeting 2018 and 2021 across open-source base models and two pre-trained models.
- Evaluation setup: TAQA evaluations use the standard normalized F1 score and include both baseline and time-aware prompting configurations.The time-aware strategy aims to temporally re-align the LLM to knowledge from a specific year.
- Target-year evaluations: Time-aware prompting evaluates models while targeting knowledge from 2018.The comparison includes several open-source base models and two pre-trained models.
- Target-year evaluations: Time-aware prompting also evaluates models while targeting knowledge from 2021.The comparison includes several open-source base models and two pre-trained models.
B. Pre-training setting
This section presents the pre-training hyperparameters used in the study.
- Table 4 summarizes the pre-training hyperparameters.
C. Evaluation dataset … C.4. KairosQA examples
The evaluation dataset is built from temporally evolving Wikidata facts, filtered for temporal validity, popularity, variation, and answer quality, then converted into relation-specific questions with year-aware distractors. KairosQA examples show both multiple-choice answers and temporally indexed candidate options.
- C.1. Facts filtering: Wikidata supplies temporally evolving facts that are converted into timestamped subject–relation–object triplets, each forming one data sample.Wikidata contains over 120 million data items and is open-source with relatively high data quality.
- C.1. Facts filtering: The filtering pipeline removes records with missing or unparseable dates, population-only subjects, and absent popularity metrics before enforcing temporal variation and popularity constraints.The dataset is restricted to subject–relation pairs whose answers change at least twice between 2018 and 2025, followed by selection of the top 20% most popular subjects.
- C.1. Facts filtering: 7,268 subject–relation pairs remain after final pruning, with only pairs having valid answers for each evaluation year retained.For example, 6,200 pairs are retained for 2018.
- C.2. Relation-Aware Filtering.: Relation-aware quality control uses Claude and manual checks to resolve answer incoherence, ambiguous award scope, achiever–work ambiguity, and unsuitable multi-winner or ill-defined awards.Affected award questions may be split into variants, while awards lacking disambiguation or unambiguous evaluation are removed.
- C.3. KairosQA generation templates: Manually predefined templates are used for each relation because ambiguous subject or relation formulations frequently produce imprecise generated questions.Templates include a {{year}} indicator so the same question structure can be used across multiple years.
- C.3. KairosQA generation templates: Distractors are generated from valid answers across recorded years and deduplicated against ground-truth answers using F1-score thresholds and fuzzy matching.The distractors are intended to be plausible, relevant, and distinct from existing answers.