Source-linked AI summary
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
Jeffrey Li, Mohammadreza Armandpour, Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Samy Bengio, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, Fartash Faghri
TL;DR
Historical web-trained LLMs become outdated, creating a need to update them over time without costly retraining. TiC-LM introduces a 114-dump Common Crawl benchmark with domain-specific temporal evaluations and finds that replay with autoregressive schedules can approach scratch retraining at 2.6× lower compute, although replay’s value varies by domain.
Problem
LLMs trained on historical web data become outdated, while retraining them from scratch as new data arrives is costly.
Method
The paper introduces TiC-LM, a benchmark built from 114 time-stratified Common Crawl dumps with general-web and domain-specific evaluations.
Results
On general web data, autoregressive schedules with replay approach periodic scratch retraining using 2.6× less compute, while replay benefits differ across domains.
Takeaways & Limitations
Continual pretraining can efficiently match periodic retraining on general web data, but replay ratios should reflect domain-specific forgetting and change.
Takeaways & Limitations
No method outperforms Oracle retraining on all evaluations.
Abstract
from arXiv · showhide
Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) - orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains.
1 Introduction
TiC-LM addresses the challenge of updating web-trained LLMs without repeatedly retraining from scratch by introducing a long-horizon, web-scale continual-learning benchmark. Its experiments show that replay and autoregressive schedules can approach periodic retraining at lower compute, while replay’s value depends on the evaluation domain.
- Motivation: The benchmark fills a gap left by prior continual-language studies that usually used single domains or fewer than three training rounds.The authors motivate a longer-horizon benchmark matching general web-scale training practice.
- Benchmark: TiC-LM introduces 114 monthly Common Crawl timesteps and time-stratified evaluations spanning general web data, Wikipedia, StackExchange, and code documentation.TIC-CC contains 2.9T possible training tokens, with training revealing one month at a time.
- Experiments: The study evaluates more than 150 optimization, replay, and regularization strategies for continual pretraining.The released code and evaluation assets are intended to support further research.
- Key findings: 2.6× less total compute makes continual pretraining competitive with models retrained from scratch every two years.A mix of learning-rate and data-replay strategies approaches the Oracle series, though trade-offs remain across domains and evaluations.
- Key findings: Replay is essential for retaining performance on earlier general-web Common Crawl dumps when training proceeds only on new data.The benchmark reports significant forgetting on general web data without replay.
- Key findings: Replay can hurt rapidly evolving StackExchange and PyTorch evaluations while benefiting more stable Math and NumPy domains.The optimal balance between learning new data and replaying older data is domain-dependent.
2 Related Work
Prior continual-language benchmarks generally focus on individual sources and short horizons, whereas TiC-LM studies temporal shifts in generic web data across 114 months. This scale supports evaluations beyond a single domain.
- Prior work: Earlier web-scale continual-training studies did not focus on temporal distribution shifts and generally used fewer than three training rounds.This limits their use for studying lifelong time-continual learning over longer horizons.
- Comparison: Table 1 compares TiC-LM with earlier LLM continual-learning studies by temporal setup and training-set size.Its Temporal column indicates whether rounds are defined across temporal distribution shifts.
- Scope: TiC-LM spans 114 Common Crawl months and 2.9T tokens, making it larger and more diverse than prior continual language-modeling benchmarks.Earlier work typically used individual sources such as news, Wikipedia, or social media and no more than 10 timesteps.
3 TIC-CC: over 10 years of web data
TIC-CC is a causally constructed, monthly time-stratified Common Crawl corpus covering more than a decade of web data. Its experiments use a smaller shard, while the full corpus leaves room for scaling.
- Dataset construction: TIC-CC contains 2.9T tokens across 114 Common Crawl splits from May 2013 through July 2024.Each split corresponds to a month, and older-month processing avoids operations dependent on future months.
- Data processing: The corpus extends DCLM-Pool with monthly parsing, RefinedWeb heuristic filtering, and within-month fuzzy deduplication.The pipeline omits a classifier trained on all months to preserve temporal causality.
- Experimental scale: Experiments train on 220–440B tokens from one global shard, while the full dataset contains 29T tokens across all monthly material.Individual months range from 100B to 500B tokens, and future work can expand beyond the experimental shard.
4 Evaluations
TiC-LM evaluates continual models on held-out Common Crawl data, evolving factual knowledge, historical question answering, code documentation, and static downstream capabilities. These evaluations separate temporal adaptation from retention and general capability preservation.
- 4 Evaluations: The evaluation section defines token, answer, and noun perplexity for measuring different kinds of temporal performance.The metrics use page tokens, question-answer pairs, or proper-noun tokens depending on the evaluation.
- 4.1 TiC-CommonCrawl (TIC-CC): TIC-CC evaluates token perplexity on held-out monthly Common Crawl pages, including full-distribution, Wikipedia, and news subsets.These sets align with the training objective and directly measure learning across months and subsets.
- 4.2 TiC-Wikipedia (TIC-WIKI): TIC-WIKI spans 2014–2024 and separates changed from unchanged factual knowledge using proper-noun perplexity.It expands TemporalWiki’s four-month coverage and produces TIC-WIKI-Diff and TIC-WIKI-Unchanged.
- 4.3 TiC-StackExchange (TIC-STACKE): TIC-STACKE measures answer perplexity on accepted StackExchange answers dated from 2008 through 2024.Examples come from selected communities and are binned by accepted-answer timestamps.
- 4.4 TIC-CODEDOCS: TIC-CODEDOCS measures token perplexity across release snapshots of NumPy and PyTorch documentation.The evaluation covers 16 NumPy releases from 2017–2024 and 11 PyTorch releases from 2021–2024.
- 4.5 Static downstream evaluations.: Static downstream evaluation uses the CORE average over 22 zero-shot and few-shot tasks to test general capabilities of base models.These tasks assess whether continual models match capabilities of models trained on all dumps.
5 Continual Learning Baseline Methods
The benchmark compares optimization, replay, regularization, and non-continual retraining strategies for time-continual language-model pretraining. Replay controls the mixture of current and prior months, while learning-rate schedules and regularization constrain continual updates.
- 5 Continual Learning Baseline Methods: The study compares optimization-based, data-replay, regularization, and non-continual oracle methods for continual pretraining.The oracle baseline retrains from scratch on equal tokens from all available months, approximately every two years.
- Optimization-based methods: Cyclic Cosine decays the learning rate within each month, whereas Cyclic Cosine + AR also decays each round’s maximum learning rate using a global cosine trajectory.Rsqrt uses a slowly decaying global schedule with linear cooldowns, and Schedule-Free uses iterate averaging instead of an explicit learning-rate schedule.
- Data replay methods: Replay mixes current-month tokens with earlier months, with α_t controlling the current-month share and the remaining budget distributed equally across prior months.The benchmark tests α_t = 1/t, which equalizes tokens across observed months, and α_t = 1/2, which reserves half the budget for the current month.
- Data replay methods: Replay-based methods may require retaining and removing old data, but the study assumes these costs are dominated by training costs and does not include them.Larger α_t values reduce the amount of replay and therefore lessen exposure to this storage limitation.
- Regularization-based methods: Regularization methods encourage new updates to remain close to the previous checkpoint through LwF’s output KL penalty or EWC’s Fisher-weighted parameter protection.These methods require extra loss terms and model copies, increasing GPU memory footprints and runtimes relative to optimizer- and replay-based methods.
6 Experiments
The experiments evaluate continual pretraining across general web data and downstream domains, revealing compute-efficient performance alongside strong trade-offs between learning recent data and retaining older knowledge. Replay and autoregressive schedules help on general web data, but their value depends on domain evolution and evaluation setting.
- 6.1 Held-out performance on TIC-CC: Continual pretraining outperforms the Oracle series on TIC-CC with 62% less compute, while replay (αt = 1/2) plus AR at 440B tokens comes within 0.0001 on backward transfer.Reusing models also enables more frequent checkpoints at lower total cost.
- 6.1 Held-out performance on TIC-CC: Cyclic Cosine achieves the best ID performance on TIC-CC but also the most forgetting, whereas AR provides the best backward transfer among fixed maximum learning-rate schedules.The best continual learning rate per cycle is 1e-4, 30× smaller than for initial pretraining.
- 6.1 Held-out performance on TIC-CC: Replay is essential for retaining older TIC-CC knowledge: it improves backward transfer by 60% for 220B runs, with αt = 1/t minimizing forgetting but αt = 1/2 offering a better ID–backward trade-off.The αt = 1/t strategy becomes problematic as the number of timesteps increases because it decreases the ratio of new data.
- 6.1 Held-out performance on TIC-CC: On specific web subsets, AR schedules and replay can improve ID over Cyclic Cosine on TIC-WIKI and TIC-CC-NEWS, unlike their behavior on full TIC-CC.These subsets evolve more slowly than general CC, although domain prevalence can still vary sharply over time.
- 6.1 Held-out performance on TIC-CC: Smaller 1B models have worse backward retention, especially without replay, but method rankings remain similar and model scaling helps less than switching from non-replay to replay.ID relative to the corresponding Oracle is close between 1B and 3B models.
- 6.2 Downstream evaluations: Continual methods outperform Oracle retraining on many downstream evaluations, but the comparison varies: Oracles retain advantages on TIC-CODEDOCS-NUMPY and TIC-STACKE-English, while some continual methods win elsewhere.Cyclic Cosine beats Oracles on TIC-STACKOVERFLOW and TIC-CODEDOCS-PYTORCH, whereas only Replay (α = 1/t) does so on TIC-STACKE-MATH.
- 6.2 Downstream evaluations: Replay helps slower-evolving domains such as TIC-STACKE-MATH and NumPy but hurts faster-evolving domains such as TIC-STACKOVERFLOW and PyTorch.Older dumps contain more relevant data for slower-evolving domains, whereas replay can overemphasize pre-release data for PyTorch.
- 6.2 Downstream evaluations: On TIC-WIKI, performance can peak years after the corresponding CC dump even without replay, while continual methods remain competitive on backward transfer and static CORE evaluations still show a gap to Oracle-2024-07.The paper attributes the delayed peak to persistent factual knowledge and delayed alignment between CC crawls and comprehensive Wikipedia coverage; the May-2013 initialization scores 48.5 versus 49.9 when trained jointly with later months.
7 Conclusion
The paper introduces a web-scale benchmark for continual LLM pretraining, combining 114 Common Crawl timesteps with time-stratified evaluations across several domains. It uses these assets to study forgetting, plasticity, and efficiency trade-offs in continual updating.
- The benchmark contains 114 Common Crawl timesteps plus time-stratified TIC-WIKI, TIC-STACKE, and TIC-CODEDOCS evaluations.
- The study highlights trade-offs between retaining older knowledge and adapting to new web data across evaluation domains.
Limitations
The authors identify unresolved limitations in both continual-learning performance and benchmark scope. No method outperforms Oracle retraining on every evaluation, while the benchmark omits several realistic pretraining and evaluation dimensions.
- No evaluated method outperforms Oracle retraining on all evaluations.
- The benchmark focuses on general web data rather than mixtures with separately curated sources such as Wikipedia or ArXiv.
- The tokenizer is fixed across timesteps, leaving model-tokenizer co-adaptation under language change unexplored.
- Dynamic evaluations are limited to perplexity-based metrics because accuracy-based variants produced little meaningful signal at the training scale.
Ethical Considerations
The work may improve access to efficient LLM training, but the underlying web data and broader accessibility also carry social and safety risks. Common Crawl-derived data may contain offensive or personally identifying content.
- More accessible and efficient LM training could reduce resource and energy barriers while increasing risks such as disinformation.
- Common Crawl and general-web training data may contain offensive and personally identifying content.
A Dataset Construction
The dataset construction builds monthly Common Crawl splits and held-out evaluations, then extends the benchmark with decade-scale Wikipedia evaluations. Preprocessing emphasizes month-local deduplication, temporal separation, and changed-versus-unchanged Wikipedia content.
- A Dataset Construction: TIC-CC is constructed from monthly Common Crawl data, extending through July 2024 with month-specific preprocessing.
- A Dataset Construction: Filtering is applied independently by month, while near-duplicate removal is restricted within each month rather than globally.
- A Dataset Construction: Held-out evaluations use a different global shard from training, although overlap across shards or months remains possible.
- B.1 TIC-WIKI: TIC-WIKI extends TemporalWiki from four months to 2014–2024 and provides perplexity- and QA-based evaluations.
- B.1 TIC-WIKI: The main TIC-WIKI evaluation uses 61 months with 10,000 changed and 10,000 unchanged sentences or paragraphs per month.
- B.1 TIC-WIKI: QA evaluations were noisy because models achieved fairly low performance, so the study focuses on perplexity evaluations while releasing QA-generation code.
- B.1.1 Data Download: Wikipedia data are obtained from historical dumps, with archived monthly snapshots used to avoid information leakage and align nearby Wikidata dumps.
- B.1.2 Data preprocessing for perplexity evaluations: Perplexity evaluations compare normalized consecutive snapshots hierarchically, separating changed from unchanged content and retaining meaningful edits.
B.1.3 Data preprocessing for QA evaluations
The paper constructs time-stratified evaluations from evolving Wikipedia, StackExchange, and code-documentation data, with metrics designed to separate in-distribution, forward, and backward transfer.
- B.1.3 Data preprocessing for QA evaluations: Natural-language Wikipedia fill-in-the-blank questions are built from changed and unchanged Wikidata statements linked to sentences mentioning each subject and object.The pipeline extracts temporal differences from consecutive dumps, filters and samples statements, and replaces the object with a blank.
- B.2.1 Data preprocessing: TIC-STACKE reconstructs monthly question-and-answer snapshots from Post.xml and PostHistory.xml, retaining high-quality answers selected by acceptance and vote comparisons.The benchmark uses StackOverflow, Mathematics, and English Language & Usage, combining adjacent months when a period has fewer than 500 questions.
- B.2.2 Analysis of StackExchange Data: StackExchange analysis characterizes category prevalence, monthly question counts, answer counts, and question-length distributions for the three selected sites.These analyses are summarized through Figures 11–14.
- B.3 TIC-CODEDOCS: TIC-CODEDOCS contains official NumPy and PyTorch documentation generated from 16 and 11 major releases, respectively, then converted into structured plaintext for perplexity evaluation.Each release is reconstructed from its corresponding repository commit and rendered with Sphinx before text extraction.
- B.4 Evaluation metrics: The evaluation matrix records performance after training through timestep i on evaluation timestep j, while ID, forward transfer, and backward transfer aggregate different matrix regions.The procedure assigns each evaluation period exclusively to its nearest preceding checkpoint and uses Oracle-2024-07-relative regret to control for month difficulty.
- C.1 General details: Experiments follow DataComp-LM’s 3B configuration, training on 64 H100 GPUs with 2048-token sequences and evaluating 12 roughly annual checkpoints from the 114 Common Crawl timesteps.The continual phases require 1.5K H100 hours at 220B scale and 4.5K at 440B scale.
C.2 Hyperparameters for continual methods
The continual-training experiments tune learning-rate schedules, replay, and regularization under limited early-round validation, revealing distinct trade-offs between retaining old data and fitting new data.
- C.2 Hyperparameters for continual methods: A 1e-4 maximum learning rate gives Cyclic Cosine the strongest ID performance on TIC-CC, whereas lower rates improve backward transfer at an ID cost.The best continual rate is at least 30× below the 3e-3 May-2013 initialization rate.
- C.2 Hyperparameters for continual methods: LwF either has little effect at small regularization weights or decreases all metrics at larger weights, while EWC requires large λ values and trades ID against backward transfer.EWC’s best tuned setting is λ = 10^7, whereas λ = 10^6 slightly improves ID but substantially worsens backward transfer.
- D.1 TiC-CommonCrawl (TIC-CC) evaluations: On TIC-CC, Cyclic Cosine achieves the best ID performance but also the most forgetting, while autoregressive schedules and replay improve retention with different domain-specific costs.Replay and autoregressive schedules can outperform Cyclic Cosine on slower-evolving Wiki and News subsets even when they lose on full TIC-CC ID.
- D.6 Extended replay exploration: Among fixed replay ratios, α = 0.5 or slightly higher is a practical compromise because more replay improves backward transfer but can reduce ID on TIC-CC.Replay (Exp) generally falls between α = 0.5 and α = 0.9 and is often dominated by α = 0.7.
D.7 Deduplicating newer against older data
The appendix examines cross-month deduplication, model-size effects, and the benchmark’s relationship to prior temporal and continual-pretraining research.
- D.7 Deduplicating newer against older data: Windowed deduplication across ten-month periods can improve some downstream evaluations when combined with replay, but it can hurt rare domains such as code.More aggressive deduplication may reduce relevant samples in rare domains while increasing data diversity for evaluations benefiting from general data, such as TIC-WIKI.
- D.8 Effects of model size on downstream evaluations.: Changing model size makes continual methods more suboptimal relative to Oracle-2024-07 on most evaluations except TIC-WIKI, while preserving the overall replay-versus-no-replay conclusions.The comparison uses 1B and 3B models trained with the same 220B-token budget.
- E Extended Related Work: The benchmark extends prior temporal evaluations by measuring general language and knowledge understanding across realistic, timestamped Wikipedia data rather than only narrowly targeted temporal facts or synthetic questions.Its TIC-WIKI construction expands on TemporalWiki and avoids LLM-generated evaluations to reduce transferred generation biases.
- E Extended Related Work: Prior temporal-generalization studies report outdated LLM knowledge across domains, while TiC-LM evaluates deterioration and continual updating across Common Crawl, Wikipedia, news, StackExchange, and code documentation.The paper emphasizes that deterioration rates differ by domain and that continual pretraining and retrieval address freshness through different cost trade-offs.
- E Extended Related Work: Earlier continual-pretraining benchmarks often use one-step updates, few rounds, or narrower domains, whereas TiC-LM studies more than 100 sequential large-scale training steps.This setup is intended to examine forgetting and scalability in multi-year web-scale pretraining.