PaperlayerView exact revision on arXiv

Source-linked AI summary

TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo

arXiv:2204.14211v3cs.CL

TL;DR

Changing facts can make language models temporally misaligned, and existing evaluation resources do not provide an ongoing benchmark for adaptation. TemporalWiki uses consecutive English Wikipedia and Wikidata snapshots for training and evaluation; updating on changed data improves new-knowledge acquisition at much lower computational cost, though the benchmark does not cover knowledge deletion.

  • Problem

    Static benchmarks and temporal-adaptation datasets do not provide ongoing evaluation as factual knowledge changes, limiting assessment of language models' ability to adapt over time.

    Source: §1 · p. 1

  • Method

    TemporalWiki builds training corpora from changed information between consecutive English Wikipedia snapshots and evaluation probes from corresponding Wikidata snapshots, enabling periodic automated model training and evaluation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Updating on TWIKI-DIFFSETS yields better acquisition of new and updated knowledge than whole-snapshot updates at 12 times less computational cost, while updated-data training remains challenging across multiple updates due to forgetting.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The benchmark supports automated periodic evaluation of models' retention of previous knowledge and acquisition of updated or new knowledge.

    Source: §1 · p. 2, §6 · p. 8

    Wikipedia and Wikidata updates do not necessarily reflect actual changes in real-world knowledge, and the work does not cover knowledge deletion.

    Source: §7 · p. 9

Abstract

from arXiv · show

Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning. The dataset and the code are available at https://github.com/joeljang/temporalwiki.

1 Introduction

Changing factual knowledge can leave language models temporally misaligned, while static or eventually outdated datasets limit ongoing evaluation. TemporalWiki uses changing Wikipedia and Wikidata snapshots to train and evaluate models periodically; experiments find updated-data training more efficient, though forgetting remains challenging across updates.

1 Introduction

  • Most language-model evaluation datasets are static, with training and test data drawn from similar points in time, despite frequent factual changes in the real world.The introduction frames this mismatch as a challenge for tasks that require recent factual information.Source: §1 · p. 1
  • Temporal misalignment arises when changing facts make training-time knowledge differ from test-time needs, such as the shift from the Delta to Omicron coronavirus variant in 2021.The paper notes that this matters for knowledge-intensive tasks and that stale model parameters can conflict with up-to-date retrieved information.Source: §1 · p. 1
  • Existing temporal-adaptation datasets are static and eventually become outdated, making automated periodic evaluation important for assessing ever-evolving models.The paper introduces TemporalWiki as a lifelong benchmark for periodic, automated training and evaluation.Source: §1 · p. 1
  • TemporalWiki compares consecutive English Wikipedia snapshots to build TWIKI-DIFFSETS and corresponding Wikidata snapshots to build TWIKI-PROBES, filtering evaluation facts to those grounded in Wikipedia articles.The evaluation instances are categorized as unchanged or changed, and benchmark creation requires no human annotation.Source: §1 · p. 2
  • TemporalWiki is designed to train and evaluate models automatically as English Wikipedia and Wikidata snapshots are updated.The benchmark is presented as responsive to changes in the world, unlike previous static benchmarks.Source: §1 · p. 2
  • Continual training on updated Wikipedia portions is more efficient than training on entire snapshots in computation and the stability-plasticity trade-off, but multiple updates remain challenging due to catastrophic forgetting.The authors also implement continual-learning baselines intended to mitigate forgetting while supporting new-knowledge learning.Source: §1 · p. 2

2 Background

Prior work studies temporal misalignment from language change and changing factual knowledge, while available temporal benchmarks are often domain-specific and static. Wikipedia and Wikidata offer dynamic sources of general world knowledge for a lifelong benchmark.

2 Background

  • Temporal misalignment can arise from either the dynamic nature of language or updates to factual information.The paper cites work on language change (Röttger and Pierrehumbert, 2021; Hombaiah et al., 2021; Rosin et al., 2021; Loureiro et al., 2022) and factual updates (Chen et al., 2021; Dhingra et al., 2022; Jang et al., 2022).Source: §2 · p. 3
  • For non-knowledge-intensive tasks, Luu et al. (2021) report severe degradation from temporal misalignment and mitigation through target-period finetuning, whereas Agarwal and Nenkova (2021) find self-labeling more effective than continued pretraining.These studies focus on temporal change in language rather than knowledge-intensive tasks.Source: §2 · p. 3
  • For factual knowledge updates, Lazaridou et al. (2021) argue that ever-evolving models are needed, Dhingra et al. (2022) propose including time information, and Jang et al. (2022) and Jin et al. (2022) use continual learning to mitigate forgetting.The cited works address factual temporal misalignment and updating model knowledge.Source: §2 · p. 3
  • Prior temporal-adaptation benchmarks use Twitter, news, or arXiv data, but these sources are domain-specific and inherently static.The paper identifies a lack of widely available resources for training and evaluating ever-evolving models.Source: §2 · p. 3
  • Wikipedia and Wikidata are presented as sources of general world knowledge whose ongoing updates support a lifelong benchmark, though updates need not represent real-world changes.The paper notes that volunteer editors make frequent English Wikipedia updates and that not every update corresponds to a real-world change.Source: §2 · p. 3

3 TemporalWiki

TEMPORALWIKI combines training corpora from consecutive English Wikipedia snapshots with evaluation probes from aligned English Wikidata snapshots. Its evaluation data distinguishes changed from unchanged facts and applies quality controls.

3.1 Generating Training Corpora from Wikipedia

  • New article IDs contribute whole articles, while existing IDs contribute only new or updated sentences found by comparing consecutive snapshots.The benchmark uses snapshot differences because most Wikipedia content is unchanged and retraining on the entire snapshot is computationally expensive.Source: §3.1 · pp. 3–4

3.2 Generating Evaluation Datasets from Wikidata

  • The probes measure stability and plasticity by classifying new relations or new objects for existing relations as CHANGED and retained facts as UNCHANGED.Each probe is constructed from consecutive Wikidata snapshots and is used to evaluate an LM updated with the corresponding TWIKI-DIFFSET.Source: §3.2 · p. 4

3.3 Quality Control for Evaluation Data

  • Quality control aligns changed facts to diffset articles and unchanged facts to Wikipedia, then filters substring overlaps, objects over five words, and overly frequent entities or relations.Mapped instances are retained only when the object appears in the article text.Source: §3.3 · pp. 4–5

4 Dataset Statistics

The benchmark is constructed from snapshots spanning August through December 2021, and the reported statistics describe monthly updates and probe composition.

Training Corpora Statistics

  • Updated and added information, measured by tokens in each TWIKI-DIFFSET, is similar in amount across months.The benchmark is constructed from August 2021 to December 2021.Source: §4 · p. 5

Evaluation Dataset Statistics

  • The reported TWIKI-PROBES statistics show similar proportions of each subject and object entity category across months.Source: §4 · p. 5
  • The authors also report the top 30 most frequent relations separately for UNCHANGED and CHANGED instances.Source: §4 · p. 5

5 Experiments with TEMPORALWIKI

The experiments compare monthly updates from full Wikipedia snapshots and changed-only data, using continual-learning baselines and evaluations of both training-corpus perplexity and factual probes. Changed-only training improves learning of updated facts but risks forgetting; continual-learning methods mitigate that trade-off, with K-ADAPTER showing robust overall probe performance at much lower training time than FULL.

5.1 Experimental Settings

  • The experiments continue pretraining GPT-2 Large (774M parameters), compare TWIKI-DIFFSETS updates with entire Wikipedia-snapshot updates using TWIKI-PROBES, and implement methods from literature known to mitigate catastrophic forgetting during TWIKI-DIFFSETS-only updates.Source: §5.1 · p. 5

5.2 Baseline Models

  • FULL trains sequentially on each month’s entire Wikipedia snapshot, whereas DIFF trains on TWIKI-DIFFSETS, averaging 4.6 billion versus 347 million token updates per month.Both methods make one pass through each month’s training data.Source: §5.2 · p. 6
  • The continual-learning baselines include rehearsal with initial-pretraining data, parameter expansion with LORA or K-ADAPTER, and regularization with RECADAM.K-ADAPTER adds 103M parameters; the study adds them once, with the optimal interval left for future work.Source: §5.2 · p. 6

5.3 Intrinsic Evaluation

  • On NON-TWIKI-DIFFSETS, DIFF’s relative perplexity rises over time while FULL’s remains constant; continual-learning methods mitigate forgetting, especially MIX-REVIEW.On TWIKI-DIFFSETS, DIFF has much lower relative perplexity than FULL, while MIX-REVIEW and RECADAM are similar to DIFF and K-ADAPTER and LORA are higher.Source: §5.3 · pp. 6–7, Figure 2 · p. 6

5.4 Extrinsic Evaluation on TWIKI-PROBES

  • K-ADAPTER shows the most robust overall TWIKI-PROBES performance, while DIFF outperforms FULL in all but one month and K-ADAPTER uses around 12 times less training time than FULL.Continual-learning methods match or surpass DIFF on CHANGED instances without sacrificing new-knowledge learning to preserve UNCHANGED knowledge.Source: §5.4 · pp. 7–8, Table 3 · p. 7
  • With temporal misalignment, FULL is influenced mainly by the number of training updates, while DIFF and K-ADAPTER generally perform strongly when training and evaluation periods align.K-ADAPTER’s perplexity does not increase much under temporal misalignment.Source: §5.4 · p. 8, Figure 4 · p. 8

6 Conclusion

TEMPORALWIKI combines TWIKI-DIFFSETS training corpora with TWIKI-PROBES evaluation datasets for automated, periodic updates. Experiments report better acquisition of updated knowledge at 12 times less computational cost than updating on entire Wikipedia snapshots, while continual-learning methods mitigate forgetting.

6 Conclusion

  • TWIKI-DIFFSETS yields better acquisition of new and updated knowledge than entire Wikipedia snapshots at 12 times less computational cost.The benchmark also shows an overall stability-plasticity enhancement compared with updating on entire snapshots.Source: §6 · p. 8
  • Continual learning methods—regularization, rehearsal, and parameter expansion—mitigate forgetting and robustly improve the overall stability-plasticity trade-off on TWIKI-PROBES.Source: §6 · p. 8
  • TEMPORALWIKI is an automated lifelong benchmark comprising TWIKI-DIFFSETS for temporal language modeling and TWIKI-PROBES for measuring the stability-plasticity trade-off.Source: §6 · p. 8
  • Temporal language modeling remains challenging, particularly as the number of LM updates increases.Source: §6 · p. 8

7 Limitations

Wikipedia and Wikidata changes do not necessarily correspond to real-world knowledge changes, and this work does not cover knowledge deletion. The examples show changed probe facts aligned with corresponding sentences in the diff training corpus.

7 Limitations

  • Wikipedia and Wikidata updates do not necessarily represent actual changes in real-world knowledge.A new Wikipedia page may contain existing knowledge, and Wikidata may add facts already known but previously missing from its coverage.Source: §7 · p. 9
  • Knowledge deletion is outside this work, although addressing deleted misinformation and bias in LMs is identified as necessary future work.Source: §7 · p. 9

A Examples of TWIKI-DIFFSETS and TWIKI-PROBES

B Details of Entity Types of Subject and Relation

Figure 6 concerns the entity types of subjects and objects in TWIKI-PROBES and compares UNCHANGED with CHANGED instances.

B Details of Entity Types of Subject and Relation

C Details of Relation Distribution

Figure 7 presents relation distributions for UNCHANGED and CHANGED TWIKI-PROBES instances. The configuration passage specifies the compute and training settings for continual pretraining and light-tuning.

C Details of Relation Distribution

D Continual Pretraining and Light Tuning Configuration

E Light-tuning results with TWIKI-PROBES

Light-tuning results are affected by template mismatch and tuning-sample choice. F1 trends are consistent with zero-shot perplexity, and K-ADAPTER performs robustly across factual categories.

F Light-Tuning Data

The light-tuning data are sampled from WikiData by time step and factual-instance category, with Relation frequencies kept proportional to their original distribution.

Loading 2204.14211v3…