PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-Aware Language Models as Temporal Knowledge Bases

Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen

arXiv:2106.15110v2cs.CL

TL;DR

Language models encode facts that change over time, but existing evidence and benchmarks offer limited insight into their temporal scope. This paper introduces TEMPLAMA and timestamp-conditioned pretraining, reporting improved fact memorization and future calibration alongside more efficient model updates.

  • Problem

    Evidence on how pretrained language models represent time-scoped facts is limited, while common QA benchmarks focus on a single time period.

    Source: §1 · p. 2, §6 · p. 11

  • Method

    The paper introduces TEMPLAMA, a diagnostic dataset for time-sensitive knowledge, and conditions masked-language-model pretraining on timestamps represented by time-string prefixes.

    Source: §1 · p. 2

  • Results

    Time-aware pretraining improves fact memorization and future calibration; updates achieve similar performance to scratch retraining with 30× fewer steps without degrading older knowledge.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Providing text creation dates can help mitigate models’ limitations in remembering the past and estimating the future.

    Source: §4 · p. 9

    Publication dates may not match the temporal scope of all events described in an article, so better strategies for assigning dates to sentences remain a direction for future work.

    Source: §4 · p. 9

Abstract

from arXiv · show

Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. But language models (LMs) are trained on snapshots of data collected at a specific moment in time, and this can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum -- those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently "refreshed" as new data arrives, without the need for retraining from scratch.

1 Introduction

Language models trained on time-specific data can mismanage changing facts, and this paper introduces TEMPLAMA and timestamp-conditioned pretraining to improve temporal recall, future calibration, and efficient model updates.

Introduction

  • Snapshot-trained models may average conflicting answers, forget facts from underrepresented periods, and miscalibrate confidence on questions beyond their training scope.The appropriate confidence for future facts depends on how predictable they are; for example, a capital may be more predictable than a governor.Source: §1 · p. 1
  • TEMPLAMA probes time-sensitive knowledge with fill-in-the-blank queries whose answers vary over time, and experiments find evidence of the identified temporal-knowledge problems.The paper motivates the dataset by noting that standard QA datasets focus on a single time period, even for temporally scoped questions.Source: §1 · p. 2
  • Uniformly sampling training-period documents and conditioning masked language modeling on time improves recall while reducing forgetting and averaging across conflicting facts.The experiments use news articles from 2010–2018 and assign facts to year-based time buckets.Source: §1 · pp. 1–2
  • Jointly modeling text and time improves perplexity on unseen future years, degrades more gracefully on TEMPLAMA, and yields faster entropy growth for frequently changing facts.The entropy comparison is qualitative and uses hand-crafted queries about facts likely to change frequently, rarely, or never.Source: §1 · p. 2
  • Similar performance to retraining from scratch was achieved with 30× fewer steps, without degrading knowledge encoded by older data.Temporal pretraining implicitly buckets data into non-overlapping time slices, making models more amenable to post-hoc finetuning.Source: §1 · p. 2

2 Methods

The methods probe masked factual spans with timestamps and compare ways to condition T5 models on time. CUSTOMNEWS supplies dated news text for adaptation, while TEMPLAMA provides temporally sensitive factual queries for evaluation.

2.1 Datasets

  • CUSTOMNEWS samples one million news articles per year from 2010–2020 and uses salient-span masking of sentences, focusing on named entities and dates.The corpus is adapted from dated web news and converted from autoregressive language modeling to a masked-language-modeling objective.Source: §2.1 · p. 3
  • TEMPLAMA contains 50,310 year-specific queries across 11 years, built from facts with changing objects and manually written cloze templates.It starts from a November 2020 Wikidata snapshot, selects nine relations, and includes overlapping valid object intervals as correct answers.Source: §2.1 · p. 3

2.2 Training and evaluation

  • Models train on 2010–2018 and are tested on those years and 2019–2020; TEMPLAMA splits keep subject entities disjoint across train, validation, and test.Token-level F1 is the main metric, with maximum F1 used for queries that have multiple targets.Source: §2.2 · p. 4
  • Training mixes CUSTOMNEWS and TEMPLAMA at a 1000:1 input ratio, adapts public T5 checkpoints for 300K steps, and primarily uses the 770M-parameter Large model.The study also compares Small (110M) and XXL (11B) models; held-out CUSTOMNEWS loss was still improving at 300K steps, though overall trends were stable.Source: §2.2 · p. 4, §2.3 · pp. 4–5

2.3 Jointly Modeling Text and Time

  • The study compares yearly T5 experts with a single Temporal T5 that receives a year prefix, alongside a Uniform model trained without time input.Yearly experts are routed by timestamp, using the closest expert for times outside 2010–2018; Uniform training shuffles examples sampled equally across time slices.Source: Figure 1 · p. 3, §2.3 · p. 4

3 Experiments

Across experiments, conditioning language models on time improves memorization of temporally scoped facts and supports more calibrated predictions about future change. The experiments also show that temporal models can be refreshed with new data while better retaining older knowledge than simple finetuning does.

3.1 Memorizing Facts Across Time

  • For 2010–18, Yearly and Temporal improve over Uniform; Yearly performs better on CUSTOMNEWS and Temporal on TEMPLAMA; the cited benchmark values are 17.8 for TEMPLAMA and 28.5 for Natural Questions.Source: §3.1 · p. 5
  • Year-specific models forget as the train–test year gap grows, with larger drops for future years; Uniform favors long-lived answers, while year-specific models favor answers lasting under five years.Temporal context trades off these extremes and yields higher overall F1; Uniform also often answers with an entity more salient during training.Source: §3.1 · p. 5, Figure 2 · p. 6
  • Larger models gain more from temporal context; when training spans 2004–18 rather than 2010–18, Temporal still beats Uniform but by a smaller gap.The results suggest increasing model size when extending the training time span.Source: §3.1 · p. 5, Table 3 · p. 6, Table 6 · p. 7
  • On closed-book CronQuestions, Uniform and Temporal improve memorization, with Temporal slightly better for Large and XXL models.CronQuestions contains 410K time-dependent questions; evaluation uses SQuAD-based exact-match and F1 metrics.Source: §3.1 · pp. 5–6, Table 4 · p. 6

3.2.1 Graceful Degradation

  • On unseen 2019–20 data, Temporal improves F1 over Uniform and has the lowest CUSTOMNEWS perplexity on seen and unseen slices.The Yearly ensemble is significantly worse on CUSTOMNEWS but comparable on TEMPLAMA; none of the models predicts TEMPLAMA facts that changed after training, and Temporal mainly carries unchanged facts forward.Source: §3.2.1 · p. 7, Table 7 · p. 7
  • As the input year moves into the future, all models’ confidence falls, but Temporal confidence falls faster for facts with multiple answers across 2010–20.This pattern reflects the intuition that facts which changed in the past are more likely to change again.Source: §3.2.1 · pp. 7–8, Figure 3 · p. 8

3.2.2 Future Relations

  • On future-relation queries, Uniform and Temporal uncertainty rises in an intuitive order, while temporal context further improves calibration for 2019–2022.The manually curated evaluation has 86 queries, and entropy conflates query-inherent uncertainty with model uncertainty; Temporal entropy for rare- and never-changing queries starts rising after 2022.Source: §3.2.2 · pp. 8–9, Figure 4 · p. 8

3.3 Cheaper Adaptation to New Data

  • With α = 0.5, a Temporal model matches scratch-training performance with 10K additional steps versus 300K, while minimally degrading on 2010–18 up to α = 0.6.Across α values, Uniform improves on 2019 but degrades on older slices; the Temporal model adapts while better preserving older-slice performance.Source: §3.3 · p. 9

4 Discussion & Limitations

Providing text dates mitigates models’ difficulty memorizing past facts and estimating future facts, but publication dates only approximate event timing, and the work has dataset, application, and societal limitations.

4 Discussion & Limitations

  • Providing text dates mitigates models’ practical difficulty memorizing past facts and estimating the future; after 50K adaptation steps with 2019 data, Temporal degrades less than Uniform on the 2010–18 slice.Figure 5 describes adaptation to new 2019 data; the result concerns performance on the 2010–18 slice.Source: §4 · p. 9, Figure 5 · p. 10
  • In CUSTOMNEWS, 3.9% of sentences explicitly mention a 2010–18 year and 2.1% mention the publication year; Temporal assigns approximate scope to the remaining 96%.The authors note that better strategies for assigning dates to these sentences are a future-work direction.Source: §4 · p. 9
  • Temporal staleness may also hinder open-book question answering when stale LMs cannot align a query with relevant retrieved text.For example, an LM without training data identifying Iota as a hurricane may fail to match a query about countries affected by the 2020 hurricane season to a passage about Iota’s damage in Nicaragua.Source: §4 · p. 9
  • TEMPLAMA is constructed synthetically from Wikidata, and the study’s temporal partitions and reasoning coverage are limited.Incomplete or incorrect KB facts can produce incorrect queries; a missing start date is assumed to mean validity from the period’s beginning. The datasets use the same yearly slices despite differing natures; other partition lengths, temporal expressions, temporal commonsense (Zhou et al. 2019), temporal ordering (Ning et al. 2020), and events (Zhou et al. 2021) were not investigated.Source: §4 · pp. 9–10
  • Closed-book models may memorize mainly popular-entity facts, risking stereotypes and unfair outcomes.Training the models required 32 Cloud TPU v3 cores for several hundred hours, with significant environmental impact; the authors hope efficient temporal updates will eventually save energy costs.Source: §4 · p. 10

5 Related Work

The paper positions its time-conditioned language model and TEMPLAMA as addressing gaps in lexical-change modeling, temporal adaptation, factual updates, and static knowledge benchmarks.

5 Related Work

  • Unlike lexical-change embedding work (Dubossarsky et al., 2019) and post-hoc BERT analyses (Giulianelli et al., 2020), this work adds time directly to T5 for multi-token, time-aware fact completion.Earlier diachronic embeddings primarily modeled lexical semantic change rather than downstream temporal awareness.Source: §5 · p. 10
  • A single model conditioned on time achieves adaptation benefits and improves memorization of time-sensitive facts.Prior work documented cross-period degradation; Delasalles et al. (2019) improved perplexity on seen and future periods with dynamic author representations, while Röttger and Pierrehumbert (2021) found pretraining adaptation did not help downstream tasks.Source: §5 · p. 10
  • Temporal conditioning addresses deployment on both future and past data: updating on new data alone degrades old-data performance, whereas temporal context alleviates this issue.Lazaridou et al. (2021) documented gradual degradation when models trained up to time T were applied after T and proposed dynamic evaluation as a potential mitigation.Source: §5 · pp. 10–11
  • Unlike editing methods based on revised input-output pairs, this work uses new unlabeled text to update predictions and infer which facts change.Related editing approaches include Sinitsin et al. (2020), Zhu et al. (2020), and De Cao et al. (2021).Source: §5 · p. 11
  • TEMPLAMA evaluates time-sensitive factual knowledge using temporal scopes rather than treating all historical answers as correct for a query.Unlike static LAMA probing (Petroni et al., 2019; Jiang et al., 2020b; Jiang et al., 2020a; Kassner et al., 2021), its masked-LM-style zero/few-shot queries are restricted to subject-relation pairs with multiple objects over time and balanced across 2010-2020; unlike TempQuestions and CronQuestions (Saxena et al., 2021), it uses that format.Source: §5 · p. 11

6 Conclusion

T5 performs poorly on temporally scoped facts, and although news-domain training improves performance, conditioning on time prefixes further improves memorization. Time-aware language models also better calibrate expected future changes and adapt more cheaply to new timestamped data.

6 Conclusion

  • Conditioning on time-string prefixes improves fact memorization beyond simply training on more data, motivating the proposed time-aware language model.Source: §6 · p. 11
  • T5 performs poorly on temporally scoped facts, while training on news-domain data improves it significantly.Source: §6 · p. 11
  • Time-aware language models better calibrate expected future changes and adapt more cheaply to new timestamped data.Source: §6 · p. 11

Supplementary Material · A TEMPLAMA Templates

TEMPLAMA builds year-specific queries from facts in nine WikiData relations by instantiating relation templates with the subject and an answer slot.

A TEMPLAMA Templates

  • TEMPLAMA uses nine WikiData relations to construct its templates.Table 8 lists the relations used.Source: A TEMPLAMA Templates · p. 11
  • Each query replaces “<subject>” with the subject entity’s name and “<object>” with “__X__”; the answer is the corresponding object entity’s name.Source: A TEMPLAMA Templates · p. 11
  • A separate query is constructed for each year that a fact is valid.Source: A TEMPLAMA Templates · p. 11

B Future Relations

The Future Relations dataset probes model calibration on answers expected to change at different rates, using queries about US cities and countries.

B Future Relations

  • Queries were constructed by searching lists of events and popular athletes and by issuing targeted queries to the WikiData Query Service.Source: B Future Relations · p. 11
  • The dataset tests calibration over future years with queries whose answers change frequently, rarely, or never; its examples ask for a US city or a country.The table groups the query types by how often their answers intuitively change.Source: Table 9 · p. 12

Related papers

Loading 2106.15110v2…