Source-linked AI summary
Time-Aware Language Models as Temporal Knowledge Bases
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen
TL;DR
The paper studies how LMs handle factual knowledge that changes over time and proposes conditioning pretraining on timestamps. Time-aware models improve memorization and future calibration, while supporting cheaper refreshes as new timestamped data arrives.
Problem
Pretrained LMs have limited evidence about how to encode and update facts whose truth changes over time, despite their use as repositories of real-world knowledge.
Method
The paper introduces TEMPLAMA and modifies masked-language-model pretraining to jointly model text with timestamp prefixes over uniformly sampled temporal data.
Results
Time-aware pretraining improves memorization of temporally scoped facts, calibration on future periods, and refreshing efficiency without degrading older knowledge.
Takeaways & Limitations
Temporal context provides a practical way to distinguish facts from different time periods and adapt pretrained LMs as new data arrives.
Takeaways & Limitations
The approach uses publication dates as approximate temporal scope, although articles may describe historical or future events; the authors identify better date assignment as future work.
Abstract
from arXiv · showhide
Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. But language models (LMs) are trained on snapshots of data collected at a specific moment in time, and this can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum -- those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently "refreshed" as new data arrives, without the need for retraining from scratch.
1 Introduction
The paper examines how language models encode facts whose truth changes over time and introduces time-aware pretraining to improve temporal knowledge handling and updates.
- Motivation: Temporally changing facts pose two central questions: whether pretrained LMs learn appropriate temporal scope and how their knowledge should be updated.
- Motivation: Snapshot-based pretraining can create averaging between conflicting facts and forgetting because training generally ignores temporal metadata.
- Dataset: TEMPLAMA is a diagnostic dataset of fill-in-the-blank queries whose answers vary with time, enabling empirical evaluation of temporally sensitive LM knowledge.
- Method: The proposed method conditions masked language modeling on time and samples documents uniformly across the training timespan, separating facts into time buckets.For T5, time can be represented by prefixing the input with a string such as “year: 2018.”
- Results: Jointly modeling text and time improves future-year perplexity and produces more graceful degradation on TEMPLAMA, with uncertainty rising fastest for frequently changing facts.
- Results: Temporally scoped pretraining enables updates with similar performance to retraining from scratch using 30× fewer steps, without degrading knowledge from older data.
2 Methods
The paper evaluates time-sensitive factual knowledge using dated news pretraining and a temporal cloze dataset, then compares models that ignore time, specialize by year, or condition one model on timestamps. Training and evaluation separate 2010–2018 seen years from 2019–2020 future years, with token-level F1 as the main metric.
- Training approaches: Uniform, Yearly, and Temporal setups respectively ignore time, use separate year-specific experts, or prepend time prefixes to a single model.Yearly routing uses the closest yearly expert for timestamps outside the training range, whereas Temporal trains one model across the data.
- Datasets: CUSTOMNEWS contains dated news articles sampled at 1M articles per year from 2010–2020 and adapted for masked span prediction.Articles are split into sentences, with named entities and dates selected as salient masked spans.
- Datasets: TEMPLAMA probes facts whose answers vary over time using 50,310 queries spanning 11 years.Queries are constructed from temporally changing Wikidata relations and use yearly timestamps with entity-disjoint train, validation, and test subjects.
- Training and evaluation: Models are trained for 300K steps, evaluated on 2010–2018 seen years and 2019–2020 future years, and scored primarily with token-level F1.TEMPLAMA splits are disjoint by subject entity to prevent test facts from appearing in training.
- Training and evaluation: The study trains T5 variants on masked span prediction, modeling targets as P(y|x, t; θ) given text, target spans, and timestamps.The temporal setup can prepend a string such as “year: 2018” to each input, while all models start from public T5 checkpoints.
- Baselines: The experiments compare time-aware variants against T5-CBQA and a Uniform model trained on the same uniformly sampled data without time inputs.The Uniform model shuffles examples rather than presenting them chronologically, isolating the effect of explicit temporal conditioning.
3 Experiments
The experiments diagnose averaging and forgetting in temporally scoped knowledge, then evaluate temporal conditioning for memorization, future calibration, scaling, and efficient adaptation. Temporal context improves performance on time-sensitive facts, degrades more gracefully on future data, and supports rapid refreshing without forgetting older knowledge.
- Memorizing Facts Across Time: 17.8 versus 28.5: T5-CBQA and T5-CBQA-ft score lower on TEMPLAMA than on Natural Questions, while news-domain training improves temporally scoped knowledge.Uniform training improves over T5-CBQA-ft, and temporal conditioning improves further over Uniform.
- Memorizing Facts Across Time: Temporal conditioning outperforms Uniform on TEMPLAMA, while Yearly performs better on CUSTOMNEWS, reflecting differences in fact duration across datasets.Yearly requires linearly more parameters and compute because it ensembles nine yearly models.
- Memorizing Facts Across Time: Models trained on a single year forget facts from both earlier and later years, with larger declines for future years.The experiments vary the gap between training and test years to expose this asymmetry.
- Scaling and Longer Time Spans: Larger models benefit more from temporal context, and the advantage persists when training spans 2004–2018, although the gap over Uniform becomes smaller.The longer span requires memorizing more facts, suggesting that model size should also increase.
- Future Relations: Temporal context improves F1 on unseen 2019–20 slices by preserving unchanged facts, although no model predicts TEMPLAMA facts that change after training.On CUSTOMNEWS, the Temporal model also performs better on the SSM objective for unseen articles.
- Graceful Degradation: At α = 0.5, 10K adaptation steps match a Temporal model trained from scratch for 300K steps without forgetting older facts.Uniform also improves on 2019 but degrades on 2010–18, whereas Temporal shows minimal degradation up to α = 0.6.
4 Discussion & Limitations
The paper finds that temporal context mitigates limitations in memorizing past facts and estimating future changes, but its time representation and evaluations remain narrow.
- Temporal context mitigates current models’ practical limitations in memorizing the past and estimating the future.
- Publication dates provide only a narrow temporal scope because articles can describe historical or future events.In CUSTOMNEWS, 3.9% of sentences explicitly mention a 2010–18 year and 2.1% mention the publication year.
- The study focuses on closed-book question answering, although stale language models may also affect retrieval alignment in open-book question answering.
- TEMPLAMA is synthetically constructed from WikiData, so incomplete or incorrect knowledge-base facts can produce incorrect queries.
- During adaptation to 2019 data, the Temporal model degrades less than Uniform on the 2010–18 slice.
- The closed-book models may memorize mainly common facts about popular entities, risking stereotype reinforcement and unfair outcomes.
5 Related Work
The paper extends temporal modeling from earlier work on diachronic embeddings and temporal generalization by conditioning a contextualized language model directly on time.
- Earlier diachronic-embedding work models changes in individual word representations, often to detect lexical semantic change.
- Unlike post-hoc contextualized-embedding analyses, this work adds a time-sensitive component directly to T5 for time-aware fact completion.
- Prior studies document degradation when models encounter test data from periods different from their training data.
- Production temporal generalization includes applying models trained through time T to data after T, while models are also applied to historical data.
- Updating on only new data degrades performance on old data, whereas temporal-context conditioning alleviates this issue.
- Unlike neural knowledge-editing methods using revised input-output pairs, this setting updates factual predictions from new unlabeled text.
- LAMA-style benchmarks probe memorized factual knowledge but generally assume that knowledge is static across time.
- TEMPLAMA resembles temporal KB-QA datasets but uses masked-language-model-style cloze probes and restricts queries to facts with multiple objects over time.
6 Conclusion
The paper concludes that time-aware language models improve temporal factual knowledge, future calibration, and adaptation to new timestamped data.
- Time-aware language models condition on string prefixes of time to represent temporally-scoped facts.
- T5 performs poorly on temporally-scoped facts, while news-domain training improves performance significantly.
- Conditioning on temporal context improves memorization beyond simply training on more data.
- Time-aware models provide better calibration of expected future changes and cheaper adaptation to new timestamped data.
A TEMPLAMA Templates
TEMPLAMA converts temporally changing WikiData facts into yearly masked-language-model queries whose answers are the corresponding object entities.
- TEMPLAMA uses nine WikiData relations to construct natural-language queries from temporally valid facts.
- Each query replaces the subject placeholder with a subject entity and the object placeholder with the masked token __X__.
- A separate query is constructed for every year in which the fact is valid.
B Future Relations
The Future Relations dataset tests model calibration on changing facts across future years. Its queries are organized by how frequently their answers change and whether the answer is a US city or country.
- Queries were constructed by searching for lists of events and popular athletes and issuing targeted queries to the WikiData Query Service.
- The table’s top section contains queries answered by US cities, while its bottom section contains queries answered by countries.
- The dataset evaluates model calibration over future years using queries about facts that change at different frequencies.The query categories cover answers that change frequently, yearly, every few years, rarely, or never.