Source-linked AI summary
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, Phil Blunsom
TL;DR
The paper asks whether static, time-overlapping language-model evaluation reflects performance on future, changing language. It introduces time-aware benchmarks and compares stale models with controls, finding that temporal degradation worsens over time, is not solved by model scaling alone, and can be mitigated by continual updating.
Problem
Static language-model training and evaluation overlook temporal generalization to future utterances, despite practical applications requiring up-to-date factual knowledge.
Method
The paper introduces dynamic streaming benchmarks and evaluates Transformer-XL models on future WMT and arXiv data using time-stratified experiments.
Results
Transformer-XL performance becomes increasingly worse with time; models can be up to 16% worse on articles published up to two years after training, and larger size alone does not solve the problem.
Takeaways & Limitations
Language-model evaluation should use temporal splits and timestamps, while adaptive approaches should keep models up to date with changing information.
Takeaways & Limitations
Dynamic evaluation improves future-data generalization but causes catastrophic forgetting of past data and does not fully solve temporal generalization.
Abstract
from arXiv · showhide
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone -- a key driver behind recent progress -- does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world. We publicly release our dynamic, streaming language modelling benchmarks for WMT and arXiv to facilitate language model evaluation that takes temporal dynamics into account.
1 Introduction
Current language-model evaluation mostly uses time-overlapping data, so it misses temporal generalization to future utterances and risks test contamination. The paper argues for timestamp-aware evaluation and adaptive models that remain up to date.
- Static pretraining and evaluation on overlapping time periods do not assess generalization to future data.This matters for systems deployed on future user utterances and evolving factual questions.
- Temporal overlap increases the risk that evaluation data are unknowingly included in pretraining data.Language-model data are time series rather than i.i.d. samples.
- Up to 16% worse perplexity occurs when Transformer-XL predicts articles published up to two years after training ends.Performance also becomes increasingly worse with time.
- Increasing model size alone does not solve temporal generalization: larger models show the same degradation over time.A smaller model trained on more recent data can outperform a 60% larger model lacking recent data.
- The paper advocates timestamp-aware benchmarks and adaptive language models that continually incorporate new information.It releases dynamic streaming benchmarks for WMT and arXiv.
2 Time-stratified language modelling
The study evaluates temporal generalization by training Transformer-XL models on past data and testing them on later articles. A same-sized, time-overlapping control enables relative comparison of temporal degradation.
- TIME-STRATIFIED setup: The TIME-STRATIFIED setup trains on documents through September 2017 and tests on 2018–2019 articles published up to two years later.The final three months of 2017 serve as validation data.
- CONTROL setup: The CONTROL setup trains on documents from the same 2018–2019 period as evaluation, excluding test documents.It approximates prevailing static language-model practice.
- CONTROL setup: Both training sets have exactly the same size, differing only in the time periods represented.Both setups use the exact same 2018–2019 test set for fair perplexity comparison.
- Relative perplexity comparison: Temporal degradation is measured through relative perplexity differences between TIME-STRATIFIED and CONTROL models for each test month.Absolute month-to-month perplexity is unreliable because document lengths vary.
- Model setup: The experiments use a 287M-parameter Transformer-XL with 18 layers and 1,024 hidden units.Larger models are explored later.
3 Language Modelling Experiments & Analysis
Transformer-XL models trained only on past data become increasingly worse at predicting future text, especially rapidly changing entities, topics, and emerging words. The analyses show that temporal degradation is strongest for proper nouns and numbers, and that long-range context can partially support adaptation.
- Up to 16% perplexity difference occurs when stale TIME-STRATIFIED models predict future articles instead of CONTROL models exposed to the test period.The models use the same dataset sizes and architectures; the main difference is the training-data time period.
- Perplexity increases as test data move farther from training, with the same temporal degradation pattern across datasets, test years, longer gaps, and German WMT.The upward slope indicates that models become increasingly outdated over time.
- Analysis: Performance degrades most rapidly for proper nouns and numbers, while common-noun degradation drives the overall trend; politics and sports change faster than lifestyle topics.The analysis links proper-noun degradation to changing named entities and identifies faster-changing topic distributions.
- Perplexity and temporal frequency shifts: The TIME-STRATIFIED model has almost 5x worse perplexity on EMERGING NEW WORDS than overall text, measuring 110 versus 22.EMERGING NEW WORDS are frequent test words that were unseen during training or had at least 5 times lower training unigram probability.
- Perplexity and temporal frequency shifts: Perplexity falls from approximately 694.95 on first occurrences of emerging words when they reappear within Transformer context, but exceeds 2,700 when the first occurrence lies outside memory.This shows that context-based copying helps temporal generalization, whereas limited memory prevents that benefit.
- Importance: The targeted evaluation emphasizes named entities, fast-changing topics, and adaptation to emerging words rather than relying on overall perplexity alone.These dimensions are presented as benchmarks for aspects of temporal generalization that matter most.
4 The effect of outdated models persists even when increasing model sizes
Increasing model size does not prevent temporal degradation in language models. By late 2019, a smaller but more up-to-date model outperforms a substantially larger, outdated model, motivating updates with new information.
- A 60% larger TIME-STRATIFIED model achieves lower absolute perplexity but does not significantly reduce temporal degradation.The 448M model is compared with the 287M model using the slopes of relative perplexity increase over time.
- 5.5% improvement in absolute perplexity does not translate into a significant effect on the temporal-degradation slope.The slope comparison is assessed with a t-test after fitting linear regressions.
- By late 2019, CONTROL287M outperforms TIME-STRATIFIED448M despite being 60% smaller and more up-to-date.This comparison holds on both datasets.
- The results indicate that maintaining temporal performance requires solutions that update model knowledge with new information.
5 Time-stratified question answering
The paper evaluates how outdated language models affect downstream question answering. In synthetic closed-book questions about 2019 government officials, accuracy deteriorates as the training data ends farther from the question year.
- Out-of-date language models can impair downstream tasks, motivating evaluation beyond intrinsic perplexity.
- The study constructs 438 synthetic closed-book questions covering 22 government roles across 11 countries.Questions use the template “Who is the [government role] of [country/state] in [month/year]?”
- Accuracy deteriorates substantially as the pretraining cutoff shifts chronologically away from 2019, the year targeted by evaluation.Models are pretrained with WMT data ending in successive years from 2012 through 2019, then evaluated on questions about 2019 officials.
6 Keeping models up-to-date: Online learning through dynamic evaluation
The paper studies dynamic evaluation as an online-learning approach for updating language models as new documents arrive. Dynamic evaluation slows temporal obsolescence, with especially large gains for emerging words, but does not eliminate degradation and can cause catastrophic forgetting.
- Dynamic evaluation continually updates a pretrained model’s parameters by performing gradient descent on newly arriving data.The approach extends online updates across chronologically ordered documents rather than only within a document.
- Dynamic evaluation reduces the speed at which TIME-STRATIFIED models become outdated across datasets.The effect is significant for ARXIV and WMT based on t-tests of fitted slope coefficients.
- 39.62% ppl. reduction from 109.73 to 66.2 is observed for EMERGING NEW WORDS.This improvement is much larger than WMT’s overall 1.25% reduction from 22.45 to 22.17.
- Lightweight adaptation can target only the embedding layer or bias terms to reduce the resources needed for updating models.The experiments update either 52M embedding parameters or 198K bias parameters.
- Dynamic evaluation alone does not fully solve temporal generalization and can cause catastrophic forgetting of past data.Residual upward slopes remain on WMT and CUSTOMNEWS despite the gentler degradation.
7 Related Work
Related work connects this study to concept drift, out-of-distribution generalization, and continual or streaming language-model learning. The paper emphasizes temporal distribution shifts as a realistic challenge distinct from topic or domain shifts.
- Concept-drift research studies changes in data streams, while NLP work has modeled lexical change using time-separated embeddings and neural networks.
- Out-of-distribution NLP research has largely focused on topic and domain shifts, whereas temporal shifts remain an important realistic challenge.
- Continual and lifelong learning aims to accumulate new knowledge while retaining relevant information from the past.
8 Conclusion
The paper finds that static training and evaluation overestimate generalization to future utterances, while Transformer language models become increasingly outdated over time. It argues that model size alone does not solve this problem and highlights temporal evaluation and updating as important directions.
- Static evaluations using overlapping time periods overestimate models’ generalization to future utterances.
- Transformer language models become increasingly outdated as the temporal distance from training grows.
- Increasing model size alone does not solve temporal degradation.
- Temporal splits and timestamp information can support more realistic evaluation while reducing test-data contamination.
- The released dynamic, streaming benchmarks provide resources for evaluating language models under temporal dynamics.
9 Broader Societal Impact Discussion
The paper connects continually updated language models to reducing harms and biases associated with stale social knowledge, while warning that retraining from scratch is costly. It therefore emphasizes efficient updating methods that also mitigate catastrophic forgetting.
- Keeping NLP models up to date with social trends and public perception is presented as relevant to mitigating harms and biases.
- Periodically retraining large models from scratch incurs increasing computational and environmental costs.
- Efficient, lightweight updating approaches should address both freshness and catastrophic forgetting.
- The benchmark is intended to measure progress on keeping language models up to date.
Checklist
The checklist reports that the paper addresses reproducibility, limitations, societal impacts, asset licensing, and data-related considerations, while noting that code is not released.
- The paper reports its limitations and discusses potential negative societal impacts of working with large-scale language models.
- The authors report that dataset details, preprocessing, training details, and hyperparameters are described and dataset splits are publicly released.
- The authors state that code is not released, but experiments can be repeated using publicly available Transformer implementations.
- The experiments used approximately 186,000 TPU hours, with 32 TPUs for training and one TPU for evaluation in each experiment.
- The released dataset splits point to original publicly available datasets, whose original licenses apply.
A.1 The effect of outdated models persists beyond the 2018/2019 test period.
Temporal degradation persists across shifted test periods and languages, with stale models performing worse as the training–test gap widens. Dynamic evaluation instead updates model parameters using previously seen test documents, but its learning-rate trade-off remains open.
- Similar temporal degradation appears across five test sets created by shifting the evaluation years toward the past.
- 15.23% relative perplexity increase occurs for the stale German TIME-STRATIFIEDde model compared with CONTROLde.
- German degradation increases with the training–test gap, especially for proper nouns and words split into more tokenizer tokens.
- Dynamic evaluation updates Transformer-XL parameters by gradient descent using knowledge from previously seen test documents.
- The dynamic model can learn emerging information and reuse it for later test-document prediction, unlike non-dynamic evaluation with fixed parameters.
- Learning rates are tuned on validation data, while choosing the best adaptation–stability trade-off without prior knowledge of the evaluation period remains future work.
B.1 Dynamic Evaluation and Catastrophic Forgetting
The experiment tests whether dynamic evaluation causes catastrophic forgetting by comparing an outdated model with one updated through 2019 on past and recent data. On both WMT and arXiv, the updated model performs better for recent years but increasingly worse on older years.
- Experiment design: The experiment compares a model trained through 2017 with one updated through 2019 on data from 2007–2017 and recent years.This evaluates whether updating on present data harms performance on the model’s initial training distribution.
- Results: On both WMT and arXiv, the dynamically updated model outperforms the outdated model for 2018 and 2019.
- Results: As evaluation moves toward the past, the updated model’s perplexity increases relative to the model trained through 2017.Figure 9 measures catastrophic forgetting as relative perplexity increase, with years shown in reverse chronological order.