Source-linked AI summary
TimeLMs: Diachronic Language Models from Twitter
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, Jose Camacho-Collados
TL;DR
Static language models leave a gap in handling temporal change, particularly on rapidly evolving Twitter data. TimeLMs addresses this by training time-specific Twitter models with quarterly continual updates. The models remain competitive on Twitter benchmarks, while pseudo-perplexity rises almost 10% after one year and generally improves with quarterly updating.
Problem
Static language-model development and evaluation limit generalization to future data, while rapidly changing social-media topics require diachronic specialization.
Method
TimeLMs trains a Twitter language-model base through 2019 and continually updates it every three months with later time-distributed data.
Results
PPPL scores tend to increase almost 10% after one year, while quarterly updates generally reduce PPPL and TimeLM-21 outperforms a similar TweetEval model in most tasks.
Takeaways & Limitations
TimeLMs provides free-to-use, up-to-date language models and shows that time-aware training is relevant for evaluating future data.
Takeaways & Limitations
The irony evaluation may contain label leakage because BERTweet was trained on tweets from the dataset’s 2014/15 period.
Abstract
from arXiv · showhide
Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models' capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift.
1 Introduction
NLP language models are largely static despite the importance of time, limiting their alignment with future data, especially on rapidly changing social media. TimeLMs addresses this gap with time-specific Twitter language models and ongoing releases.
- Static model development and evaluation limit generalization to future data and can reduce experimental reliability through benchmark–pretraining overlap.
- Rapidly changing social-media topics make the lack of diachronic specialization particularly concerning.
- TimeLMs provides time-specific language models specialized to Twitter data, with models planned for periodic training, improvement, and release.
2 Related Work
Prior work addresses temporal modeling, continual learning, and domain adaptation, while Twitter language models largely leave language change over time unaddressed. TimeLMs places the time variable at the center of Twitter specialization and supports usable releases.
- Existing NLP work includes temporal language representations, continual learning, and domain adaptation through pretraining on specialized corpora.
- Twitter-focused models such as BERTweet, TweetEval, and XLM-T are prominent social-media language models.
- These Twitter models barely address diachronic language, and prior efforts do not make time central to social-media language-model specialization.
- TimeLMs aims to address these gaps while releasing time-aware models with usable software and reliable infrastructure.
3 TimeLMs: Diachronic Language Models from Twitter
TimeLMs trains Twitter language models for successive time periods using a base model followed by quarterly continual updates. The released models are built from cleaned, time-distributed tweet corpora and made available through Hugging Face.
- Data collection: Tweets are collected through the Twitter Academic API at five-minute intervals across hours, days, and yearly quarters, with English-only and content filters.
- Data cleaning: The corpus removes highly active users, duplicates, and near-duplicates, and replaces most user mentions with @user.
- The training pipeline first builds a base model from data through 2019, then continually trains language models every three months.
- Model release: The base 2019-90M model uses 90 million tweets, while quarterly updates add 4.2M tweets and later models reach 124M tweets.
- Evaluation data: Quarter-specific test sets contain 300K independently sampled and cleaned tweets per quarter.
4 Evaluation
Evaluation combines Twitter classification benchmarks with time-aware pseudo-perplexity across periods. The results assess competitiveness, temporal degradation, and whether continual updating improves performance on newer data.
- Evaluation design: The evaluation uses seven-task TweetEval classification and pseudo-perplexity on corpora from different time periods, with the latter as the core time-aware evaluation.
- TweetEval: TweetEval covers seven heterogeneous tasks, including emoji prediction, emotion, hate speech, irony, offensive language, sentiment, and stance detection.
- Caveat: The irony dataset may contain label leakage because BERTweet was trained on tweets from the dataset’s 2014/15 period.
- TweetEval results: TimeLM-21 outperforms the similar TweetEval model in most tasks and remains generally competitive on past tweets.
- Time-aware evaluation: PPPL scores tend to increase almost 10% after one year, while quarterly updates generally reduce PPPL consistently.
5 Python Interface
The released Python interface provides streamlined access to TimeLMs and related evaluation tools, including time-specific model selection, pseudo-perplexity scoring, language-model evaluation, and masked-token prediction. It also supports testing across quarterly data and matching tweets to models based on their dates.
- Interface and model selection: The interface offers easy access to language models trained for specific periods through the Transformers package and Hugging Face hub.Users can select models using predefined modes, custom dates or quarters, and quarterly evaluation.
- Interface and model selection: Users can select models by using the model corresponding to each tweet’s quarter, a custom date or quarter, or all available quarterly models.The corresponding mode requires tweet dates, while examples’ created_at fields are otherwise optional.
- Supported functions: The interface supports pseudo-perplexity scoring, evaluation on released or customized test sets, and masked-token prediction.Pseudo-perplexity can be computed for supplied tweets or texts, including a provided random subset of test data from 2020 and 2021.
- Quarterly evaluation: Table 3 compares pseudo-perplexity across quarterly Twitter test sets and reports changes between previous-quarter and same-quarter-updated models.Pseudo-perplexity is lower-is-better, and each quarter spans three months.
- Supported functions: The evaluation methods return model-specific pseudo-perplexity scores and the pseudo-log-likelihood scores assigned to individual tweets.The eval_model method also supports other models available through the Transformers package.
6 Analysis
The analysis examines temporal degradation, the relative roles of training size and recency, and qualitative evidence that quarterly models track period-specific trends. Across these analyses, newer data generally favors more recent models, while continuous updating preserves performance on earlier data.
- 6.1 Degradation analysis: Up to 13.68% degradation occurs when the earlier 2020-Q1 model is evaluated on the latest 2021-Q4 data.The comparison uses each model’s latest training period as the reference point.
- 6.1 Degradation analysis: 2021-124M generally performs better on early 2020 quarters, whereas quarterly updating performs slightly better on the latest quarters.The 2021-124M model was trained on all 2020–2021 data at once, while 2021-Q4 updates 2021-Q3.
- 6.2 Time and size control experiment: The 2021-Q3-2x model outperforms default 2021-Q3 across all tested quarters, while 2021-Q4 performs best on the most recent 2021-Q4 test set.The control experiment matches 2021-Q3-2x’s total training tweets to the 2021-Q4 model, separating training size from recency.
- 6.3 Qualitative analysis: Quarterly models predict period-specific masked tokens differently, with the most pertinent quarter showing background knowledge more aligned with contemporary trends.Examples include increasing awareness of being fully vaccinated in COVID-related tweets and changed “Squid Game” predictions in 2021-Q4.
- 6.3 Qualitative analysis: More recent models achieve better PLL scores for tweets about contemporary events such as NFTs and “Squid Game,” but stagnate or degrade on an older reality-show tweet.The analysis uses tweets requiring awareness of trends or events tied to specific periods.
7 Conclusion
TimeLMs are trained on Twitter data from different time periods and supported by an ongoing release initiative. The evaluation shows that time-aware training matters in practice because performance degrades on future data.
- TimeLMs are language models trained on Twitter over different time periods.
- The initiative plans to train and release updated language models every three months.The models are intended to remain free to use and up to date.
- A Python interface supports model loading, use, and time-aware evaluation.
- Evaluation shows clear performance degradation when models are applied to future data.This setting is described as common in practice.
- Future work will integrate the time-span variable into language models using string prefixes.