PaperlayerView exact revision on arXiv

Source-linked AI summary

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren

arXiv:2110.08534v3cs.CL

TL;DR

Lifelong pretraining asks whether continually adapting pretrained language models to shifting corpora can improve downstream performance while retaining earlier knowledge. The paper evaluates continual learning over domain-incremental research papers and chronological tweets, finding distillation-based approaches most effective across retention, adaptation, and temporal generalization.

  • Problem

    Static pretrained language models may become outdated as domains, language, and data distributions shift, motivating study of continuous adaptation for downstream performance.

  • Method

    The paper formulates lifelong language model pretraining, constructs domain-incremental and chronological text streams with downstream tasks, and systematically evaluates continual pretraining algorithms.

  • Results

    Distillation-based approaches are most effective for knowledge retention while also improving adaptation to latest data and temporal generalization.

  • Takeaways & Limitations

    The findings motivate further research on continual pretraining and continual learning algorithms for adapting language models to emerging corpora.

  • Takeaways & Limitations

    The setup sequentially updates the model without access to earlier corpora, reflecting privacy and computation constraints but limiting retraining over all prior data.

Abstract

from arXiv · show

Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data distributions that deviate from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after fine-tuning). We evaluate PTLM's ability to adapt to new corpora while retaining learned knowledge in earlier corpora. Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains. The algorithms also improve knowledge transfer, allowing models to achieve better downstream performance over the latest data, and improve temporal generalization when distribution gaps exist between training and evaluation because of time. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.

1 Introduction

The paper formulates lifelong language model pretraining as continual adaptation of pretrained language models to sequentially emerging domains and time-shifted data. It introduces evaluation streams and compares continual-learning approaches, finding distillation-based methods strongest for retention while improving adaptation and temporal generalization.

  • 1 Introduction: Shifting domains, vocabulary, and language over time can make fine-tuning from a static pretrained model an ineffective initialization for downstream tasks.The introduction motivates continual adaptation by contrasting deployed data with the corpora used for initial pretraining.
  • 1 Introduction: The paper defines lifelong language model pretraining to maintain and adapt a pretrained model as distinct domains and time-stamped corpora emerge sequentially.This addresses the limitation of static or outdated pretraining when deployed on shifting distributions.
  • 1 Introduction: The paper creates pretraining data streams, downstream tasks, and a systematic evaluation protocol for studying continual pretraining algorithms.The testbed simulates practical maintenance and adaptation of pretrained language models over emerging corpora.
  • 1 Introduction: Distillation-based approaches are most effective for retaining knowledge in the domain-incremental research paper stream.The evaluation compares model-expansion, memory-based, and distillation-based continual-learning algorithms.
  • 1 Introduction: Distillation-based approaches also improve adaptation to latest data and temporal generalization in the chronologically ordered tweet stream.The paper evaluates retention on research papers and latest-data adaptation plus temporal generalization on tweets.

2 Problem Formulation

The paper formulates lifelong pretraining as sequentially updating a language model across evolving unlabeled data domains without revisiting earlier corpora. It evaluates retention, adaptation, and temporal generalization using domain-incremental research papers and chronologically ordered tweets.

  • 2.1 Lifelong Pretraining of PTLMs: A PTLM visits T unlabeled corpora sequentially as their domain distributions evolve through topical or temporal shifts.The stream can move from computer science to biomedical papers or from past to recent tweets.
  • 2.1 Lifelong Pretraining of PTLMs: The model updates on each corpus without accessing earlier data, reflecting privacy and computational constraints.The study uses RoBERTa-base and denotes the checkpoint after domain t as f_t.
  • 2.1 Lifelong Pretraining of PTLMs: Checkpoint utility is measured after fine-tuning on downstream tasks associated with domains already visited, without access to pretraining corpora.This setup evaluates whether sequential pretraining preserves useful knowledge for downstream applications.
  • 2.2 Data Streams & Downstream Datasets: The study constructs two streams: sequential research-paper domains and tweets ordered by year.The paper stream covers biomedical, computer science, material science, and physics; the tweet stream uses 2014, 2016, 2018, and 2020 corpora.
  • 2.2 Data Streams & Downstream Datasets: Downstream evaluation spans paper-domain tasks such as relation extraction and named entity recognition, plus tweet hashtag and emoji prediction.Tweet tasks use label ranking average precision and Macro-F1, respectively, with 1M tweets held out from each year.
  • 2.3 Evaluation Protocol: The evaluation protocol measures knowledge retention and transfer across earlier domains, emphasizing a single model applicable to all paper-stream domains.Retention is assessed using downstream performance on earlier or current domains visited during pretraining.
  • 2.3 Evaluation Protocol: It also measures adaptation to the latest domain and temporal generalization from earlier-time training data to latest-time evaluation data.Latest-domain adaptation uses f_T on latest-domain tasks, while temporal generalization fine-tunes on S_t and evaluates on S_T for t < T.

3 Methods

The methods compare model-expansion, memory-based, regularization-based, and distillation-based approaches for continual language-model pretraining. Distillation methods preserve prior-model outputs or representations using current-domain and replay examples.

  • Baselines and method categories: The study compares continual-learning methods spanning model expansion, memory replay, regularization, and knowledge distillation, alongside task-specific, sequential, and multitask-pretraining baselines.Sequential pretraining processes domains in order, whereas multitask learning retrains over all previously seen corpora but requires retaining earlier data.
  • Model expansion and regularization: Adapter methods add trainable domain-specific modules while freezing the remaining model, whereas Layer Expansion learns separate top transformer layers and prediction heads per domain.Online EWC is also evaluated as a regularization baseline that penalizes parameter changes.
  • Memory replay: Experience Replay mitigates forgetting with a fixed-size memory containing balanced examples from all seen domains, periodically replayed during pretraining.The default memory contains 100k examples and is updated after each domain finishes.
  • Knowledge distillation: Distillation-based approaches retain one previous model checkpoint and regularize differences between previous and current models using stream or replay examples.The paper adapts logit, representation, contrastive, and SEED distillation techniques to continual pretraining; SEED is also combined with logit distillation.

4 Results

Across research-paper and tweet streams, lifelong pretraining generally improves retention, transfer, adaptation to newer data, and temporal generalization. Distillation-based continual learning is strongest overall, although it can limit learning in later domains and requires extra computation.

  • Research paper stream: Distillation methods, especially Logit-KD and SEED-Logit-KD, improve downstream F1 over Sequential Pretraining by at least 1.0% on earlier domains D1 and D2.They do not improve later domains D3 and D4, possibly because distillation makes the model rigid when acquiring new knowledge.
  • Research paper stream: Task-Specific and Sequential Pretraining consistently outperform RoBERTa-base, particularly on Biomedical and Computer Science domains.These comparisons support lifelong pretraining as a way to retain knowledge across domain corpora.
  • Research paper stream: Performance improvements are more significant in low-resource settings, indicating that lifelong pretraining can improve data efficiency.The analysis evaluates final pretrained models under different numbers of training examples.
  • Research paper stream: Sequential Pretraining generally underperforms Multi-Task Learning except on the final domain, while remaining more computationally efficient and storage-efficient.The authors speculate that continual learning provides a curriculum in which individual tasks are easier to learn.
  • Computational costs: Distillation-based continual learning requires additional computation for performance gains, but arbitrary continual learning algorithms cannot convert extra computation into equivalent improvements.The paper quantifies computational cost using forward and backward passes and reports controlled-cost experiments separately.
  • Tweet stream: On chronologically ordered tweets, lifelong pretraining and continual learning improve downstream performance on later data and temporal generalization over outdated training data.Sequential Pretraining already improves over models trained only on 2020 data, with further gains from continual learning algorithms.

5 Related Works

Prior work examines domain and temporal adaptation of language models, while continual-learning research in NLP has largely focused on classification. This paper addresses the gap between sequential pretraining and continual-learning algorithms for emerging language-model corpora.

  • Domain and Temporal Adaptation of Language Models: Prior studies examine adapting pretrained language models to domain-specific and emerging data, but sequential pretraining work did not investigate continual-learning algorithms.
  • Domain and Temporal Adaptation of Language Models: Recent studies demonstrate the necessity of adapting language models over time as data distributions evolve.
  • Continual Learning Algorithms in NLP: Continual learning in NLP has mainly focused on classification, using stored or pseudo examples and, more recently, knowledge distillation with generated pseudo examples.

6 Conclusion

The paper formulates lifelong language model pretraining and evaluates continual adaptation across data streams, finding distillation-based approaches most effective while highlighting task-dependent KD performance and an unresolved limitation.

  • 6 Conclusion: The work formulates lifelong language model pretraining and constructs two data streams associated with downstream datasets.
  • 6 Conclusion: Distillation-based approaches were most effective for knowledge retention, adaptation to the latest data, and temporal generalization across the evaluation setups.The study constructed two data streams associated with downstream datasets to evaluate these capabilities.
  • 6 Conclusion: Whether any distillation-based continual-learning variant consistently outperforms Logit-KD remains unresolved, with KD performance appearing highly task-dependent.

A Detailed Experiment Settings · B Low-Resource Fine-Tuning

The experiments use stream-specific continual-pretraining schedules and validation-based hyperparameter tuning, then evaluate final checkpoints under low-resource fine-tuning. Sequential Pretraining helps most in low-resource ChemProt and SciERC settings, with Logit-KD providing further gains when it beats RoBERTa-base.

  • A Detailed Experiment Settings: Each corpus holds out 128,000 sentences for masked-language-modeling evaluation, and each training example is visited once because the pretraining corpora are large.Held-out validation sets provide the basis for evaluating MLM performance.
  • A Detailed Experiment Settings: Masked-language-modeling perplexity on held-out validation sets is used to tune hyperparameters for both task-specific and continual-learning models.Common hyperparameters are tuned with first-task models, whereas continual-learning-specific settings use the first two domains.
  • A Detailed Experiment Settings: The final-time-step pretrained models are fine-tuned and evaluated on Twitter Hashtag prediction and Emoji prediction.Table 6 summarizes full downstream performance for these two prediction tasks.
  • B Low-Resource Fine-Tuning: Sequential Pretraining’s advantage over RoBERTa-base is larger for low-resource ChemProt and SciERC fine-tuning, while Logit-KD further improves cases where Sequential Pretraining wins.The comparison uses final checkpoint t = 4 models trained with different amounts of downstream data.
  • B Low-Resource Fine-Tuning: Downstream performance is measured from final model checkpoints across varying amounts of fine-tuning data to assess low-resource behavior.Figure 6 compares Sequential Pretraining, RoBERTa-base, and Logit-KD under these fine-tuning conditions.

C Full Results over the Tweet Stream

Tables 6 and 7 report the complete Tweet Stream results, extending the main-text evaluation with additional earlier-year downstream tasks and 2014-to-2020 temporal generalization.

  • C Full Results over the Tweet Stream: Tables 6 and 7 summarize full Tweet Stream results, adding downstream performance on 2014 and 2016 data (D1, D2) and temporal generalization from 2014 to 2020 (D1 →D4).These results extend the main-text Table 5 evaluation.

D Dataset Details … E.2 SEED Distillation

The study evaluates continual pretraining on large research-paper and tweet streams using domain-specific downstream tasks and temporal prediction datasets. Its continual-learning methods include contrastive distillation and SEED distillation, which regularizes representations using queued examples beyond the current batch.

  • D Dataset Details: Research-paper pretraining uses four S2ORC streams containing 6.6M, 12.1M, 7.8M, and 7.5M full-text papers.
  • D Dataset Details: Downstream evaluation covers relation extraction, sentence-role labeling, citation intent, scientific relation extraction, synthesis procedures, named entities, keyphrases, and hyponyms across four research domains.
  • D Dataset Details: Tweet-stream pretraining uses 25M tweets per year, while held-out yearly tweets support multi-label hashtag and single-label emoji prediction.
  • D Dataset Details: Hashtag prediction uses the 200 most frequent yearly hashtags with 10k examples per split, while balanced emoji datasets contain 20 classes and matching split sizes.
  • E.1 Contrastive Distillation: Contrastive distillation adds an unsupervised SimCSE objective to continual language-model pretraining so sentence representations better reflect semantic similarity.
  • E.1 Contrastive Distillation: Contrastive distillation uses teacher and student temperatures of 0.05 and 0.01, respectively, and computes cross-entropy between their intra-batch similarity distributions.
  • E.2 SEED Distillation: SEED distillation maintains a fixed-size queue of current-domain examples and distills similarities between batches and queued examples using teacher and student models.
  • E.2 SEED Distillation: Because the queued target set can greatly exceed batch size, SEED distillation regularizes richer representational similarity information.

F Analysis and Controlled Experiments of Computational Costs · G Experiments with RoBERTa-large

The analysis finds that extra computation does not yield a simple performance tradeoff: more training steps can increase forgetting, while more replay can overfit memory. With RoBERTa-large, SEED-Logit-KD or Logit-KD achieves the best final performance across the evaluated tasks.

  • F Analysis and Controlled Experiments of Computational Costs: Sparse Logit-KD reduces distillation frequency to every r′=10 steps while replaying and distilling one memory batch, yielding C=2.4b when r and r′ are 10.Its costs are Cf=(1+2/k+1/k′)b and Cb=(1+1/k)b.
  • F Analysis and Controlled Experiments of Computational Costs: Increasing Sequential PT training steps by 1.2 times did not improve D4 performance and slightly reduced performance on Chemprot, ACL-ARC, and SciERC.The reduction may reflect increased forgetting in earlier domains.
  • F Analysis and Controlled Experiments of Computational Costs: Increasing ER replay frequency from k=10 to k=5 greatly increased the perplexity of M1, while extra replay can cause overfitting to replay memory.The controlled experiments compare computation-matched variants rather than assuming more replay is uniformly beneficial.
  • F Analysis and Controlled Experiments of Computational Costs: The results show that additional computation may be necessary for Logit-KD and SEED-Logit-KD, but performance cannot be simply traded against computation across arbitrary algorithms.More effective continual learning methods can improve performance despite higher computational costs.
  • G Experiments with RoBERTa-large: With RoBERTa-large, SEED-Logit-KD or Logit-KD achieves the best final pretrained-checkpoint performance on Chemprot, RCT-Sample, ACL-ARC, and SciERC.Some continual learning methods nevertheless have lower F1 at the initial time step.
  • G Experiments with RoBERTa-large: The RoBERTa-large experiments evaluate downstream models fine-tuned from lifelong-pretrained checkpoints at multiple time steps across D1 and D2 tasks.Figure 7 uses t∈{1,2,3,4} for Chemprot and RCT-Sample, and t∈{2,3,4} for ACL-ARC and SciERC.

H Experiments with BERT on Tweet Stream After 2019

On post-2019 tweet streams, continual pretraining improves BERT-base downstream performance, with Logit-KD generally outperforming Sequential PT despite smaller gains across nearby time periods. Logit-KD also improves temporal generalization in most cross-year hashtag prediction settings.

  • H Experiments with BERT on Tweet Stream After 2019: The experiment continually pretrains BERT-base, originally trained on pre-2019 Wikipedia, on four tweet corpora spanning the first and second halves of 2019 and 2020.The corpora are paired with hashtag prediction and cross-year hashtag prediction datasets.
  • H Experiments with BERT on Tweet Stream After 2019: Sequential PT clearly outperforms non-continually-pretrained BERT-base, while Logit-KD generally further improves hashtag prediction except on the first half of 2019.These downstream results are reported for hashtag prediction after fine-tuning the final pretrained model in Table 10.
  • H Experiments with BERT on Tweet Stream After 2019: The authors attribute the less significant improvements relative to the main setup to the small temporal gaps between the four tweet corpora.The corpora cover consecutive half-year periods from early 2019 through late 2020.
  • H Experiments with BERT on Tweet Stream After 2019: Logit-KD improves over Sequential PT in two out of three cross-year hashtag prediction setups, indicating stronger temporal generalization.The cross-year hashtag prediction results are presented in Table 11.

I Analysis of Data Streams · J Ethic Risks

The analysis links differing vocabulary-distribution gaps to transfer and latest-domain inconsistencies, while distillation-based continual learning consistently reduces forgetting. Ethical risks include persistent bias in continually updated models and restricted research-only use of released tweet data.

  • I Analysis of Data Streams: Vocabulary gaps are about 1e−5 between tweet domains versus 1e−2 between research-paper domains.Figure 8 summarizes cosine distances between vocabulary distributions across domains.
  • I Analysis of Data Streams: The analysis reports controlled-cost distillation results, post-2019 hashtag prediction, and temporal-generalization evaluations in Tables 9–11.The supplied table captions identify these three evaluation scopes without providing cell values.
  • I Analysis of Data Streams: Biomedical and Material Science domains have more similar vocabulary distributions, explaining improved D1 performance after pretraining on D3.This relationship is reported alongside downstream results in Fig. 4(a,b).
  • I Analysis of Data Streams: Distillation-based continual-learning algorithms reduce forgetting consistently across streams despite substantial differences in vocabulary-distribution changes.The analysis contrasts this consistent finding with stream-dependent effects on latest-domain performance.
  • J Ethic Risks: Continually pretrained models require identifying and removing biased pretraining content because it can affect downstream predictions and amplify earlier bias.The paper warns that continual updates may give earlier bias a profound negative impact.
  • J Ethic Risks: Future work should develop methods for forgetting biased knowledge from language models.The paper frames this as a preferable direction for addressing bias in continually updated models.
  • J Ethic Risks: Data released in the paper, especially the tweet stream, should be used only for research purposes.
Loading 2110.08534v3…