PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning of Large Language Models: A Comprehensive Survey

Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, Hao Wang

arXiv:2404.16789v3cs.LGcs.AIcs.CL

TL;DR

Continual LLM research addresses performance degradation associated with models trained on static, pre-collected datasets. This survey synthesizes continual-LLM training research, reports zero forgetting across diverse settings, and identifies unresolved needs for algorithmic diversity and understanding large-model behavior.

  • Problem

    LLMs are typically trained on static, pre-collected datasets, motivating study of performance degradation as continual learning settings evolve.

  • Method

    The paper provides a comprehensive survey of continual LLMs and summarizes recent advances in their training.

  • Results

    Zero forgetting is reported across diverse tasks, model architectures, and learning paradigms.

  • Takeaways & Limitations

    The survey identifies missing cornerstones, including algorithmic diversity and fundamental understanding of large models’ behaviors.

  • Takeaways & Limitations

    The field still lacks algorithmic diversity and a fundamental understanding of large models’ behaviors such as knowledge forgetting, transfer, and generalization.

Abstract

from arXiv · show

The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains -- a phenomenon known as "catastrophic forgetting". While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs. In this survey, we provide a comprehensive overview of the current research progress on LLMs within the context of CL. This survey is structured into four main sections: we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity (or horizontal continual learning), i.e., continual adaptation across time and domains (Section 3). We then summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (Section 5). Finally, we discuss intriguing questions pertaining to continual learning for LLMs (Section 6). The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-survey.

1 INTRODUCTION

The survey frames continual learning for LLMs as adaptation to evolving domains, tasks, and preferences while retaining prior capabilities. It organizes this landscape around vertical and horizontal continuity, three learning stages, evaluation resources, and open research questions.

  • Motivation: Static pre-training leaves LLMs prone to performance degradation over time and across content domains, while specialization requires additional fine-tuning.Re-training on newly collected pre-training data is described as prohibitively expensive and impractical in real-world scenarios.
  • Motivation: Continual learning trains models sequentially on tasks while aiming to preserve performance across tasks despite limited or absent access to previous data.The absence of optimization constraints from unseen previous data makes retaining past knowledge difficult and produces catastrophic forgetting.
  • Survey contribution: The survey reviews continual LLM research, emphasizes underexplored work in CPT and DAP, and discusses emergent properties, changing roles of conventional CL types, memory constraints, and prospective directions.Its conclusion identifies algorithmic diversity and fundamental understanding of forgetting and transfer as missing cornerstones.
  • Directions of continuity: Vertical continuity adapts LLMs from broad general capabilities to narrower specific domains while retaining general reasoning and question-answering abilities.The survey illustrates this direction with healthcare institutions developing medical-domain LLMs for users.
  • Directions of continuity: Horizontal continuity adapts LLMs across time and domains, often through multiple training stages that increase vulnerability to forgetting.Social-media platforms are presented as updating models for recent trends while preserving existing-user experience in advertising and recommendations.
  • Directions of continuity: The survey separates vertical and horizontal continuity as a framework for analyzing complex language-model continual-learning paradigms, including approaches spanning both directions.It notes that future designs might alternate between horizontal and vertical continual learning.
  • Survey organization: Within modern continual learning, the survey distinguishes Continual Pre-Training, Domain-Adaptive Pre-training, and Continual Fine-Tuning, alongside evaluation protocols, datasets, and emerging research questions.CFT coverage includes continual instruction tuning, continual model refinement, continual model alignment, and continual multimodal LLMs; the survey also highlights limited technique diversity in DAP.

2 BACKGROUND AND RELATED WORK

This section introduces LLMs, their adaptation and alignment objectives, and continual learning’s central challenge of retaining prior capabilities while learning new tasks. It also reviews continual-learning scenarios, technique families, and evaluation measures relevant to continual LLMs.

  • Large Language Models: LLMs use transformer-based pre-training on large unlabeled corpora to develop language understanding, generation, and emergent capabilities.Scaling parameters and training data supports abilities including in-context learning, instruction following, and multi-step reasoning.
  • LLM Adaptation: Instruction Tuning adapts LLMs to follow textual instructions and bridge general knowledge with task-specific performance.It can also improve interaction and align outputs with human expectations and preferences.
  • LLM Adaptation: Model Refinement addresses model errors while preserving performance on other inputs, whereas Model Alignment targets human values, ethics, and preferences.Alignment approaches include reinforcement-learning methods guided by feedback and supervised methods trained on preference datasets.
  • Continual Learning: Catastrophic forgetting is performance degradation on old tasks when models learn new ones, especially when little past data can be preserved.Continual learning studies methods for adapting models across task sequences without forgetting prior knowledge.
  • Continual Learning: Continual learning scenarios comprise Task-Incremental, Domain-Incremental, and Class-Incremental Learning, distinguished by task indices, task formulation, and new classes.Existing techniques are grouped into replay-, regularization-, architecture-, optimization-, and representation-based methods.
  • Continual Learning Techniques: Replay methods retain a small observed-data buffer and retrain on it, while regularization methods penalize parameter deviation from the previous model.Replay is valued for simplicity, stability, and high performance with small episodic memory; regularization weights deviations using parameter importance.
  • Continual Learning Techniques: Architecture expansion can achieve zero forgetting when task IDs are available or correctly inferred, but is mainly used in Task-Incremental Learning.Difficulty inferring task IDs limits its use in Domain-Incremental and Class-Incremental Learning.

3 CONTINUAL LEARNING MEETS LARGE LANGUAGE MODELS: AN OVERVIEW

The survey frames continual learning for LLMs along vertical and horizontal continuity, covering adaptation through the production pipeline and updates across time and domains. These settings differ in data relationships, forgetting risks, resource constraints, and the availability of historical data.

  • Overview: The survey’s framework distinguishes vertical and horizontal continuity as two directions for analyzing continual LLM adaptation.It presents a production pipeline spanning continual pre-training, domain-adaptive pre-training, continual fine-tuning, and deployment.
  • Vertical Continuity: Vertical continuity adapts an LLM from general pre-training through domain adaptation and downstream fine-tuning, with emphasis on retaining general knowledge.The pipeline may include additional stages or layers for entities with distinct objectives within a domain.
  • Vertical Continuity: Vertical continuity faces task heterogeneity because upstream and downstream tasks can differ in formulation, model structure, and training scheme.Common responses include freezing shared parameters or reformulating downstream tasks to match pre-training tasks.
  • Vertical Continuity: Vertical continuity also faces inaccessible upstream data when confidentiality and collection protocols prevent downstream entities from accessing prior datasets.Methods address this by using public datasets or generating pseudo-examples as proxy pre-training data.
  • Horizontal Continuity: Horizontal continuity updates deployed models as new data arrives across time and domains, requiring adaptation to dynamic distributions without prohibitive retraining.LLMs struggle to generalize to future unseen data and retain complete past knowledge under temporal or domain shifts.
  • Horizontal Continuity: With full historical-data access, simple sparse replay can effectively mitigate forgetting, while other continual-learning techniques can outperform naive solutions.These findings support the relevance of continual-learning methods to continual LLM training, although storage constraints may be meaningless for large institutions retaining all historical data.
  • Horizontal Continuity: Horizontal continuity can produce substantial forgetting when task distributions change, despite LLMs showing some robustness against catastrophic forgetting.The setting involves potentially abrupt distributional shifts and longer training sequences than vertical continuity.

4 LEARNING STAGES OF CONTINUAL LARGE LANGUAGE MODELS

The survey reviews continual learning across continual pre-training, domain-adaptive pre-training, and continual fine-tuning, highlighting distinct forgetting and adaptation patterns. Across these stages, evidence supports benefits from continual adaptation but also exposes gaps in technique diversity, multi-stage settings, and production realism.

  • Continual Pre-Training (CPT): CPT can improve downstream performance, particularly when distributional shifts are gradual or correlated, while supporting generalization to new domains.The survey also reports forward transfer and adaptation benefits across several CPT settings.
  • Continual Pre-Training (CPT): CPT research remains uneven: about half of examined papers propose novel techniques, while others study adaptation effects or apply existing methods.Among proposed CPT techniques, architecture expansion is most common; replay and parameter regularization are used less often.
  • Continual Pre-Training (CPT): Temporal and language shifts can produce substantial forgetting, although replay, parameter expansion, differential updates, and calibrated replay rates can improve adaptation or retention.Findings vary by setting: replay may be ineffective under some shifts, whereas parameter expansion shows robust performance across reported conditions.
  • Domain-Adaptive Pre-training (DAP): DAP generally improves downstream performance across domains, and even small amounts of domain-specific data can yield notable gains.Reported evaluations span domains including biomedical, computer science, news, and reviews.
  • Domain-Adaptive Pre-training (DAP): DAP is usually single-stage, despite widespread use of continual-learning techniques, and only one listed study uses two DAP stages.The survey identifies a gap between current DAP studies and multi-stage continual adaptation in production environments.
  • Continual Fine-Tuning (CFT): CFT explicitly incorporates continual-learning techniques more consistently than CPT or DAP, with all 35 reviewed papers deploying them and half developing new techniques.The surveyed CFT literature also shifts emphasis from class-incremental learning toward task- and domain-incremental learning.

5 EVALUATION PROTOCOLS AND DATASETS

The survey reviews evaluation protocols that measure knowledge probing, forgetting, learning efficiency, and transfer for continually adapted LLMs. It also catalogs benchmark datasets by distributional shift, semantic domain, source, and application while excluding datasets outside its continual-adaptation scope.

  • Evaluation Protocols: LAMA probes world knowledge by converting facts into cloze statements that LLMs must complete.It has been used extensively for CPT under temporal shifts.
  • Evaluation Protocols: FUAR measures the balance between knowledge forgetting and knowledge learning during CPT; FUAR below 1 indicates higher learning efficacy.A FUAR value of 1 represents equal forgetting and learning.
  • Evaluation Protocols: TRACE introduces X-Delta metrics to quantify forward transfer on specific LLM abilities during continual instruction tuning.Its evaluation tasks cover general ability, instruction following, and related ability dimensions.
  • Datasets: The dataset review organizes benchmarks by dataset type, distributional shifts, semantic domains, sources, and applications.The survey focuses on datasets designed for continual learning or continual domain-adaptive pre-training.
  • Datasets: The survey excludes ordinary vertical DAP datasets unless they target continual DAP and omits general continual-fine-tuning datasets already extensively studied elsewhere.These exclusions define the scope of the dataset inventory.

6 DISCUSSION

The discussion highlights distinctive continual-learning behaviors and priorities for LLMs, including emergent memory phenomena, shifting efficiency constraints, theoretical gaps, and controllable memory. It also distinguishes how continual-learning scenarios and techniques map onto LLM adaptation.

  • 6.1 Intriguing Properties Emergent in Continual LLMs: Sequentially fine-tuned large models can recover forgotten document information before encountering those documents again, a phenomenon termed anticipatory recovering.This behavior suggests sequential memorization and motivates research into more complex structured learning environments as model size increases.
  • 6.2 Continual Learning Settings for LLMs: Continual LLM studies emphasize task-incremental and domain-incremental learning more than class-incremental learning.Generative next-word prediction makes it natural to integrate class-incremental settings into broader language-modeling frameworks without an incremental classification head.
  • 6.2 Continual Learning Settings for LLMs: Language models can infer domain information from unseen instructions, blurring the boundary between task-incremental and domain-incremental learning in continual instruction tuning.The shared task definition and input-output format also make domain-incremental learning relevant to continual pre-training and domain-adaptive pre-training.
  • 6.3 Roles of Memory in Continual LLMs: When memory storage is affordable, the central challenge shifts from storage efficiency to computational efficiency.The discussion prioritizes efficient adaptation, selective replay, and restricting updates and FLOPs, while tightened-memory settings remain important.
  • 6.4 Prospective Directions: Existing continual-learning theories may not transfer directly to large pre-trained LLMs, leaving a notable gap in robust theoretical understanding.The discussion calls for guarantees and explanations of forgetting behavior, while noting that current validation is limited in scale.
  • 6.3 Roles of Memory in Continual LLMs: Replay without specific design can cause inefficient updates and slow convergence, motivating selective replay methods such as KPIG and forgetting forecasting.KPIG dynamically selects replay data using Key-Part Information Gain, while forgetting forecasts support selective replay in continual model refinement.
  • 6.3 Roles of Memory in Continual LLMs: External controllable memory could make LLM knowledge more interpretable and manipulable, including for machine unlearning and one-shot knowledge updates.The Kanerva Machine supports writing, reading, generating, sequential writing, and forgetting, enabling updates without costly retraining or fine-tuning.

7 CONCLUSION

The survey organizes continual LLM research within a broader modern framework and reviews recent training and deployment advances. It identifies missing algorithmic diversity and limited understanding of forgetting, transfer, and acquisition as priorities for future work.

  • 7 CONCLUSION: The survey reviews recent continual LLM advances in training and deployment and categorizes problems and tasks within a broader stratified framework.Its goal is to provide a comprehensive account from a continual-learning perspective.
  • 7 CONCLUSION: The field still lacks algorithmic diversity and fundamental understanding of large models’ knowledge forgetting, transfer, and acquisition.These missing cornerstones are identified despite widespread and growing community interest.
  • 7 CONCLUSION: The survey aims to encourage practitioners to explore continual-learning techniques for developing robust and self-evolving AI systems.This stated aim follows its holistic and detailed review of the area.

SUPPLEMENTARY MATERIAL

The supplementary material introduces notation, foundational LLM concepts, pre-training and adaptation, mainstream model families, and basic continual-learning techniques. It also summarizes language-model objectives and emergent capabilities associated with scale.

  • Foundations: The supplementary material establishes notation and introduces LLM pre-training, downstream adaptation, mainstream LLM families, and basic continual-learning techniques.It defines conventions for scalars, vectors, matrices, norms, loss, probability, expectation, and integer sets.
  • Notation: The material also introduces empirical loss notation and probability, expectation, and integer-set conventions used throughout the paper.These definitions support the formal presentation of later objectives and continual-learning settings.
  • Language-model foundations: Scaling language-model parameters and training data improves language understanding and generation while producing emergent abilities such as in-context learning and multi-step reasoning.The supplementary discussion describes these abilities as absent from small-scale language models.
  • Language-model foundations: Pre-training gives language models broad language representations, after which fine-tuning on small task-specific datasets supports diverse NLP tasks.Transformer-based pre-trained models establish a universal hidden embedding space from large unlabeled text corpora.

A.1.1 Pre-Training of LLMs.

The appendix formalizes LLM pre-training, adaptation, and continual-learning settings. It covers autoregressive and masked objectives, instruction-oriented adaptation, catastrophic forgetting, memory constraints, and task-, domain-, and class-incremental learning.

  • Pre-Training of LLMs: Decoder-only LLMs commonly use autoregressive language modeling, predicting each next token from all preceding tokens and minimizing negative log-likelihood.GPT, PaLM, and LLaMA are identified as major decoder-only model families.
  • Pre-Training of LLMs: Encoder-only models such as BERT commonly use masked language modeling, predicting masked tokens from the unmasked input sequence.T5 is described as using a sequence-to-sequence masked language-modeling objective.
  • Adaptation of LLMs: Instruction tuning adapts a language model’s parameters using curated instruction-output examples to improve responses to specific instructions.Model refinement instead updates incorrect outputs on an editing set while preserving outputs outside that set.
  • Continual Learning: Continual learning adapts models to sequential tasks while seeking to preserve previous performance, because later tasks can cause catastrophic forgetting.The appendix frames efficient adaptation without forgetting as a central continual-learning objective.
  • Continual Learning: Continual-learning formulations are defined under memory constraints over sequences of task distributions and observations.The framework distinguishes joint task distributions from shared input-output distributions across stages.
  • Continual Learning: Task-incremental learning supplies task identity, domain-incremental learning keeps input and output spaces shared without task IDs, and class-incremental learning requires inferring labels and task identity.The appendix presents class-incremental learning as the most challenging of the three scenarios.

A.2.2 Techniques of Continual Learning.

The survey groups continual-learning techniques into replay-, regularization-, and architecture-based methods, emphasizing trade-offs between retaining prior knowledge, learning current data, and respecting memory constraints.

  • Continual-learning techniques are categorized as replay-, regularization-, architecture-, optimization-, and representation-based methods.
  • Replay-Based Methods: Replay methods retain a small episodic buffer of past examples, offering simplicity, stability, and strong performance despite theoretical generalization concerns.DER++ also replays logits, while other work improves buffer efficiency through exemplar selection and low-fidelity storage.
  • Regularization-Based Methods: Regularization methods approximate past-domain losses with parameter-space penalties weighted by parameter importance and a retention–adaptation coefficient.EWC uses diagonal Fisher Information Matrix values, whereas MAS estimates importance online and without supervision.
  • Regularization-Based Methods: An identity importance matrix reduces regularization to an l2 penalty that equally penalizes parameters and can remain effective for continual LLMs.
  • Architecture-Based Methods: Architecture expansion can achieve zero-forgetting when task IDs are available or inferable, but task-ID difficulty limits its use mainly to task-incremental learning.Progressive networks add laterally connected neurons, while CoLoR trains separate LoRA modules and infers task IDs through prototypes.

B EVALUATION PROTOCOLS AND DATASETS

The survey’s evaluation-and-dataset overview points readers to appendices covering continual-learning metrics, continual-LLM-specific metrics, and datasets for the discussed topics.

  • Appendix B.1 reviews common continual-learning evaluation metrics and provides their formal definitions.
  • Appendix B.2 introduces metrics designed specifically for continual LLMs.
  • Appendix B.3 outlines datasets available for each topic discussed in the survey.

B.1 Evaluation Metrics of Continual Learning

Continual-learning evaluation uses a performance matrix to derive overall performance, forgetting, backward transfer, and forward transfer, with metrics adapted to LLM evaluation settings.

  • Backward Transfer: Backward Transfer captures performance improvements on prior tasks after learning new tasks and is commonly represented by negating forgetting.These concepts support metrics including knowledge retention, unchanged-knowledge performance, average increased perplexity, and test-and-edit retention.
  • The Performance Matrix P records model performance after training on task i and evaluating on task j, using metrics such as perplexity, zero-shot scores, accuracies, and probing results.
  • Overall Performance: Overall Performance averages model performance across tasks and is commonly reported after all T tasks are completed.Some variants weight tasks by data importance, with names including example accuracy, whole accuracy, and edit success rate.
  • Forgetting: Forgetting measures the largest averaged performance drop on previously learned tasks during continual training and is typically reported at the final stage.
  • Forward Transfer: Forward Transfer measures generalization to later tasks relative to baseline performance, including downstream improvements from domain-adaptive pre-training.It is also called temporal generalization or knowledge transfer.

B.2 Continual LLMs’ Evaluation Protocols

Continual-LLM evaluation combines general language-model, knowledge, transfer, alignment, and retention measures to assess changing capabilities and trade-offs.

  • Knowledge Evaluation: LAMA probes world knowledge by converting facts into cloze statements, and continual-pre-training variants distinguish invariant, updated, and newly acquired knowledge.
  • Knowledge Retention: FUAR measures the balance between forgotten knowledge and updated or acquired knowledge when overall performance is too coarse.A FUAR value of 1 denotes an equal trade-off, while values below 1 indicate more knowledge acquired per unit of forgotten time-invariant knowledge.
  • Instruction Tuning: TRACE evaluates continual instruction tuning across general ability, instruction following, and safety using X-Delta metrics for forward transfer.Its evaluation tasks differ from the tasks used for continuous fine-tuning.
  • Model Alignment: Continual model alignment commonly evaluates natural-language generation with BLEU-4, METEOR, and ROUGE-L.The metrics respectively emphasize n-gram precision, synonym- and stemming-aware correspondence, and longest-common-subsequence recall.
  • Model Alignment: The surveyed protocols reflect different dimensions of text-generation quality and are tailored to specific language-processing tasks.

B.3 Datasets

The survey reviews datasets for benchmarking continual learning in LLMs across pre-training, instruction tuning, model refinement, alignment, and multimodal settings. It also identifies gaps in evaluation, including the lack of a widely recognized temporal-shift benchmark for continual pre-training.

  • Datasets for Continual Pre-Training and Domain Adaptive Pre-Training: Continual pre-training under temporal shifts lacks a widely recognized benchmark; TimeLMs instead uses quarterly Twitter corpora collected until 2022.
  • Datasets for Continual Pre-Training and Domain Adaptive Pre-Training: CPT and DAP research often constructs test beds from public domain datasets, including ∗DAPT, DEMix, ∗CPT, ∗DAS, TemporalWiki, and CC-RecentNews.∗DAPT spans BioMed, Computer Science, News, and Reviews, while DEMix contains eight semantic domains and over 73.8 billion tokens.
  • Datasets for Continual Instruction Tuning: Continual instruction tuning requires benchmarks that assess both retention of old knowledge and integration of new instructions.Traditional metrics may be too simplistic to evaluate continual learning in LLMs comprehensively.
  • Datasets for Continual Instruction Tuning: Replay generally yields the best performance across all methods in the CITB benchmark.
  • Datasets for Continual Model Refinement: Continual model-refinement datasets primarily cover fact checking and question answering, with FEVER, VitaminC, zsRE, NQ, T-rex, CounterFact, and SCOTUS among the reported resources.SCOTUS evaluates document classification for U.S. Supreme Court cases into 11 topics.
  • Datasets for Continual Model Alignment: Continual model-alignment studies use datasets including IMDB, HH-RLHF, Reddit TL;DR, Common Sense QA, Reading Comprehension, and Translation.These datasets support evaluation of adaptation under continuous learning conditions and alignment with human expectations.
Loading 2404.16789v3…