Source-linked AI summary
Continual Learning of Natural Language Processing Tasks: A Survey
Zixuan Ke, Bing Liu
TL;DR
Continual learning supports incremental model updates as tasks and data change, without repeatedly retraining from scratch. This survey reviews NLP continual learning across settings, techniques, transfer, and inter-task class separation. It concludes that the coverage is not exhaustive, as related work may have been missed.
Problem
Incrementally updating models matters because retraining from scratch is usually expensive and time-consuming, while continual learning must address forgetting and transfer across tasks.
Source: §1 · p. 1Method
The paper surveys NLP continual learning across settings and techniques, including forgetting prevention, knowledge transfer, and inter-task class separation.
Source: §1 · pp. 1–2, §2.1 · p. 2, §4 · p. 4Results
The survey finds that many NLP problems are formulated as domain-incremental learning, with instructions enabling different task types to use a shared language-model head.
Source: §3.2 · p. 3Takeaways & Limitations
The survey presents continual learning as a way to accumulate knowledge over time and use it to support better future learning.
Source: §6 · p. 7The covered systems are not exhaustive, and rapidly growing research means related papers may have been missed.
Source: §8 · p. 7
Abstract
from arXiv · showhide
Continual learning (CL) is a learning paradigm that emulates the human capability of learning and accumulating knowledge continually without forgetting the previously learned knowledge and also transferring the learned knowledge to help learn new tasks better. This survey presents a comprehensive review and analysis of the recent progress of CL in NLP, which has significant differences from CL in computer vision and machine learning. It covers (1) all CL settings with a taxonomy of existing techniques; (2) catastrophic forgetting (CF) prevention, (3) knowledge transfer (KT), which is particularly important for NLP tasks; and (4) some theory and the hidden challenge of inter-task class separation (ICS). (1), (3) and (4) have not been included in the existing survey. Finally, a list of future directions is discussed.
1 Introduction
Continual learning incrementally updates models as tasks and data change, avoiding the cost of retraining from scratch. This survey broadens prior NLP-focused coverage by reviewing CL settings, techniques, knowledge transfer, and challenges including inter-task class separation.
1 Introduction
- CL incrementally learns tasks as their training data may become unavailable, avoiding the expense and time of retraining a model from scratch.The motivation includes adapting language models to changing environments and systems to new skills or products.Source: §1 · p. 1
- A CL system should limit catastrophic forgetting while supporting forward and backward knowledge transfer across tasks.Forgetting can occur when learning a new task modifies parameters learned for previous tasks; transfer can help new tasks or improve earlier ones.Source: §1 · p. 1
- The survey covers all CL settings and a wide range of NLP approaches, extending an earlier NLP survey that omitted knowledge transfer and settings.It also reviews NLP-specific approaches and the recently raised inter-task class separation challenge.Source: §1 · p. 1
- The survey extends existing CL settings to make them more general and up-to-date.Source: §1 · p. 2
2 Settings and Learning Modes of CL
Continual learning settings differ in task labels and test-time task-ID availability, while learning modes describe how training data arrive. The survey distinguishes these settings because their challenges and approaches differ.
2.1 CL Settings
- Task-incremental learning provides task-IDs in training and testing, typically using an exclusive output head per task; its main challenge is bidirectional knowledge transfer.Source: §2.1 · p. 2
- Domain-incremental learning shares class labels across tasks, and systems may work without test-time task-IDs when tasks are similar, distinguishable, or supported by replay.The survey also describes DIL as a special case of TIL, while noting that domains can conflict and task-IDs are generally needed.Source: §2.1 · p. 2
- Class-incremental learning uses one model across disjoint task classes without a task-ID at test time, creating an inter-task class separation challenge.Because previous-task data are unavailable during new-task learning, the model cannot establish boundaries between current and previous classes; CIL accuracy is often lower than TIL or DIL.Source: §2.1 · p. 2
2.2 Learning Modes
- Offline learning permits multiple training epochs per task, whereas online learning trains each accumulated stream batch in one iteration and one epoch.Source: §2.2 · p. 2
- Online learning distinguishes hard task boundaries, where task data arrive consecutively, from blurry boundaries, where data from different tasks may intertwine.Source: §2.2 · p. 2
3 NLP Tasks in CL
The survey organizes NLP continual-learning problems by task type and summarizes how NLP problems are formulated in existing work. It observes that many NLP problems are treated as domain-incremental learning, often through a shared language-model generation head.
3.1 Types of Tasks
- The survey groups NLP continual-learning tasks into end-task, few-shot end-task, and domain-adaptive pre-training types.Source: §3.1 · p. 3
- End-tasks include problems such as sentiment classification, named entity recognition, and summarization.Source: §3.1 · p. 3
- Few-shot end-task learning must address both continual learning and possible overfitting from learning a few-shot task.Source: §3.1 · p. 3
- Domain-adaptive pre-training uses a domain corpus to adapt or fine-tune a language model, either from scratch or from a pretrained checkpoint.Source: §3.1 · p. 3
3.2 CL Formulation of NLP Problems
- The survey’s summary of existing NLP problem formulations is explicitly non-exhaustive.Source: §3.2 · p. 3
- A large number of NLP problems belong to domain-incremental learning, unlike the few CV problems formulated that way.The survey attributes this pattern mainly to language models’ text-generation nature and the use of natural-language instructions to convert tasks into generation with a shared LM head.Source: §3.2 · p. 3
4 Approaches for Continual Learning
The survey organizes continual-learning approaches around catastrophic-forgetting prevention, bidirectional knowledge transfer, and inter-task separation, noting that settings differ in their challenges and methods.
4.1 Approaches for CF Prevention
- CF prevention includes regularization, replay, parameter isolation, and NLP-specific instruction methods; regularization trades new-task learning against forgetting, while isolation can constrain capacity and transfer.Replay may store prior samples or generate pseudo-samples, which can be difficult for summarization and aspect sentiment classification.Source: §4.1 · p. 4
Instruction-based Methods
- Instruction-based methods condition language models on task-specific instructions, but conditioning may not prevent forgetting and has so far been used only in DIL and CIL.An instruction is described as a few words.Source: Instruction-based Methods · pp. 5–6
4.2 Approaches for Knowledge Transfer
- Knowledge-transfer approaches must support forward and backward transfer; the survey groups them into methods leveraging task similarity and methods leveraging replay data.Similarity-based methods use proxies because task similarity is not provided by task data.Source: §4.2 · p. 6
- Similarity-based KT uses importance, feature, or gradient proxies; gradient thresholds may be inaccurate, and this proxy has so far been used only in CV.Feature proxies include capsule routing and contrastive learning.Source: §4.2 · p. 6
- KT approaches include disentangling shared and task-specific knowledge, meta-learning across tasks, and conditioning on instructions related to previous tasks.Meta-learning designs use new-task examples in the support set and examples from all learned tasks in the query set.Source: §4.2 · p. 6
4.3 Dealing with Inter-task Class Separation
- For CIL, inter-task class separation arises because prior-task data is unavailable when learning current-task decision boundaries; within-task prediction and per-task OOD detection are necessary and sufficient for strong performance.A proposed approach combines a TIL method that overcomes forgetting with OOD detection and achieves state-of-the-art CIL performance.Source: §4.3 · p. 6
- Replay only partially addresses ICS because its data is limited, while post-prediction methods may work well when domains are especially dissimilar and easy to distinguish.Post-prediction methods are mainly used in DIL systems.Source: §4.3 · p. 6
5 Observations and Future Directions
The survey identifies unresolved challenges in knowledge transfer, class-incremental and domain-adaptive learning, scalability, and temporal continual learning, and outlines future directions based on these observations.
Knowledge transfer
- Knowledge transfer remains challenging in TIL because task sequences mix similar and dissimilar tasks, and incorrect similarity detection can cause serious forgetting, especially in backward transfer.Some TIL methods achieve no catastrophic forgetting, but the amount of transfer remains limited.Source: §5 · p. 7
CIL and DIL
- In CIL and DIL, catastrophic forgetting and inter-task class separation remain major issues; OOD detection has not matched the accuracy of training all tasks and classes together.The all-tasks/classes-trained model is described as an upper bound.Source: §5 · p. 7
Continual domain-adaptive pre-training
- Continual domain-adaptive pre-training is still in its infancy, and fine-tuning on one task can damage general knowledge retained in the language model for subsequent tasks.The survey describes initial attempts as very limited.Source: §5 · p. 7
Scalability
- Most continual-learning systems cover fewer than 100 tasks, and network capacity remains an unresolved concern despite preliminary distillation, pruning, and regularization approaches.It remains unclear how capacity affects performance or how to alleviate the issue effectively.Source: §5 · p. 7
Temporal continual learning
- Temporal continual learning has received limited attention; it involves one task that must be continually updated over time.The cited work so far is limited to some analysis and dataset papers.Source: §5 · p. 7
6 Conclusion
The conclusion frames continual learning as important to intelligent agents and presents a survey of recent NLP advances and future directions.
6 Conclusion
- The paper argues that an intelligent agent needs continual learning to accumulate knowledge over time and use it for better future learning.The conclusion presents this capability as a condition for an agent to be truly intelligent.Source: §6 · p. 7
- The paper surveys recent advances in continual learning for NLP.Source: §6 · p. 7
- The authors hope the survey will inspire NLP researchers to design better algorithms.Source: §6 · p. 7
7 Ethics Statement
The ethics statement describes the work as a survey of publicly available, peer-reviewed research and reports no perceived risks or disadvantage.
8 Limitation
The section notes that the survey is not exhaustive, then reviews NLP techniques and continual-learning approaches based on regularization and replay.
8 Limitation
- The survey's coverage is not exhaustive, and its search focused mainly on papers from major NLP, machine-learning, and computer-vision conferences.Because continual learning is growing rapidly, related papers may have been missed.Source: §8 · p. 7
A NLP Preliminaries
- Light-weight adapter, prefix, and prompt tuning add few parameters to a pretrained Transformer; task instructions can format tasks as generation and may also support knowledge transfer.Tuning only added parameters with the language model frozen can achieve performance similar to tuning the whole model; in continual learning, light-weight tuning reduces parameters saved by parameter-isolation methods.Source: A NLP Preliminaries · p. 13
B.1 Regularization-based Methods
- Regularization methods protect prior-task knowledge by penalizing changes to important parameters or distilling earlier model responses.EWC weights parameter changes by their importance; NLP variants include RMR-DSE, AEWC, and SRC, while ExtendNER, CFID, CID, and PAGeR use response KL-divergence.Source: §B.1 · p. 13
B.2 Replay-based Methods
- Replay methods reuse raw samples, prototypes, or generated samples to mitigate forgetting, with approaches varying in memory selection, weighting, and task setting.Examples include CID's handling of old/new data imbalance, Continual-T0's 1% memory buffer, IDBR's centroid-based selection, and MeLL's prototype replay.Source: §B.2 · p. 13
- Some replay systems use stored examples at inference or generate prior-task samples for joint training with new data.MBPA++ and Meta-MBPA++ retrieve nearest saved samples for sample-specific fine-tuning; LAMOL and ACM generate old-task samples, and ACM selectively shares adapters for transfer.Source: §B.2 · pp. 13–14
- Other replay designs combine generation with prompts or distillation, while some cited replay approaches are specified only by their high-level role.PCLL conditions generated prior-task samples on saved task prompts; LFPT5 combines sample generation with prompt distillation.Source: §B.2 · p. 14
B.3 Parameter-isolation Based Methods
Parameter-isolation methods limit forgetting by separating task parameters, expanding architectures, or selecting task-specific prompts, masks, or generated parameters. Instruction-based methods instead condition the language model on task-specific instructions to address forgetting.
B.3 Parameter-isolation Based Methods
- HAT masks layer outputs with task-specific gates and blocks gradients to neurons used by previous tasks to avoid catastrophic forgetting.The sigmoid gate acts as a pseudo-step function; inverse-mask multiplication blocks those gradients.Source: §B.3 · p. 14
- SupSup trains a binary mask network for each task to isolate subnetworks, but masks may match the original model's size and require efficiency tricks.The approach uses a straight-through trick.Source: §B.3 · p. 14
- Other parameter-isolation approaches assign adapters per task, use perplexity or weighted perplexity to select them, or replay data while expanding language-model layers.AdapterCL infers task identity through perplexity; DEMIX weights all pretrained adapters by perplexity; ELLE uses large replay data.Source: §B.3 · p. 14
- Dynamic architectures add a prompt or expand the network for each task, while ProgM transfers learned knowledge to each expanded component and uses the last component at test time.C-PT adds task-specific prompts; TPEM expands the network as tasks arrive.Source: §B.3 · p. 14
- Parameter pools select prompts by cosine similarity, while Hypernet generates task-solver parameters from task representations and regularizes changes to its generator.CLIF applies Hypernet to generate adapter parameters.Source: §B.3 · p. 14
B.4 Instruction-CF Based Methods
- Instruction-based methods condition the language model on task-specific instructions; ENTAILMENT also turns intent classification into binary classification to help alleviate forgetting.ConTinTin uses different instructions for different tasks.Source: §B.4 · p. 14
C.1 Similarity-based Methods
Similarity-based methods encourage knowledge transfer by identifying related tasks or shared features, while replay-based methods use past-task data or learning objectives that incorporate it. The section also describes instruction-based and meta-learning approaches to transfer and forgetting prevention.
C.1 Similarity-based Methods
- ACM learns task-specific adapters and mixing coefficients to identify reusable adapters for new tasks, and uses pseudo-replay to prevent forgetting.Source: §C.1 · p. 14
- Capsule routing groups similar tasks so their task capsules and shareable features can be updated together, enabling knowledge transfer.CTR uses a binary gate and B-CL uses similarity-based weights to control updates; fixed parameter sharing can cause forgetting for dissimilar tasks.Source: §C.1 · p. 14
- CLASSIC encourages knowledge transfer by generating an augmented view from previous tasks through self-attention and applying contrastive loss to shared knowledge.It separately prevents forgetting through parameter isolation.Source: §C.1 · p. 15
C.2 Replay-based Methods
- Replay-based methods can address inter-task class separation to some extent by letting the model see a limited number of past examples; IDBR also adds next-sentence prediction to encourage general task knowledge.Source: §C.2 · p. 15
- Meta-learning defines source tasks with support and query sets, then fine-tunes the meta-trained representation on target tasks; in continual learning, query sets include old-task data, making these methods replay-based.The meta-representation specifies how network parameters are learned.Source: §C.2 · p. 15
- MeLL separates shared and task-specific information by fine-tuning the language model on query data while updating class prototypes on support data; Meta-MBPA++ retrieves replay samples in its inner loop and generalizes across tasks in its outer loop.ConTinTin uses task-specific instructions to prevent forgetting and positive instructions or samples to enable transfer; C-PT fuses previous instructions to learn a general skill.Source: §C.2 · p. 15
D Details of Systems for Inter-task Class Separation
For inter-task class separation, the survey describes replay and post-prediction methods that infer which task or domain a test example belongs to. It also outlines evaluation measures and reference baselines for continual-learning systems.
D Details of Systems for Inter-task Class Separation
- Replay-based methods can help with inter-task class separation to some extent because they expose the model to a limited number of past examples.Source: D Details of Systems for Inter-task… · p. 15
- AdapterCL and DEMIX use test-sample perplexity to select or predict task identity, while RMR-DSE compares test features with domain-shifting vectors.AdapterCL selects the task model with lowest perplexity; RMR-DSE vectors represent differences between past and current model representations.Source: D Details of Systems for Inter-task… · p. 15
E Continual Learning Evaluation
- Continual-learning evaluation assesses average performance across learned tasks, forgetting, and knowledge transfer.Source: E Continual Learning Evaluation · p. 15
E.1 Reference Baselines
- Naive continual learning is a lower-bound reference with expected worst forgetting and base transfer, while separately trained task models provide a control for assessing forgetting or transfer.The separate-model baseline is not a continual-learning setting and has neither forgetting nor transfer.Source: §E.1 · pp. 15–16
- Multi-task learning is an upper-bound reference for task-incremental learning and requires multiple heads; Aggregate is an upper-bound reference for domain-incremental and class-incremental learning.Source: §E.1 · p. 16
E.2 Evaluation Metrics
The survey defines metrics for final average performance, backward and forward transfer, and forgetting, with a class-incremental accuracy summary. It notes that one forward-transfer measure does not capture transfer after learning the new task.
E.2 Evaluation Metrics
- Average Accuracy measures performance after all T tasks have been learned by averaging the model's performance across task test sets.Source: §E.2 · p. 16
- Average Incremental Accuracy summarizes average accuracy across the class sets in class-incremental learning.The average accuracy for each task can also be plotted instead of reporting one summary number.Source: §E.2 · p. 16
- Forgetting Rate averages the accuracy lost on all tasks except the last, comparing each task's accuracy just after learning with its accuracy after the final task.Higher rates indicate more forgetting; negative rates indicate positive knowledge transfer.Source: §E.2 · p. 16
- Forward Transfer compares performance on a task after learning it with that task's individually trained performance, while FWT′ compares performance before learning it with a randomly initialized network.Source: §E.2 · p. 16
- Backward Transfer measures how subsequent-task learning changes performance on learned tasks, with positive values indicating improvement on previous tasks.Source: §E.2 · p. 16