PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning for Large Language Models: A Survey

Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, Gholamreza Haffari

arXiv:2402.01364v2cs.CLcs.LG

TL;DR

LLMs require continual updates despite the cost of frequent retraining, motivating a survey of continual learning methods for these models. The paper organizes the literature by learning stages and updated information, and identifies benchmark gaps and future challenges.

  • Problem

    LLMs are costly to retrain frequently but need updates for new skills and evolving human knowledge, values, and language.

  • Method

    The survey categorizes LLM continual learning by continual pretraining, instruction tuning, alignment, and the types of information updated.

  • Results

    The survey presents a multi-stage framework and synthesizes research on updating facts, domains, languages, tasks, values, and preferences.

  • Takeaways & Limitations

    The survey provides a structured account of how continual learning is applied to LLMs and highlights challenges and future research directions.

  • Takeaways & Limitations

    Continual alignment lacks standardized benchmarks for systematically evaluating how models learn new preferences over time.

Abstract

from arXiv · show

Large language models (LLMs) are not amenable to frequent re-training, due to high training costs arising from their massive scale. However, updates are necessary to endow LLMs with new skills and keep them up-to-date with rapidly evolving human knowledge. This paper surveys recent works on continual learning for LLMs. Due to the unique nature of LLMs, we catalog continue learning techniques in a novel multi-staged categorization scheme, involving continual pretraining, instruction tuning, and alignment. We contrast continual learning for LLMs with simpler adaptation methods used in smaller models, as well as with other enhancement strategies like retrieval-augmented generation and model editing. Moreover, informed by a discussion of benchmarks and evaluation, we identify several challenges and future work directions for this crucial task.

1 Introduction

Continual learning for LLMs is needed to keep pace with evolving human knowledge, values, and language, but it requires a distinctive multi-stage framework. The survey organizes this framework and contrasts it with other model-improvement strategies.

  • LLMs need regular updates to reflect evolving human knowledge, values, and linguistic patterns.
  • Continual learning, retrieval-augmented generation, and model editing pursue different model-improvement purposes.The supplied passage contrasts continual learning with strategies focused primarily on domain-specific accuracy or factual knowledge.
  • The survey categorizes LLM continual learning into continual pretraining, instruction tuning, and alignment.These stages respectively expand language understanding, improve responses to user commands, and maintain adherence to values and societal norms.
  • This multi-stage process differs from the more linear adaptation strategies used for smaller models.
  • The survey is organized around continually updated information and distinct LLM learning stages rather than only continual-learning strategies.

2 Preliminary and Categorization

The paper frames LLM continual learning as sequential adaptation across pretraining, instruction tuning, and alignment, with knowledge transfer across stages. It provides stage- and information-based taxonomies for organizing the literature.

  • Preliminaries: Continual learning addresses sequential task streams in which a model must acquire, update, accumulate, and exploit knowledge over time.
  • Preliminaries: Catastrophic forgetting occurs when training on new data significantly diminishes performance on old tasks.
  • Framework: The framework aligns LLM continual learning with continual pretraining, continual instruction tuning, and continual alignment.These stages enrich knowledge and adapt to domains, improve instruction following, and continuously align outputs with human values.
  • Framework: Cross-stage transfer must preserve abilities and knowledge learned in previous stages, because iterative training can cause forgetting across stages.
  • Categorization: The stage-based categorization covers factual, domain, and language updates in continual pretraining; task, domain, and tool updates in instruction tuning; and value and preference updates in alignment.
  • Categorization: The survey also categorizes methods by the information updated during continual learning, including facts, domains, tasks, values, and preferences.

3 Continual Pre-training (CPT)

Continual pretraining keeps LLMs current, adaptable, and effective by updating facts, domains, coding capabilities, and language coverage. The survey organizes CPT around domain-incremental and domain-specific approaches.

  • 3 Continual Pre-training (CPT): Continual pretraining regularly updates LLMs with recent information, specialized domains, coding capabilities, and additional languages.These updates support relevance, evolving user needs, and diverse applications.
  • 3.1 CPT for Updating Facts: Dynamic datasets and continual knowledge learning support efficient factual updates while reducing forgetting and preserving language-model performance.Continual learning with different data can achieve comparable or better perplexity than training on an entire snapshot, using minimal training data.
  • 3.2 CPT for Updating Domains: Domain-incremental pretraining accumulates knowledge across domains, whereas domain-specific pretraining develops general models into domain experts using specialized datasets and tasks.Both approaches aim to support new expertise while preserving general knowledge.
  • 3.2 CPT for Updating Domains: FinPythia-6.9B and EcomGPT-CT illustrate domain-adaptive continual pretraining for financial and e-commerce applications.These studies demonstrate domain expertise across different specialized settings.
  • 3.3 CPT for Language Expansion: Language expansion requires coverage of underrepresented languages together with cultural contexts, regional dialects, and contemporary slang.These capabilities support communication across diverse racial, social, and cultural groups.
  • 3.4 CPT for Coding: CodeTask-CL benchmarks continual code learning across diverse tasks, input-output formats, and programming languages.The benchmark addresses continual learning for programming-language understanding and generation.

4 Continual Instruction Tuning (CIT)

Continual instruction tuning incrementally teaches LLMs to follow instructions across tasks, domains, and tools. The surveyed methods address transfer, computational cost, catastrophic forgetting, and adaptation to changing tool environments.

  • 4 Continual Instruction Tuning (CIT): Continual instruction tuning continually fine-tunes LLMs to follow instructions and transfer knowledge to future tasks.The survey divides CIT into task-incremental, domain-incremental, and tool-incremental settings.
  • 4.1 Task-incremental CIT: Task-incremental CIT trains LLMs on sequences of task-specific instructions to acquire abilities for novel tasks.Direct fine-tuning on new-task data can cause catastrophic forgetting, motivating replay and other mitigation methods.
  • 4.1 Task-incremental CIT: SLM uses vector-space retrieval to support scalable knowledge expansion and quick adaptation without the performance loss associated with catastrophic forgetting.Its retrieval mechanism differs from earlier replay-based and regularization-based approaches.
  • 4.1 Task-incremental CIT: Progressive Prompts freezes most parameters and learns a fixed number of task-specific prompt tokens, reducing computational cost while alleviating forgetting.The method also improves transfer of knowledge to future tasks.
  • 4.2 Domain-incremental CIT: Domain-incremental CIT fine-tunes LLMs on sequences of domain-specific instructions to acquire knowledge for tasks in novel domains.TAPT adapts models across biomedicine, computer science, news, and shopping-review data before evaluating domain-specific classification.
  • 4.2 Domain-incremental CIT: Training-data order significantly affects continual multi-domain performance, motivating Mixed Fine-tuning to learn multiple domain abilities.ConPE retrieves domain knowledge to revise an LLM’s initial response into a final answer.
  • 4.3 Tool-incremental CIT: Tool-incremental CIT continually integrates calculators, search engines, databases, APIs, software libraries, and domain utilities into LLM capabilities.ToolkenGPT represents each tool as a token whose embedding is learned during instruction tuning, enabling adaptation by adding tokens.

5 Continual Alignment (CA)

Continual alignment updates LLM behavior as societal values, ethical guidelines, demographic groups, and preferences evolve. Preference alignment must balance new rewards with retention of earlier preferences under difficult training conditions.

  • 5 Continual Alignment (CA): Continual alignment adapts LLM outputs to changing societal values, social norms, ethical guidelines, and demographic preferences.Preferences can differ across demographic groups and change over time.
  • 5.1 Continual Value Alignment: Continual value alignment incorporates new ethical or cultural norms while unlearning outdated notions, extending editing and unlearning concerns into preference learning.The passage states that model editing and knowledge unlearning have not yet been explored in preference learning.
  • 5.2 Continual Preference Alignment: Adding demographic groups or value types requires retaining previous preferences while maximizing rewards for newly introduced preferences.Open-source aligned LLMs commonly use RLHF for safety and may add attributes such as helpfulness and fairness.
  • 5.2 Continual Preference Alignment: Continual preference learning faces stable, efficient-training difficulties because LLMs combine large vocabularies with many parameters.CPPO weights samples in PPO, while continual DPO uses Monte Carlo estimation and prior policies to regularize learning on new tasks.

6 Benchmarks

Continual-learning benchmarks span evolving factual knowledge, instruction-following tasks, and preference datasets. The survey highlights a major evaluation gap: dedicated benchmarks for continual alignment are absent.

  • 6 Benchmarks: Systematic evaluation requires benchmarks with high-quality data sources and diverse content.The survey summarizes benchmark datasets for several continual-learning settings.
  • 6.1 Benchmarks for CPT: TemporalWiki evaluates retaining past knowledge and acquiring new knowledge from consecutive Wikipedia and Wikidata snapshots.Firehose provides 100 million tweets from one million users over six years, while CKL uses web and news data.
  • 6.2 Benchmarks for CIT: CITB contains over 1,600 NLP tasks across 76 types, while ConTinTin contains 61 tasks across six categories.Both benchmarks are derived from instruction datasets and use text-to-text task formats.
  • 6.3 Benchmarks for CA: Dedicated continual-alignment benchmarks are notably absent despite experiments using SHP and Helpful & Harmless preference datasets.SHP contains 385,000 preferences across 18 subjects, whereas HH covers helpful and harmful interactions.

7 Evaluation

The survey evaluates continual learning across task sequences using average performance, forward transfer, and backward transfer, while also measuring cross-stage forgetting in general abilities, instruction following, and safety.

  • 7.1 Evaluation for Target Task Sequence: Continual learning is evaluated over task sequences using average performance, Forward Transfer Rate, and Backward Transfer Rate.Average performance captures adaptation over time; FWT assesses prior knowledge’s effect on a new task before training, while BWT measures forgetting on old tasks.
  • 7.1 Evaluation for Target Task Sequence: FWT measures how knowledge from previous tasks affects initial performance on a new task before dedicated training.The metric evaluates transfer from earlier tasks into the initial ability on the next task.
  • 7.1 Evaluation for Target Task Sequence: BWT measures catastrophic forgetting by comparing performance on old tasks before and after learning new ones.The comparison directly targets retention of previously learned task abilities.
  • 7.1 Evaluation for Target Task Sequence: Average performance, such as average accuracy, assesses learning and adaptation across a sequence of data streams or tasks.The metric summarizes performance over the continual-learning sequence.
  • 7.2 Evaluation for Cross-stage Forgetting: Cross-stage evaluation uses General Ability Delta, Instruction Following Delta, and Safety Delta to quantify changes after sequential training.These metrics compare general-task, instruction-following, and safety performance against the initial model, using R0,i as the baseline performance on task i.

8 Challenges and Future Works

The survey identifies computation efficiency, social responsibility, automatic learning, and controllable forgetting as major challenges for continual learning in LLMs.

  • Computation-efficient Continual Learning: Computation-efficient continual learning seeks to reduce resources required for continual pretraining while balancing computational cost against model performance.The discussion highlights architectures, algorithms, data structures, and energy-efficient models for scaling continual pretraining.
  • Social Good Continual Learning: Socially responsible continual learning requires secure handling of sensitive data and alignment with human values, culture, and societal norms.Privacy and data security are especially emphasized for continual instruction tuning, while ethical and cultural considerations matter for preference learning.
  • Automatic Continual Learning: Automatic continual learning aims to let LLMs adjust to novel tasks and user preferences without manual intervention.Proposed directions include collaborative multi-agent learning and self-planning algorithms that adapt strategies using performance feedback.
  • Continual Learning with Controllable Forgetting: Controllable forgetting seeks selective retention and removal of information to reduce catastrophic forgetting and unlearn misinformation or outdated knowledge.The challenge is particularly relevant to continual pretraining and also concerns maintaining accuracy and reliability over time.

9 Conclusion

The survey frames continual learning as necessary for keeping LLMs current and presents it as a multi-stage process spanning pretraining, instruction tuning, and alignment. It organizes research by learning stage and information type while identifying challenges and future directions.

  • 9 Conclusion: Continual learning enables LLMs to be regularly and efficiently updated as human knowledge, language, and values change.The conclusion presents updating as necessary for maintaining currency over evolving information and norms.
  • 9 Conclusion: The survey characterizes LLM continual learning as a multi-stage process involving continual pretraining, instruction tuning, and alignment.It contrasts this complexity with continual-learning approaches used for smaller models.
  • 9 Conclusion: The paper categorizes updates by learning stages and information types and discusses major challenges and future work directions.Its stated goal is to provide a comprehensive account of recent developments in continual learning for LLMs.
Loading 2402.01364v2…