PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su

arXiv:2403.01244v2cs.CLcs.AI

TL;DR

Catastrophic forgetting threatens LLMs as they learn new tasks, while conventional rehearsal depends on earlier training data that may be unavailable. SSR generates synthetic rehearsal instances with the base LLM, refines outputs using the latest LLM, and selects diverse examples. It achieves superior or comparable performance to conventional rehearsal approaches with higher data utilization efficiency, and preserves general-domain generalization.

  • Problem

    LLMs can forget previously acquired knowledge during continual learning, and rehearsal may be infeasible when previous training data is unavailable.

    Source: §1 · p. 1

  • Method

    SSR uses the base LLM with in-context demonstrations to synthesize instances, the latest LLM to refine their outputs, and diverse selected instances for future rehearsal.

    Source: §1 · pp. 1–2

  • Results

    SSR achieves superior or comparable performance to conventional rehearsal-based approaches with higher data utilization efficiency, while preserving general-domain generalization capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SSR preserves acquired knowledge without relying on real data during rehearsal and is presented as a flexible approach for continual learning in real-world settings.

    Source: §6 · p. 9

    SSR may not achieve the best FWT score, and LLM-generated synthetic instances may contain unsafe content due to training-data bias.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.

1 Introduction

Continual learning can make LLMs forget previously acquired knowledge, while rehearsal may be impractical when earlier training data is unavailable. SSR synthesizes rehearsal instances and achieves superior or comparable performance to conventional rehearsal approaches with higher data utilization efficiency.

1 Introduction

  • Catastrophic forgetting—the loss of previously acquired knowledge when learning new instances—limits LLM effectiveness during continual learning.Continual learning incrementally introduces new instruction-tuning data.Source: §1 · p. 1
  • Rehearsal methods retain ability by training on real data from earlier stages, but that data may be unavailable for publicly released LLM checkpoints.The paper asks whether LLM ability can be maintained without using real data from previous stages.Source: §1 · p. 1
  • SSR uses the LLM to synthesize rehearsal instances, refining their outputs with the latest LLM and selecting diverse, high-quality instances for future stages.The base LLM generates instances through in-context learning with few-shot demonstrations.Source: §1 · pp. 1–2
  • SSR achieves superior or comparable performance to conventional rehearsal-based approaches with higher data utilization efficiency, and preserves general-domain generalization capabilities.The reported evaluations include task sequences derived from SuperNI, AlpacaEval, and MMLU.Source: §1 · p. 2

2 Related Work

Prior continual-learning approaches manage interference through parameter constraints, task-specific parameters, or rehearsal data, each with practical trade-offs. SSR instead rehearses on self-synthesized data without requiring previous real data or an additional generative model.

2 Related Work

  • Regularization methods constrain parameter updates, while architecture-based methods use task-specific parameters; the former require tuned hyperparameters and the latter add parameters.The paper describes the added parameters as potentially inflexible or infeasible for various LLMs.Source: §2 · p. 2
  • Rehearsal methods store prior-task data for future use, and language-model approaches typically rely on at least some previous data.The text names experience replay and representation consolidation as examples of rehearsal techniques.Source: §2 · p. 2
  • Prior work leaves practical gaps: previous data may be unavailable, data-free distillation targets classification, and added generative models can be costly to train.Self-distillation also faces forgetting in the latest LLMs and knowledge discrepancies across stages.Source: §2 · p. 2
  • SSR rehearses on self-synthesized data using demonstrations, without requiring previous real data or an additional generative model.The paper presents these properties as advantages for data efficiency and application flexibility.Source: §2 · p. 2

3 Rehearsal-Based Continual Learning

Rehearsal-based continual learning augments each stage’s training data with examples from earlier stages before updating the LLM. SSR’s framework instead describes using selected synthetic instances for future rehearsal.

3 Rehearsal-Based Continual Learning

  • At each stage, rehearsal-based methods expand the current instruction data with sampled training instances from previous stages.The augmented data is then used to fine-tune the current LLM.Source: §3 · p. 2
  • The rehearsal ratio r determines the percentage of sampled training instances, and the first stage fine-tunes the base LLM on current-stage data alone.The first-stage setup is described as alleviating catastrophic forgetting, as verified in prior studies.Source: §3 · pp. 2–3
  • SSR’s framework generates synthetic instances with the base LLM, refines their outputs using the latest LLM, then selects diverse, high-quality instances for future rehearsal.This is the three-step framework shown in Figure 2.Source: Figure 2 · p. 3

4 Our Framework

SSR has three stages: synthesize instances through in-context learning, refine their outputs with the latest LLM, and select instances for rehearsal. The selected synthetic data is carried into later stages alongside current training data.

In-Context Learning Based Instance Synthesis

  • In-context synthesis prompts the base LLM with demonstrations, which may come from previous instruction data or be manually constructed to contain similar knowledge.Sampling multiple times yields different synthetic instances; the base LLM is used because in-context learning tends to degrade after supervised fine-tuning on specific tasks.Source: §4 · p. 3

Synthetic Output Refinement

  • The latest LLM refines each synthetic output based on its input so the refined instance retains knowledge acquired by that latest model.The refinement addresses low-quality synthetic instances with unreliable outputs.Source: §4 · p. 3

Rehearsal with Selected Synthetic Instances

  • Clustering groups refined instances, and selecting examples near cluster centroids aims to preserve diversity and quality in rehearsal data.Selected synthetic data from earlier stages is reused rather than regenerated, and the augmented data fine-tunes the current LLM.Source: §4 · pp. 3–4

5 Experiments

Across SuperNI task sequences, SSR outperforms or matches rehearsal baselines while using less real data, and it preserves general-domain capabilities. Analyses examine its robustness to ICL settings, synthetic-output refinement, instance selection, and rehearsal quantity.

5.1 Setup

  • Experiments use default continual-learning orders over 5 or 10 SuperNI tasks, sampling 2,000 training and 500 evaluation instances per task; performance uses ROUGE-L-based AR, FWT, and BWT metrics.Source: §5.1 · p. 4

5.2 Experiments on 5 SuperNI Tasks

  • About 2 scores of improvement in both AR and BWT over rehearsal-based baselines; SSR also approaches MTL’s AR upper bound on five SuperNI tasks.SSR uses only 1% real data for ICL and synthetic data from previous stages for rehearsal.Source: Table 1 · p. 5, §5.2 · p. 5

5.3 Experiments on 10 SuperNI Tasks

  • SSR surpasses rehearsal baselines across all metrics on ten SuperNI tasks, with stronger AR and BWT throughout; its FWT eventually surpasses RandSel(10%) after trailing early.The reported results cover the final comparisons and performance over the continual-learning sequence.Source: Table 2 · p. 6, Figure 3 · p. 6, §5.3 · p. 6

5.4 Experiments on the Generalization Capability Preservation of Alpaca-7b

  • SSR achieves the best results on five newly learned tasks while maintaining comparable or better AlpacaEval and MMLU performance, preserving Alpaca-7b’s generalization without Alpaca-52k rehearsal data.The experiments evaluate Alpaca-52k alongside five SuperNI tasks.Source: §5.4 · pp. 6–7, Table 3 · p. 7

5.5 Analysis

  • SSR performs well without the original base LLM or prior-training demonstrations, and input-only demonstrations yield comparable performance, indicating output annotations are not essential for ICL.Using Alpaca-7b for ICL causes a slight performance decrease, which the authors attribute to its ICL capability limitation.Source: Table 4 · p. 7, §5.5 · p. 7
  • Without synthetic output refinement, SSR has lower AR and significantly worse BWT; refinement lets rehearsal preserve the latest LLM’s predictive distribution and acquired knowledge.The authors attribute the unrefined variant’s decline to noise from the base LLM.Source: Figure 4 · p. 7, §5.5 · p. 7
  • Increasing the SSR ratio can improve final AR, but training cost and memory matter; SSR also outperforms L2 and EWC, which can be combined with it.The ratio is the selected synthetic-data quantity relative to the original training-data size.Source: Figure 8 · p. 8, §5.5 · pp. 8–9, Table 5 · p. 9
  • Real inputs with synthetic outputs outperform real-instance rehearsal, while SSR sometimes surpasses both; random selection can outperform SSR, so unsupervised K-means is not essential.Supervised K-means slightly improves AR and BWT, while random synthetic-instance selection can sometimes exceed it.Source: §5.5 · p. 8

6 Conclusion

The conclusion presents Self-Synthesized Rehearsal (SSR) as a data-efficient approach to continual learning that preserves LLM capabilities without real rehearsal data.

6 Conclusion

  • SSR mitigates catastrophic forgetting while preserving knowledge without relying on real data during rehearsal.Source: §6 · p. 9
  • Experiments show SSR has data efficiency and superior performance to conventional rehearsal-based approaches.Source: §6 · p. 9
  • SSR preserves LLM generalization in both specific and general domains, with flexibility and robustness in real-world scenarios.Source: §6 · p. 9

Limitations

SSR does not always achieve the best FWT score, and its synthetic instances may potentially contain unsafe content due to training-data bias.

Limitations

  • SSR may not always achieve the best FWT score, despite superior AR and BWT.Source: Limitations · p. 9
  • Synthetic instances generated by LLMs may potentially contain unsafe content due to training-data bias.Source: Limitations · p. 9

A Details of the Selected 10 SuperNI Tasks

The appendix lists the selected SuperNI tasks and uses abbreviations for them in the paper.

A Details of the Selected 10 SuperNI Tasks

B More Details of Experiments on 5 SuperNI Tasks

The appendix reports continual-learning orders and final FWT results for five SuperNI tasks, with SSR's FWT comparison depending on rehearsal ratio and training stages.

B More Details of Experiments on 5 SuperNI Tasks

C More Details of Experiments on 10 SuperNI Tasks

On 10 SuperNI tasks, SSR retains superiority in AR and BWT, while its FWT is comparable to or below rehearsal-based baselines. Heatmaps also show minimal future-stage performance change after SSR learns a task, unlike the rapid forgetting of the non-rehearsal baseline.

C More Details of Experiments on 10 SuperNI Tasks

D More Implementation Details of Experiments on the Generalization Capability Preservation of Alpaca-7b

The implementation details specify the continual-learning order, demonstration-data proportions, clustering counts, and SSR retention ratios. The accompanying captions identify the selected-task table and ROUGE-L heatmaps for 10 SuperNI tasks.

D More Implementation Details of Experiments on the Generalization Capability Preservation of Alpaca-7b

Loading 2403.01244v2…