PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Catastrophic Forgetting in Large Language Model Tuning

Hongyu Li, Liang Ding, Meng Fang, Dacheng Tao

arXiv:2406.04836v1cs.CLcs.AI

TL;DR

Catastrophic forgetting threatens LLM fine-tuning, and its underlying causes remain largely unexplored. The paper links forgetting to loss-landscape flatness and applies sharpness-aware minimization to flatten the landscape. It reports that the method mitigates forgetting, especially in larger models, and complements existing anti-forgetting methods.

  • Problem

    Catastrophic forgetting can compromise LLM fine-tuning, while its underlying causes remain largely unexplored.

    Source: §1 · p. 1

  • Method

    The paper applies sharpness-aware minimization to flatten the loss landscape as an optimization-based approach to mitigating catastrophic forgetting.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The authors report that flattening the loss landscape mitigates forgetting, with effectiveness especially evident in larger models.

    Source: §1 · p. 1, §6 · p. 4

  • Takeaways & Limitations

    The authors report that their optimizer can complement rehearsal and Wise-FT to enhance LLM resistance to forgetting.

    Source: §1 · p. 1

    The study focuses on forgetting during LLM fine-tuning and does not directly address forgetting during other stages, such as post-deployment updates.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Catastrophic Forgetting (CF) means models forgetting previously acquired knowledge when learning new data. It compromises the effectiveness of large language models (LLMs) during fine-tuning, yet the underlying causes have not been thoroughly investigated. This paper takes the first step to reveal the direct link between the flatness of the model loss landscape and the extent of CF in the field of LLMs. Based on this, we introduce the sharpness-aware minimization to mitigate CF by flattening the loss landscape. Experiments on three widely-used fine-tuning datasets, spanning different model scales, demonstrate the effectiveness of our method in alleviating CF. Analyses show that we nicely complement the existing anti-forgetting strategies, further enhancing the resistance of LLMs to CF.

1 Introduction

Catastrophic forgetting (CF) threatens the effectiveness of LLM fine-tuning, while its underlying causes remain largely unexplored. This work links CF to loss-landscape flatness and uses sharpness-aware minimization to mitigate forgetting and complement existing strategies.

1 Introduction

  • CF means LLMs forget prior knowledge when learning new data, and prior work reports degraded general performance after fine-tuning.Understanding fine-tuning effectiveness matters for downstream tasks, but the causes of CF remain largely unexplored.Source: §1 · p. 1
  • Prior anti-forgetting approaches can require costly data construction and training, and rehearsal may be infeasible when pretrained-model data cards are unclear.Anti-forgetting training can also be unstable and expensive.Source: §1 · p. 1
  • The paper reports a direct correlation between loss-landscape flatness and CF, and finds flatter landscapes associated with less severe forgetting.The authors describe this as the first empirical reveal of that direct correlation.Source: §1 · p. 1
  • The authors introduce sharpness-aware minimization to flatten the loss landscape and report that it complements rehearsal and Wise-FT.They present the approach as an optimization-based way to mitigate LLM CF.Source: §1 · pp. 1–2

2 Unveiling the Hidden Nexus: CF and the Model LLS

The section probes the relationship between catastrophic forgetting and loss-landscape flatness through visualization and quantitative measures. Across increasingly difficult continued-training settings, sharper landscapes accompany larger general-task performance drops.

2 Unveiling the Hidden Nexus: CF and the Model LLS

  • The authors investigate the relationship using loss-landscape visualization, flatness metrics, and general-task performance in simulated continued-learning settings.The simulations use Llama2-7b tuned on Alpaca, Alpaca→Open-Platypus, and Alpaca→Auto-Wiki, with data gaps increasing across settings.Source: §2 · p. 2

2.2 Quantitative flatness degree and general performance of different CF settings

  • MMLU performance drops by up to -17.2 on average across the increasingly difficult settings, while flatness-degree averages rise from 74.59 to 78.50.The reported ordering is (a)≥(b)≥(c) for flatness degree and (a)≤(b)≤(c) for task difficulty, indicating a positive correlation between forgetting severity and landscape sharpness.Source: Table 1 · p. 2, §2.2 · p. 2

2.1 Visualized LLS comparison of different CF settings

  • As the continued-learning task gap grows from (a) to (c), the loss landscape and contour lines become sharper and more disturbed.The authors present this visualization as intuitive evidence of a correlation between landscape flatness and forgetting degree.Source: §2.1 · p. 2

3 Improving CF with Sharpness-Aware Optimization

The paper applies sharpness-aware minimization to flatten the optimization landscape and mitigate catastrophic forgetting. Its experiments vary instruction-tuning datasets and model sizes, assessing retained general knowledge across several task categories.

3 Improving CF with Sharpness-Aware Optimization

  • SAM seeks weights with loss that does not increase substantially under a perturbation constrained to a neighborhood of radius ρ.The paper describes a two-step gradient-descent process: compute the perturbation, then update the weights.Source: §3 · p. 3

4 Experimental Setting

  • Experiments use Alpaca, ShareGPT52K, MetaMathQA, and Open-Platypus, with TinyLlama-1.1B, Llama2-7B, and Llama2-13B.The Alpaca-trained instruction-tuning model serves as the baseline for subsequent fine-tuning experiments.Source: §4 · p. 3
  • Evaluation covers domain knowledge, reasoning, understanding, and exams, with TruthfulQA used to assess retention of specific abilities.The listed tasks include MMLU, SuperGLUE, Hellaswag, Boolq, Siqa, RACE, Openbookqa-fact, Csl-dev, and ARC-c.Source: §4 · p. 3

5 Experimental Results

The section introduces four experimental analyses of the proposed method.

5 Experimental Results

  • The authors assess the proposed method through four experimental analyses.Source: §5 · p. 3

4. Complementary to existing works

SAM mitigates forgetting across fine-tuning datasets and model sizes, outperforms Wise-FT and Rehearsal, and provides additional gains when combined with them. The paper links loss-landscape flatness to forgetting and proposes sharpness-aware optimization to flatten the landscape.

4. Complementary to existing works

  • Across ShareGPT52K, Open-Platypus, and MetaMathQA fine-tuning settings, SAM markedly mitigates CF and brings performance comparable to the baseline.The comparison uses balanced training costs: two epochs without SAM and one with SAM.Source: §4 · pp. 3–4, Table 2 · p. 4
  • For Alpaca→Open-Platypus, ∆DK is 0.52% for TinyLlama-1.1B, 23.19% for Llama2-7B, and 41.37% for Llama2-13B; SAM significantly mitigates CF as model size increases.The authors report that forgetting severity increases with model size.Source: §4 · p. 4, Table 2 · p. 4
  • 7.01% (SAM), 5.82% (Wise-FT), and 3.79% (Rehearsal) are the reported ∆Mitigating CF values; SAM significantly outperforms both methods.The comparison uses Table 2(c).Source: §4 · p. 4, Table 2 · p. 4
  • Wise-FT(w/) improves ∆Mitigating CF by 0.97%, while Rehearsal(w/) improves it by 3.02%, showing incremental benefits when combined with SAM.The authors describe the combination as orthogonal.Source: §4 · p. 4

6 Conclusion

  • The conclusion links loss-landscape flatness to CF and presents sharpness-aware optimization as a way to flatten the landscape and mitigate forgetting.The authors suggest SAM may become a standard strategy for mitigating CF during LLM tuning.Source: §6 · p. 4
  • Future work will explore orthogonal approaches to forgetting in LLM tuning, including advanced optimizers, learning curricula, and model merging.The cited examples include Sun et al. (2024), Ding et al. (2021), and Li et al. (2023).Source: §6 · p. 5

Limitations

The authors limit their findings to flatness-related CF during LLM fine-tuning; other factors and lifecycle stages remain outside the study. They also report adherence to the ACL Ethics Policy and provide experimental details and code for reproducibility.

Limitations

  • The experiments focus on the link between loss-landscape flatness and CF during fine-tuning, leaving other potential CF factors unexplored.The authors caution that findings may apply only to specific phases of the LLM lifecycle, not post-deployment updates.Source: Limitations · p. 5

Ethics Statement

  • The authors state that they adhere to the ACL Ethics Policy and that the models and datasets used are publicly available and widely adopted.They report that experimental results are reported accurately and objectively and believe the research poses no ethical issues.Source: Ethics Statement · p. 5
  • The paper says it provides detailed experimental setup in the appendix and releases code to facilitate reproduction.The cited passage gives the code repository address.Source: Ethics Statement · p. 5

A Appendix

The appendix describes training datasets and evaluation tasks used in the study, including instruction, conversation, reasoning, and general-knowledge resources.

A.1.1 Training Datasets

  • The training datasets include Alpaca instruction demonstrations, ShareGPT52K conversations, MetaMathQA math instructions, Open-Platypus reasoning data, and Auto-Wiki sentence pairs.The appendix describes Alpaca as 52,000 instructions, ShareGPT52K as approximately 52,000 conversations, and MetaMathQA as 395,000 instructions.Source: §A.1.1 · p. 7

A.1.2 Evaluation Tasks

  • Evaluation resources include MMLU for stored knowledge, AX-b and AX-g diagnostics, and RTE, COPA, HellaSwag, BoolQ, and SIQA tasks.AX-b tests sentence-pair logical relations; AX-g tests pronoun reference, while the other listed tasks cover entailment, causal inference, commonsense, or social interaction.Source: §A.1.2 · pp. 7–8
  • The evaluation set also includes RACE reading comprehension, OpenBookQA multi-step reasoning, CSL scientific literature, ARC-c science questions, and TruthfulQA.The appendix describes RACE as containing more than 28,000 passages and nearly 100,000 questions.Source: §A.1.2 · p. 8

A.2 Experimental Setup

The setup trains base models on Alpaca first, then compares downstream fine-tuning with and without SAM using specified training budgets and a recommended SAM ρ value.

A.2 Experimental Setup

  • The base models are TinyLlama, Llama2-7B, and Llama2-13B; initial Alpaca training uses three epochs, learning rate 2e-5, and batch size 128.The authors say this configuration yields Alpaca performance roughly equivalent to the original model reported by Taori et al. (2023).Source: §A.2 · p. 8
  • For subsequent fine-tuning, both settings use learning rate 5e-6 and batch size 128; training uses two epochs without SAM and one with SAM.The no-SAM optimizer is AdamW.Source: §A.2 · p. 8
  • The authors recommend ρ = 2 for SAM training.Source: §A.2 · p. 9

A.3 Flatness Metric

The section describes three measures of loss-landscape flatness: average curvature, average gradient, and mean absolute gradient. Lower curvature and lower gradient measures indicate flatter regions.

A.3 Flatness Metric

  • Lower average curvature indicates a flatter loss landscape, with overall curvature computed as the mean absolute curvature across the parameter space.The local curvature is defined from the function’s second derivatives.Source: §A.3 · p. 9
  • Average gradient is the mean of gradient vectors within a region; smaller values typically correspond to flatter loss-landscape regions.Source: §A.3 · p. 9
  • Mean Absolute Gradient (MAG) measures the average rate of loss change across the parameter space, with lower MAG tending to indicate a flatter landscape.The passage identifies each t_i as a data point in the loss landscape.Source: §A.3 · p. 9

A.4 Experiment on different Epochs

Across the appendix experiments, SAM is reported to outperform Wise-FT and Rehearsal, perform best across tested epoch settings, preserve fine-tuning performance in specific domains with limited impact, and generalize across model sizes.

A.4 Experiment on different Epochs

  • Across the three tested settings on the same dataset, SAM with the method consistently achieves the best results; epoch 2 is superior to epoch 1 in most cases.The comparison accounts for differing training epochs between SAM with and without the method.Source: §A.4 · p. 9

A.5 Performance in fine-tuning domain

  • SAM mitigates catastrophic forgetting with limited impact on fine-tuning performance in specific domains.Source: §A.5 · p. 9

A.6.1 Comprehensive Experimental Results on different datasets

  • The Llama2-7B comprehensive results report model performance across evaluation tasks during fine-tuning on different datasets.Source: §A.6.1 · p. 9

A.6.2 Comprehensive Experimental Results across different model sizes

  • Experiments on Open-Platypus across different model sizes are presented as evidence of SAM’s generality across sizes.Source: §A.6.2 · p. 9

A.6.3 Comprehensive Experimental Results of comparison with other advanced methods

  • SAM significantly outperforms Wise-FT and Rehearsal in the comprehensive comparison.Source: §A.6.3 · p. 9

A.6.4 Comprehensive Experimental Results of complementary to existing works

Combining SAM with existing anti-forgetting methods provides incremental benefits in mitigating catastrophic forgetting.

A.6.4 Comprehensive Experimental Results of complementary to existing works

Related papers

Loading 2406.04836v1…