Source-linked AI summary
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang
TL;DR
The paper asks whether continual instruction tuning makes LLMs forget general knowledge, a practical concern when original training data are inaccessible. It empirically evaluates this across models, tasks, scales, and architectures, finding widespread forgetting but mitigation from general instruction tuning.
Problem
The paper examines whether continual instruction tuning causes LLMs to forget stored general knowledge when original training data are inaccessible.
Method
The study continually trains BLOOMZ, mT0, LLAMA, and ALPACA on five instruction tasks and evaluates domain knowledge, reasoning, reading comprehension, and bias.
Results
Catastrophic forgetting generally occurs across LLMs, intensifies from 1b to 7b parameters, is milder in BLOOMZ than mT0, and can be alleviated by general instruction tuning.
Takeaways & Limitations
Continual instruction tuning can degrade retained general knowledge, while architecture choice and prior general instruction tuning may influence knowledge retention.
Takeaways & Limitations
The experiments cover models from 1b to 7b, use one task order, and evaluate only selected popular benchmarks because of computational constraints.
Abstract
from arXiv · showhide
Catastrophic forgetting (CF) is a phenomenon that occurs in machine learning when a model forgets previously learned information while acquiring new knowledge for achieving a satisfactory performance in downstream tasks. As large language models (LLMs) have demonstrated remarkable performance, it is intriguing to investigate whether CF exists during the continual instruction tuning of LLMs. This study empirically evaluates the forgetting phenomenon in LLMs' knowledge during continual instruction tuning from the perspectives of domain knowledge, reasoning, and reading comprehension. The experiments reveal that catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b parameters. Surprisingly, as the model scale increases, the severity of forgetting intensifies in such a model sale range which may result from the much significant initial performance in the larger LLM. Comparing the decoder-only model BLOOMZ with the encoder-decoder model mT0, BLOOMZ exhibits less forgetting and retains more knowledge. Interestingly, we also observe that LLMs can mitigate language biases, such as gender bias, during continual fine-tuning. Furthermore, our findings indicate that general instruction tuning can help alleviate the forgetting phenomenon in LLMs during subsequent fine-tuning.
1 Introduction
The paper investigates whether continual instruction tuning causes catastrophic forgetting of LLMs’ general knowledge and how model properties and tuning strategies affect it. Experiments find forgetting across models, with stronger effects at larger scales, while general instruction tuning can mitigate it.
- Continual instruction tuning may affect general knowledge stored in LLMs because original training data are inaccessible during task-specific fine-tuning.
- The study asks whether general knowledge is forgotten and how model scale, architecture, and general instruction tuning affect or mitigate forgetting.
- Catastrophic forgetting is generally present across LLMs, and its severity intensifies as model scale increases from 1b to 7b parameters.The authors suggest that larger models’ stronger initial performance may lead to more pronounced degradation after fitting new tasks.
- BLOOMZ shows milder forgetting than mT0 at comparable scale, while diverse general instruction tuning can alleviate forgetting during subsequent fine-tuning.
- The empirical evaluation measures retention across domain knowledge, reasoning, and reading comprehension while also examining bias evolution.
2 Related Work
The paper situates continual instruction tuning within instruction-tuning and catastrophic-forgetting research. It emphasizes that prior work leaves the effect on pretrained general knowledge insufficiently examined.
- Instruction tuning aligns pretrained language-model responses with human intents or preferences and supports models such as T0, mT0, and BLOOMZ.
- Prior catastrophic-forgetting studies mainly evaluate performance decreases on previously learned tasks or average final performance during continual learning.
- The study’s framework continually trains an initial model with instruction tasks and evaluates domain knowledge, reasoning, reading comprehension, and bias.
3 Method
The method sequentially fine-tunes an initial LLM on five generation tasks, then evaluates retained general knowledge and bias across multiple benchmark sets. Forgetting is summarized by the average decrease in evaluation results across task stages.
- Continual Instruction Tuning: The model sequentially learns generation tasks T = {T_m}, using only the corresponding data D_m at each continual-training stage.
- Continual Tasks: The five instruction tasks are text simplification, empathetic dialogue generation, inquisitive question generation, explanation generation, and constrained headline generation.
- Continual Tasks: Training uses 100,000 samples in the order Simp → Emdg → InqQG → Exp → HGen, with a general prompt template preceding each task-specific prompt.
- Evaluation: General-knowledge retention is evaluated through MMLU domain knowledge, commonsense and mathematical or dialogue reasoning, RACE reading comprehension, and CrowSPairs bias.
- Evaluation Metric: The forgetting metric FG averages decreases in evaluation results R_m across evaluation elements as the model undergoes continual task training.
- Evaluation Metric: Most multiple-choice tasks use zero-shot accuracy, MMLU uses five-shot evaluation, and CrowSPairs bias is measured from stereotypical-sentence preference.
4 Experimental Setting
The study uses BLOOMZ, mT0, LLAMA, and ALPACA to examine continual instruction tuning across model architectures and scales. It compares models before and after tuning on corresponding tasks under specified training conditions.
- Models: BLOOMZ, mT0, LLAMA, and ALPACA are included to study continual instruction tuning across model families.BLOOMZ and mT0 support architecture comparisons, while LLAMA and ALPACA support analysis of general instruction tuning.
- Models: BLOOMZ is evaluated at 1.1b, 1.7b, 3b, and 7.1b scales, while mT0 is evaluated at 1.2b and 3.7b.These settings provide comparisons across model scale and architecture.
- Models: BLOOMZ is a decoder-only model based on BLOOM, whereas mT0 is an encoder-decoder model based on T5.
- Models: LLAMA is an open-source decoder-only model, and ALPACA is fine-tuned from LLAMA-7b using 52K instruction data.
- Training Details: Training uses Adam with batch size 4 per device and learning rate 2e-5; scheduler settings differ between BLOOMZ/mT0 and LLAMA/ALPACA.BLOOMZ and mT0 use a constant scheduler, while LLAMA and ALPACA use a cosine scheduler.
5 Experimental Results and Analysis
Continual instruction tuning improves target-task performance but generally causes catastrophic forgetting of general knowledge across domain knowledge, reasoning, and reading comprehension. Forgetting varies with model scale, architecture, and prior general instruction tuning, while bias measures generally improve.
- 5.1 Main Results: Instruction tuning improves performance on the corresponding instruction tasks, including BLOOMZ-7.1b Exp from 51.47% to 68.71% BLUEScore.These gains show that continual tuning benefits the newly targeted tasks despite later losses on general knowledge.
- 5.1 Main Results: BLOOMZ-7.1b MMLU-SocialScience performance drops from 36.18% to 26.06%, and additional instruction tasks produce more significant forgetting.On MMLU-Other, BLOOMZ-7.1b falls from 36.18% to 26.35%, compared with 30.58% to 25.97% for BLOOMZ-1.1b.
- 5.1 Main Results: Positive FG values across domain knowledge, reasoning, and reading comprehension indicate that general knowledge is forgotten during continual instruction tuning.Reading comprehension suffers the most drastic forgetting, followed by domain knowledge.
- 5.1 Main Results: BLOOMZ-7.1b has FG values of 26.75% for reading comprehension, 18.37% for domain knowledge, and 13.62% for reasoning, while bias FG values are mostly above zero.The bias pattern suggests mitigation of biases related to race, color, and gender; stereotype-conforming appearance preferences decrease from 75.0% to 63.88%.
- 5.2 Effect of Scales: Domain-knowledge FG rises from 9.54% in BLOOMZ-1.1b to 18.37% in BLOOMZ-7.1b, indicating increasingly severe forgetting with model scale.Larger models start with stronger performance, but final performance is relatively similar across scales, producing larger performance degradation.
- 5.3 Effect of Model Architecture: At comparable scale, BLOOMZ-1.1b and mT0-1.2b have similar FG values, whereas BLOOMZ-3b has an FG value of 11.09, 5.64 lower than mT0-3.7b.The results suggest decoder-only BLOOMZ retains more knowledge than encoder-decoder mT0 during continual instruction tuning.
- 5.4 Effect of General Instruction Tuning: ALPACA maintains more general knowledge than LLAMA after continual fine-tuning when their initial performances are similar, supporting a mitigating role for general instruction tuning.Mixing 10,000 general instruction samples with continual tasks also mitigates forgetting in BLOOMZ-7b and LLAMA-7b.
6 Conclusion
The study finds catastrophic forgetting is prevalent during continual instruction tuning, with greater forgetting at larger scales and differences across architectures. General instruction tuning may alleviate forgetting, while larger-scale investigation remains needed.
- Catastrophic forgetting is generally prevalent during continual fine-tuning across the evaluated LLMs.
- As model scale increases, forgetting becomes more severe across domain knowledge, reasoning, and reading comprehension.
- BLOOMZ retains knowledge and skills better than the encoder-decoder model mT0 during continual fine-tuning.
- General instruction tuning may help alleviate catastrophic forgetting during subsequent fine-tuning.
- The study identifies larger model scales and more effective mitigation methods as important directions for future research.
7 Limitations
The study is an initial analysis constrained by computation, task-order choices, and a limited selection of evaluation benchmarks. These constraints bound the model scales, experimental configurations, and breadth of evaluation.
- Restricted computation prevented experiments on models larger than the evaluated 1b-to-7b scale range.
- The experiments use one controlled task order, which may affect the observed forgetting phenomenon.
- Only some popular benchmarks were adopted because evaluating many alternatives would make computation prohibitively expensive.
A Instruction Task Details
The study provides instruction samples for the continual instruction tasks in Table 7.
- Instruction samples for the continual instruction tasks are presented in Table 7.
- Table 7 documents examples of the instruction data adopted in the study.
- The table supports inspection of the task inputs used during continual instruction tuning.