Source-linked AI summary
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, Xiangzhan Yu
TL;DR
Sequential fine-tuning of deep pretrained language models can cause catastrophic forgetting, creating a need for less-forgetting adaptation. The paper jointly recalls pretraining knowledge and learns downstream tasks through data-free Pretraining Simulation and Objective Shifting, achieving state-of-the-art GLUE performance and stronger BERT-base results than directly fine-tuning BERT-large.
Problem
Sequential transfer learning can cause catastrophic forgetting as pretrained language models adapt to downstream tasks.
Method
The recall-and-learn mechanism uses multi-task learning with Pretraining Simulation to recall knowledge without pretraining data and Objective Shifting to focus gradually on downstream learning.
Results
The method achieves state-of-the-art performance on GLUE and enables BERT-base to outperform directly fine-tuned BERT-large.
Takeaways & Limitations
RECADAM integrates the proposed mechanisms into Adam and provides an open-source optimizer for fine-tuning deep pretrained language models with less forgetting.
Takeaways & Limitations
Pretraining Simulation relies on a stronger assumption that each diagonal Fisher information value is independent of its corresponding parameter.
Abstract
from arXiv · showhide
Deep pretrained language models have achieved great success in the way of pretraining first and then fine-tuning. But such a sequential transfer learning paradigm often confronts the catastrophic forgetting problem and leads to sub-optimal performance. To fine-tune with less forgetting, we propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Specifically, we propose a Pretraining Simulation mechanism to recall the knowledge from pretraining tasks without data, and an Objective Shifting mechanism to focus the learning on downstream tasks gradually. Experiments show that our method achieves state-of-the-art performance on the GLUE benchmark. Our method also enables BERT-base to achieve better performance than directly fine-tuning of BERT-large. Further, we provide the open-source RecAdam optimizer, which integrates the proposed mechanisms into Adam optimizer, to facility the NLP community.
1 Introduction
Sequential fine-tuning of pretrained language models can cause catastrophic forgetting, motivating a recall-and-learn approach based on multi-task learning. The paper introduces data-free pretraining simulation, objective shifting, and the RECADAM optimizer to improve fine-tuning.
- Motivation: Catastrophic forgetting occurs during adaptation when models forget previously learned knowledge and overfit to target domains.Existing work mainly reduces forgetting through fine-tuning strategies such as discriminative fine-tuning, slanted triangular learning rates, gradual unfreezing, or parameter mixing.
- Motivation: Multi-task learning can avoid forgetting by jointly training on pretraining and downstream tasks, but conventional methods require pretraining data and optimize both task types.Pretraining data may be inaccessible or too large, while the practical objective is downstream-task performance.
- Proposed Approach: Pretraining Simulation recalls pretrained knowledge without pretraining data by simulating the pretraining objective using only pretrained parameters.This mechanism addresses the data obstacle in applying multi-task learning during adaptation.
- Proposed Approach: Objective Shifting balances knowledge recalling and new-task learning by gradually shifting the multi-task objective toward the downstream task.The mechanism addresses the mismatch between multi-task and downstream-only optimization objectives.
- Results and Resources: Experiments show significant gains over vanilla BERT-base fine-tuning, better BERT-base results than directly fine-tuning BERT-large, and state-of-the-art GLUE performance with ALBERT-xxlarge.The paper also releases RECADAM, an Adam-based optimizer integrating the proposed mechanisms.
2 Background
Sequential transfer learning trains on source and target tasks in stages, whereas multi-task learning trains on them simultaneously. Applying multi-task learning to pretrained-language-model adaptation faces unavailable pretraining data and a mismatch between optimization objectives.
- Sequential Transfer Learning: Sequential transfer learning first trains on source tasks and then adapts the model to target tasks.Fine-tuning updates pretrained parameters, while feature extraction keeps the pretrained model fixed during adaptation.
- Sequential Transfer Learning: Deep pretrained language models achieve broad NLP success but tend to forget source-task knowledge while learning target-task knowledge.This forgetting problem arises during adaptation in sequential transfer learning.
- Multi-task Learning: Multi-task learning trains on multiple tasks simultaneously by sharing knowledge and improving performance across them.Its objective includes both source and target tasks.
- Multi-task Learning: The multi-task objective includes source-task loss, which can inherently avoid catastrophic forgetting; λ ∈(0, 1) balances source and target tasks.The source-task loss remains part of the optimization objective throughout training.
- Challenges: Applying multi-task learning during adaptation is challenged by inaccessible pretraining data and the difference between downstream loss and the weighted multi-task loss.The downstream objective is LossT, whereas multi-task learning optimizes LossM, a weighted combination of LossT and LossS.
3 Methodology
The method combines Pretraining Simulation with Objective Shifting to retain pretrained knowledge while gradually prioritizing downstream-task learning. RECADAM integrates these mechanisms into Adam through decoupled parameter penalties and annealing.
- Pretraining Simulation: Pretraining Simulation approximates the source-task objective with a quadratic penalty that keeps parameters close to their pretrained values.The approximation is motivated by Laplace and Fisher-information assumptions and avoids requiring pretraining data.
- Objective Shifting: Fine-tuning and multi-task learning emerge as limiting cases of the method when k →∞ and k →0, respectively.With k →0, the model optimizes 1/2(LossT + LossS) simultaneously.
- Objective Shifting: Objective Shifting replaces a fixed multi-task coefficient with a sigmoid annealing function λ(t) over fine-tuning timesteps.The hyperparameters k and t0 control the annealing rate and timing.
- Objective Shifting: Training initially emphasizes pretraining tasks, then increasingly focuses on target tasks while continuing to recall pretrained knowledge.At the end of training, the optimization objective becomes LossT.
- RECADAM Optimizer: RECADAM integrates the quadratic penalty and annealing coefficient into Adam by decoupling them from gradient updates.This applies the same penalty rate (1 −λ(t))γ to all training-model weights while adapting only the target-task gradient.
- RECADAM Optimizer: RECADAM requires only a one-line modification to Adam plus pretrained parameters and a few mechanism-specific hyperparameters.The implementation retains Adam’s schedule-multiplier procedure and returns optimized parameters after the stopping criterion.
4 Experiments
Experiments evaluate RECADAM on GLUE using BERT-base and ALBERT-xxlarge, comparing it with vanilla fine-tuning and initialization strategies. RECADAM improves performance across most tasks, especially those with limited training data, while objective shifting exposes a forgetting–convergence trade-off.
- Setup: Experiments evaluate BERT-base and ALBERT-xxlarge on the GLUE benchmark, reporting dev-set results for eight tasks and task-specific metrics.WNLI is excluded; STS uses Pearson correlation, CoLA uses Matthews correlation, MNLI uses match accuracy, and other tasks use accuracy.
- Results with BERT-base: RECADAM outperforms vanilla fine-tuning on 7 of 8 GLUE tasks with BERT-base and improves average median performance by 1.1%.The baseline is rerun without further MNLI pretraining, and results report median and maximum over five runs.
- Results with BERT-base: For tasks with fewer than 10k training examples, RECADAM improves BERT-base performance by 1.7% on average over vanilla fine-tuning.The authors attribute these gains to learning target tasks while recalling pretraining knowledge under data scarcity.
- Results with BERT-base: RECADAM with BERT-base exceeds reported BERT-large median results on more than half the tasks and improves the average across GLUE tasks by 0.2%.Reported task improvements include +4.0% on RTE, +0.4% on STS, +1.8% on CoLA, +0.4% on SST, and +0.1% on QQP.
- Results with ALBERT-xxlarge: RECADAM outperforms vanilla fine-tuning on 5 of 8 tasks with ALBERT-xxlarge and reaches 90.2% average median performance.On tasks with fewer than 10k examples, the average median improvement is +1.5%; overall improvement over the baseline is 0.7%.
- Analysis: Random initialization enables RECADAM to outperform vanilla fine-tuning on all four tested tasks, while pretrained initialization matches it on STS.The authors suggest random initialization provides a larger parameter search space than pretrained initialization.
- Analysis: Smaller objective-shifting rates reduce pretrained-knowledge forgetting but require more timesteps to converge on the target task.The trade-off connects objective shifting with the balance between recalling pretrained knowledge and learning the downstream task.
5 Related Works
Related work addresses catastrophic forgetting through replay, parameter isolation, regularization, and multi-task learning. These approaches are constrained in sequential transfer learning when pretraining data or pretraining-task knowledge is unavailable.
- Forgetting remedies: Replay-based methods reduce forgetting by replaying samples from previous tasks while learning a new task.This approach requires access to earlier-task samples.
- Forgetting remedies: Parameter-isolation methods avoid forgetting by assigning task-specific parameters and freezing them for subsequent tasks.The strategy separates parameter updates across tasks rather than sharing all parameters during adaptation.
- Forgetting remedies: Regularization-based methods recall previous knowledge through an additional penalty, using either data-focused distillation or prior-focused constraints.Prior-focused methods such as EWC avoid direct pretraining-data access but require pretraining knowledge such as a Fisher information matrix.
- NLP context: The paper positions its approach as a response to catastrophic forgetting in NLP’s widely used sequential transfer-learning setting.The related-work context identifies forgetting as a recurring issue when pretrained models learn new target-task knowledge.
6 Conclusion
The paper bridges sequential fine-tuning and multi-task learning to address catastrophic forgetting when transferring deep pretrained language models. It introduces data-free pretraining simulation, objective shifting, and the RecAdam optimizer.
- The method bridges sequential fine-tuning and multi-task learning to solve catastrophic forgetting in deep pretrained language-model transfer.
- Pretraining Simulation learns the pretraining task without requiring pretraining data during joint learning.
- Objective Shifting balances learning of pretraining and downstream tasks during adaptation.
- The authors provide the open-source RecAdam optimizer, integrating the proposed mechanisms into Adam to facilitate use of deep pretrained language models.