Source-linked AI summary
Understanding the Role of Training Regimes in Continual Learning
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, Hassan Ghasemzadeh
TL;DR
Catastrophic forgetting limits sequential task learning because models often lack stability for preserving earlier knowledge. This paper studies training regimes through loss-landscape geometry and finds that techniques producing wider minima can reduce forgetting and outperform some regularization- and memory-based approaches.
Problem
Catastrophic forgetting degrades performance on earlier tasks when previous-task data are unavailable, while limited prior work has examined how training regimes affect forgetting.
Method
The paper analyzes forgetting from an optimization and loss-landscape perspective, studying dropout, learning-rate schedules, and batch-size changes as ways to affect local-minimum geometry.
Results
The derived measure of loss geometry has high correlation with forgetting, and the studied techniques empirically reduce forgetting and outperform some regularization- and memory-based approaches.
Takeaways & Limitations
Simple training techniques can improve neural-network stability in continual learning and can be incorporated alongside alternative approaches.
Takeaways & Limitations
The analyzed techniques may have confounding factors beyond loss-surface wideness, motivating further theoretical research.
Abstract
from arXiv · showhide
Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. From the perspective of the well established plasticity-stability dilemma, neural networks tend to be overly plastic, lacking the stability necessary to prevent the forgetting of previous knowledge, which means that as learning progresses, networks tend to forget previously seen tasks. This phenomenon coined in the continual learning literature, has attracted much attention lately, and several families of approaches have been proposed with different degrees of success. However, there has been limited prior work extensively analyzing the impact that different training regimes -- learning rate, batch size, regularization method-- can have on forgetting. In this work, we depart from the typical approach of altering the learning algorithm to improve stability. Instead, we hypothesize that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting. In particular, we study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima and consequently, on helping it not to forget catastrophically. Our study provides practical insights to improve stability via simple yet effective techniques that outperform alternative baselines.
1 Introduction
The paper studies catastrophic forgetting in continual learning and asks whether simple training-regime choices can improve stability without changing the learning algorithm. It focuses on how dropout, learning-rate schedules, and batch size affect local-minimum geometry and forgetting.
- Continual learning trains a neural network on sequential tasks without access to data from earlier tasks, so performance on older tasks can degrade.
- Catastrophic forgetting reflects a stability-plasticity dilemma: networks need plasticity for new knowledge but stability to retain previous knowledge.
- Prior work has proposed many approaches to catastrophic forgetting, but systematic analysis of common training regimes remains limited.
- The paper hypothesizes that the geometry of task-specific local minima correlates with forgetting and examines training regimes that widen those minima.
- Dropout, learning-rate decay, and shrinking batch size are presented as simple techniques that can affect stability and outperform more complex continual-learning algorithms when jointly tuned.
2 Related work
Related work groups continual-learning methods into replay, explicit regularization, and parameter-isolation approaches. The paper positions its training-regime analysis alongside these established families.
- Continual-learning methods are categorized into replay-based, explicit-regularization, and parameter-isolation groups.
- Replay methods store information from earlier tasks, including class-centered samples, parameter gradients, or reservoir-sampled examples.
- Explicit-regularization methods constrain parameter updates to preserve previous knowledge, with EWC using Fisher information to estimate weight importance.
- Parameter-isolation methods dedicate different parameter subsets to tasks, often using gating or pruning to control plasticity and improve stability.
- The paper focuses specifically on catastrophic forgetting, while acknowledging continual learning’s connections to meta-learning, few-shot learning, transfer learning, and task detection.
3 Forgetting during training
The paper models forgetting through the geometry of task minima and parameter displacement. Its analysis predicts, and experiments support, lower forgetting for wider minima and smaller task-to-task movement.
- The analysis considers sequential supervised tasks in parameter space, measuring forgetting after learning the second task without access to earlier-task examples.
- Under a second-order Taylor approximation, forgetting is bounded by the first task’s maximum Hessian eigenvalue multiplied by the squared relocation norm.
- For a fixed parameter displacement, wider first-task minima with lower curvature produce less forgetting, as illustrated and empirically observed on Rotated MNIST and Permuted MNIST.
- In high-dimensional settings, the full Hessian spectrum can matter more than its largest eigenvalue, especially when many eigenvalues have low magnitude.
- Controlling curvature alone may not minimize forgetting because the bound also depends on the norm of parameter displacement.
- The paper hypothesizes that wider convergent minima correlate with less sequential forgetting and that wider minima may also be closer to one another.
- Across training regimes with varied dropout, learning rates, decay, batch sizes, and initialization, the derived measure highly correlates with forgetting while task-2 accuracy remains roughly 90%.
4 Training Regimes: techniques affecting stability and forgetting
The paper analyzes how learning rate, batch size, dropout, and weight decay shape loss-minimum geometry and parameter trajectories relevant to continual-learning stability. It argues that regimes favoring wider minima and controlled parameter change can reduce forgetting.
- Optimization setting: learning rate, batch size, and optimizer: Learning rate and batch size affect both endpoint curvature and the optimization trajectory, linking training-regime choices to loss-minimum geometry.A high learning rate or small batch size can limit the maximum spectral norm along the SGD path.
- Optimization setting: learning rate, batch size, and optimizer: A high learning rate can favor wider minima but also increase weight updates, making it a double-edged choice for continual optimization.The proposed schedule starts with a high initial learning rate, then decreases it across tasks while shrinking batch size.
- Optimization setting: learning rate, batch size, and optimizer: The stable regime combines dropout, a large initial learning rate with exponential decay, and a small batch size to affect neural-network stability.The paper contrasts this regime with plastic training that does not use these techniques.
- Regularization: dropout and weight decay: Dropout encourages flatter minima by reducing curvature-related terms, whereas the paper argues that L2 regularization can work worse in continual learning.The stated reasons include dropout’s data dependence and interactions between L2 regularization and batch normalization.
- Regularization: dropout and weight decay: Prior continual-learning studies report benefits from dropout relative to weight decay, while the paper extends dropout analysis through regularization and loss-landscape perspectives.The cited empirical study associates dropout with better continual-learning performance and weight decay with increased forgetting and lower final-model performance.
5 Experiments and results
The experiments compare stable and plastic training regimes across continual-learning benchmarks and then evaluate the stable regime against established algorithms. Stable training is associated with lower curvature, smaller parameter changes, and reduced catastrophic forgetting, while outperforming other methods in the reported comparisons.
- 5.2 Stable versus Plastic networks: Stable SGD uses dropout, a decayed large initial learning rate, and a small batch size, whereas naive SGD omits these techniques.The experiments distinguish Stable and Plastic (Naive) regimes when studying curvature and forgetting.
- Experimental setup: The experiments use Permuted MNIST, Rotated MNIST, and Split CIFAR-100, with five-run averages and standard deviations reported for the experimental results.The datasets include both common and more challenging continual-learning benchmarks.
- 5.3 Comparison with other methods: The evaluation uses average accuracy and average forgetting, with forgetting defined as the decrease from each task’s peak accuracy to its final accuracy.These metrics are reported after sequential training and are illustrated over time in Figure 4.
- 5.2 Stable versus Plastic networks: Figure 3 shows stable training reducing catastrophic forgetting alongside lower curvature and smaller parameter changes than the plastic regime.Its rows track validation accuracy, the twenty sharpest curvature eigenvalues, and ℓ2 parameter distances across tasks.
- 5.3 Comparison with other methods: The scaled experiment increases the sequence from 5 to 20 tasks and evaluates Split CIFAR-100 with restricted episodic memory for A-GEM and ER-Reservoir.The study reports average accuracy and forgetting after sequential learning.
- 5.3 Comparison with other methods: The stable network is reported as a strong competitor to various continual-learning algorithms and as consistently better than other algorithms in the authors’ comparison.The proposed techniques are described as orthogonal to other methods and incorporable into them.
6 Conclusion
The paper concludes that loss-landscape and optimization analysis identifies practical training techniques associated with reduced forgetting and greater stability. It also acknowledges confounding factors and calls for further theory, while positioning the techniques as complementary to existing approaches.
- 6 Conclusion: The paper identifies training regimes and techniques that contribute to forgetting through a loss-landscape and optimization perspective.The resulting techniques are presented as practical ways to reduce forgetting and increase stability in maintaining previous knowledge.
- 6 Conclusion: The techniques were analyzed through the wideness of loss surfaces around local minima, but other confounding factors may also reduce catastrophic forgetting.The authors call for more theoretical research on the stability-plasticity trade-off.
- 6 Conclusion: The simple techniques were empirically more effective than some regularization-based and memory-based approaches while remaining orthogonal to them.The paper leaves incorporating these recommendations into other methods as future work.
Broader Impact
Reducing catastrophic forgetting can support continual learning applications, but preserving prior knowledge may conflict with requests to forget identifiable data.
- Continual learning can reduce retraining costs and the need to retain all previous data.
- Reducing catastrophic forgetting may risk privacy when users explicitly want particular data forgotten.
- Future continual learning agents should distinguish identifiable data that must be forgotten from general knowledge that should be maintained.
Supplementary material
The supplementary material extends the paper’s analyses, documents experimental details, and adds experiments applying stability techniques to other continual learning methods.
- Appendix A: Appendix A extends the continual-learning forgetting analysis presented in Section 3.
- Appendix B: Appendix B provides experimental-setup and hyper-parameter details, alongside code and reproduction scripts.
- Appendix C: Appendix C adds Figure 1 accuracy results, an expanded Table 1, parameter-norm results, and first-task accuracy comparisons.
- Appendix C: Appendix C.5 applies the stability techniques to A-GEM, EWC, and ER-Reservoir.
A Further analysis
The analysis links catastrophic forgetting to local-minimum geometry and parameter displacement: wider first-task minima reduce forgetting, while wider second-task minima reduce movement between task solutions.
- Forgetting analysis: Forgetting is defined through the change in the first task’s loss after sequentially training the second task.
- First-task curvature: Controlling the Hessian spectrum alone cannot ensure minimal forgetting because the displacement norm must also be controlled.
- Second-task curvature: Decreasing λmax 2 lowers the bound on ∥∆w∥, allowing a near-optimum second-task solution to be reached closer to the first-task solution.
- Training regime: Learning-rate decay combines an initially large learning rate with exploration, increasing the chance of finding wider minima and reducing forgetting.
- First-task curvature: For a fixed ∆w, wider curvature of the first task corresponds to less forgetting.
- Second-task curvature: Wider curvature of the second task is associated with smaller ∥∆w∥ during optimization.
B Experimental Setup Details
The experiments use established continual-learning benchmarks, comparable architectures and metrics, reproducible code, and setups ranging from five-task regime studies to twenty-task evaluations.
- Experimental design: The study uses Permuted MNIST, Rotated MNIST, and Split CIFAR-100, including a challenging 20-task CIFAR-100 setting.
- Experimental design: Architectures match prior studies: two-layer MLPs are used for five-task MNIST experiments, while a ResNet is used for 20-task experiments.
- Evaluation: Accuracy and forgetting are reported as the principal evaluation metrics.
- Experimental regimes: Experiment 1 studies training-regime effects with five tasks and five training epochs per task.
- Experimental regimes: Experiment 2 evaluates a stable training regime with 20 tasks and one training epoch per task.
- Results: The stable network suffers much less catastrophic forgetting than the plastic network, with a relatively small drop in current-task accuracy.
C.2 Extended version of Table 1 in Section 3: disentangling the stability of different tasks
The experiment separates the stability of the first and second tasks to assess their respective roles in reducing forgetting on the first task. Stable training uses dropout, learning-rate decay, and a small batch size, while plastic training omits these stabilizing choices.
- The study evaluates forgetting across four combinations of stable or plastic training for the first and second tasks.The comparison isolates whether stability in the current or subsequent task matters more for retaining the first task.
- Stable networks use dropout probability 0.25, learning-rate decay 0.4, and batch size 16; plastic networks use neither regularization nor decay and batch size 256.
- Stable/Stable yields the least forgetting, whereas Plastic/Plastic yields the most across five random-seed runs.The reported values are means with standard deviations over five runs.
- The large difference between Stable/Plastic and Plastic/Stable suggests that widening the current task’s minima matters more than widening subsequent tasks’ minima for reducing forgetting.
C.3 Additional result for experiment 1: comparing norms of weights
Additional experiments examine how stable training changes weight norms and task retention, and whether the same regime benefits existing continual-learning methods. The reported results associate stable training with smaller optimal-solution norms, better first-task memory, and higher average accuracy for several methods.
- Stable networks have smaller norms for the optimal solutions after each task than plastic networks.The paper suggests smaller norms may reduce displacement during sequential task optimizations.
- The stable network remembers the first task much better than other methods during validation across 20 sequential tasks.Figure 8 tracks the first task’s validation accuracy throughout the learning experience.
- The paper asks whether other methods benefit from stable training and reports that the answer is yes.
- Stabilizing training improves average accuracy by 12.9% for EWC, 15.8% for A-GEM, and 9% for ER-Reservoir.
- Stable A-GEM and stable ER-Reservoir outperform Stable SGD because they retain episodic memory, while Stable SGD uses no memory.The comparison concerns average-accuracy evolution during the learning experience.