PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming catastrophic forgetting in neural networks

Brandon Shuen Yi Loke, Filippo Quadri, Gabriel Vivanco, Maximilian Casagrande, Saúl Fenollosa

arXiv:2507.10485v1cs.LGcs.IR

TL;DR

Catastrophic forgetting limits continual learning because neural networks can lose earlier knowledge while learning successive tasks. This study reproduces and extends EWC evaluation in supervised PermutedMNIST and RotatedMNIST experiments, comparing it with L2 regularization and unregularized SGD. EWC generally reduces forgetting, although it can slightly compromise learning efficiency on new tasks.

  • Problem

    Catastrophic forgetting causes neural networks to lose performance on previously learned tasks when acquiring new ones, limiting continual learning.

  • Method

    The study evaluates EWC in supervised sequential learning on PermutedMNIST and RotatedMNIST, comparing it with L2 regularization and unregularized SGD.

  • Results

    EWC better preserves earlier-task performance than naive training, while sometimes producing slightly lower learning performance on subsequent tasks.

  • Takeaways & Limitations

    EWC is an effective regularization method for mitigating catastrophic forgetting and supports continual learning across sequential tasks.

  • Takeaways & Limitations

    The implementation did not fully match the original paper’s reported accuracy, possibly because of hyperparameter, architecture, or computational differences.

Abstract

from arXiv · show

Catastrophic forgetting is the primary challenge that hinders continual learning, which refers to a neural network ability to sequentially learn multiple tasks while retaining previously acquired knowledge. Elastic Weight Consolidation, a regularization-based approach inspired by synaptic consolidation in biological neural systems, has been used to overcome this problem. In this study prior research is replicated and extended by evaluating EWC in supervised learning settings using the PermutedMNIST and RotatedMNIST benchmarks. Through systematic comparisons with L2 regularization and stochastic gradient descent (SGD) without regularization, we analyze how different approaches balance knowledge retention and adaptability. Our results confirm what was shown in previous research, showing that EWC significantly reduces forgetting compared to naive training while slightly compromising learning efficiency on new tasks. Moreover, we investigate the impact of dropout regularization and varying hyperparameters, offering insights into the generalization of EWC across diverse learning scenarios. These results underscore EWC's potential as a viable solution for lifelong learning in neural networks.

I. INTRODUCTION

The study addresses catastrophic forgetting in continual learning by extending EWC evaluation to supervised PermutedMNIST and RotatedMNIST settings. It reproduces prior comparisons while examining regularization, dropout, and training choices.

  • I. INTRODUCTION: Catastrophic forgetting occurs when learning new tasks rewrites earlier task weights and degrades previously acquired performance.The paper presents overcoming this obstacle as essential for adaptable systems in dynamic environments.
  • I. INTRODUCTION: EWC selectively reduces the plasticity of weights important for prior tasks, limiting disruption when learning subsequent tasks.The method is inspired by neurobiological synaptic consolidation and represents prior-task parameters as permissible ranges.
  • I. INTRODUCTION: The project extends prior EWC work by reproducing and comparing results on PermutedMNIST and RotatedMNIST in supervised learning.The comparison includes EWC, naive L2 regularization, and no regularization.
  • I. INTRODUCTION: The study evaluates EWC’s effectiveness in mitigating catastrophic forgetting and its potential as a general solution for sequential supervised task learning.Additional experiments examine dropout and varying training choices alongside the main reproduction.
  • I. INTRODUCTION: The implementation uses a fully connected network with two 400-neuron ReLU hidden layers, cross-entropy training, batch normalization, and early stopping.When original parameters were unclear, cross-validation selected values minimizing validation error; training duration varied from 20 to 100 epochs per task.

B. Continual Learning Tasks

The paper evaluates continual learning on sequential MNIST tasks formed by fixed pixel permutations or fixed rotations. Models train on each task in order and are tested on earlier tasks to measure forgetting.

  • B. Continual Learning Tasks: Two catastrophic-forgetting benchmarks are considered: PermutedMNIST and RotatedMNIST.Both transform the same MNIST images into task-specific inputs.
  • B. Continual Learning Tasks: PermutedMNIST applies a unique fixed pixel permutation π_t to each task’s images.Each transformed sample x′ is obtained from x according to the task permutation.
  • B. Continual Learning Tasks: RotatedMNIST applies a fixed angle α_t to the digits for each task.Each image x is replaced by R_αt(x).
  • B. Continual Learning Tasks: Tasks are presented in a fixed sequence, with the model trained on each task in turn.Testing on previously learned tasks assesses performance retention after subsequent training.

C. Regularization Approaches

The study compares sequential SGD without regularization, L2 regularization, and EWC. L2 penalizes parameter changes uniformly, whereas EWC scales penalties using parameter importance estimated by Fisher information.

  • C. Regularization Approaches: Naive SGD trains sequentially without preserving older-task knowledge, which typically produces substantial catastrophic forgetting.It serves as the unregularized baseline.
  • C. Regularization Approaches: The dataset visualizations contrast original MNIST digits with fixed pixel permutations and 50° rotations.These transformations define the two benchmark task families used in the comparisons.
  • C. Regularization Approaches: L2 regularization mitigates forgetting by applying a penalty to parameter changes from parameters optimized for an earlier task.The coefficient λ_L2 controls the strength of this constraint.
  • C. Regularization Approaches: EWC improves on L2 by weighting each parameter’s penalty according to the diagonally approximated Fisher information matrix.The Fisher matrix is computed after training the earlier task.
  • C. Regularization Approaches: Parameters with higher F_A,i receive stronger penalties, preserving parameters considered important for task A.The formulation generalizes by accumulating penalties across previously learned tasks.

D. Training and Cross Validation

Training uses mini-batch SGD with momentum, while cross-validation selects optimization and regularization settings. Experiments quantify forgetting through accuracy on previously learned tasks across sequential PermutedMNIST and RotatedMNIST training.

  • D. Training and Cross Validation: Mini-batch SGD with momentum trains the models, and validation accuracy guides selection of batch size, momentum, and regularization coefficients.The tested batch sizes were 32, 64, and 128; regularization ranges differed between L2 and EWC.
  • D. Training and Cross Validation: The experiments cover hyperparameter cross-validation, sequential PermutedMNIST learning, and sequential RotatedMNIST learning.Their analyses quantify catastrophic forgetting and knowledge preservation across regularization schemes.
  • D. Training and Cross Validation: Approximately 95% validation accuracy was achieved by unregularized SGD and L2 SGD with batch size 32 and momentum 0.6.For L2, the best observed coefficient was λ_L2 ≈ 0.01.
  • D. Training and Cross Validation: Excessively high λ_EWC values sometimes caused loss divergence during training.This establishes a practical boundary on EWC coefficient selection.
  • D. Training and Cross Validation: The final replication used batch size 64, momentum 0.6 for SGD and L2, momentum 0 for EWC, λ_L2 = 0.01, and λ_EWC between 10000 and 20000.Additional SGD-Dropout tuning selected a learning rate of 1 × 10^-3 and hidden-layer width 800, though these values were not necessarily globally optimal.

B. Permuted MNIST

The PermutedMNIST experiment sequentially trains SGD, L2-regularized SGD, and EWC on distinct pixel permutations, reevaluating earlier tasks to measure forgetting. It forms part of a broader comparison across PermutedMNIST and RotatedMNIST, including taskwise and average performance.

  • Distinct fixed pixel permutations define the sequential PermutedMNIST tasks.
  • Models are trained on each new permutation, then tested again on previously learned permutations to quantify forgetting.
  • The comparison includes plain SGD, L2-penalized SGD, and EWC.
  • Task-specific accuracy curves and overall average accuracy assess each method’s retention across tasks.
  • The study also evaluates sequential learning on rotated MNIST and includes hyperparameter optimization through cross-validation.

V. DISCUSSION

The discussion finds that EWC generally preserves earlier-task performance better than SGD and L2, while sometimes reducing learning performance on later tasks. Across three- and ten-task settings, EWC reduces forgetting, but results depend on task difficulty and training configuration.

  • EWC generally prevents catastrophic forgetting better than the other regularization schemes, particularly for Task A.
  • Improved retention on Task A can coincide with slightly lower learning performance on subsequent tasks.
  • Permuted tasks show less forgetting than widely separated rotated tasks, while rotations of 0°, 10°, and 20° perform nearly like PermutedMNIST.
  • In the Mixed Task case, EWC performs better than SGD and L2, indicating stronger cross-task generalization and reduced forgetting.
  • With ten fixed epochs per task, EWC shows less catastrophic forgetting than SGD, especially on the first task.
  • The validation-based early-stopping setup often trained tasks for only two epochs because forgetting triggered stopping, limiting its realism for broader scenarios.

APPENDIX

The appendix documents figure-based comparisons of regularization methods on three-task and ten-task benchmarks, alongside utility scripts supporting dataset generation, EWC and L2 implementation, training, and visualization.

  • Figure 4 compares different regularization methods on three-task benchmarks.
  • Figure 5 analyzes performance on the ten-task benchmark.
  • Utility scripts support dataset generation, model training, evaluation, and visualization.
  • The utils folder contains separate implementations for EWC, L2 regularization, training functions, visualization, and dataset utilities.

B. Notebooks •

The notebooks provide reproducibility infrastructure for optimizing model parameters and building and training the fully connected neural network. The section also explains how to reproduce subfigures in Figures 4 and 5.

  • The cross-validation notebook is used to optimize model parameters.
  • The neural-network notebook builds and trains the fully connected neural network used in the project.
  • The reproduction instructions explain how to recreate subfigures in Figures 4 and 5.

C. Figure 4

Figure 4 uses fully connected networks across Permuted MNIST, Rotated MNIST, mixed task sequences, and a dropout variant, while Figure 5 varies training patience settings.

  • Figure 4 subfigures were generated using the first part of the NN_notebook.
  • Figures 4a–4c use FCN networks with Permuted MNIST, rotated tasks, or mixed rotated and permuted task sequences.Figure 4b uses 0°, 40°, and 90° rotations; Figure 4c combines Rotated 0°, Permuted, and Rotated 90° tasks.
  • Figure 4d evaluates an FCN_Dropout network using Permuted MNIST.
  • Figure 4e uses an FCN network with Rotated MNIST tasks at 0°, 10°, and 20°.
  • Figures 5a–5d were generated using the second notebook section with train_with_avg_perf patience settings ranging from 5 to 100 epochs.Figures 5a and 5b use patience values of 15 and 10 epochs without early stopping; Figures 5c and 5d use 5 and 100 epochs.

TASK DISTRIBUTION

The work was primarily collaborative, with specific responsibilities assigned for L2 and SGD, EWC, cross-validation, and report preparation.

  • Most of the work was carried out collaboratively by the entire group.
  • Saúl Fenollosa and Maximilian Casagrande primarily handled L2 and SGD.
  • Filippo Quadri and Gabriel Vivanco primarily handled EWC.
  • Brandon Shuen Yi Loke primarily handled cross-validation, while all group members contributed to the report.
Loading 2507.10485v1…