PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming Catastrophic Forgetting by Incremental Moment Matching

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, Byoung-Tak Zhang

arXiv:1703.08475v3cs.LGcs.AI

TL;DR

Catastrophic forgetting causes neural networks to lose earlier-task information after learning new tasks. The paper proposes IMM, which incrementally matches task-posterior moments and uses transfer techniques to smooth the parameter search space. Across varied datasets, IMM balances old- and new-task information and reports competitive or superior performance, while its assumptions can be strained by heterogeneous tasks and transfer.

  • Problem

    Neural networks trained on new tasks can forget information from previous tasks, motivating methods for continual learning.

  • Method

    IMM merges Gaussian approximations of task-specific posterior distributions through mean- or mode-based moment matching, complemented by transfer techniques.

  • Results

    Mode-IMM outperforms mean-IMM and comparative models across various datasets, while IMM with LwF exceeds previous state-of-the-art performance in the reported results.

  • Takeaways & Limitations

    IMM can balance old- and new-task information and adjust task importance in real time by changing the mixing weight αt.

  • Takeaways & Limitations

    IMM assumptions do not always hold for heterogeneous tasks, and transfer techniques can shift mean-IMM’s optimal α away from 1/k.

Abstract

from arXiv · show

Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task. Here, we propose a method, i.e. incremental moment matching (IMM), to resolve this problem. IMM incrementally matches the moment of the posterior distribution of the neural network which is trained on the first and the second task, respectively. To make the search space of posterior parameter smooth, the IMM procedure is complemented by various transfer learning techniques including weight transfer, L2-norm of the old and the new parameter, and a variant of dropout with the old parameter. We analyze our approach on a variety of datasets including the MNIST, CIFAR-10, Caltech-UCSD-Birds, and Lifelog datasets. The experimental results show that IMM achieves state-of-the-art performance by balancing the information between an old and a new network.

1 Introduction

Catastrophic forgetting causes neural networks to lose earlier-task information after learning a new task. IMM addresses this by matching posterior moments and using transfer techniques to create a smoother parameter-loss search space.

  • Catastrophic forgetting occurs when neural networks trained with stochastic gradient descent forget previous-task information after learning a new task.
  • IMM approximates task-specific Gaussian posteriors with one Gaussian posterior for the combined task and introduces mean-IMM and mode-IMM for merging them.Mean-IMM averages network parameters; mode-IMM uses a Laplacian approximation to find a mode of the Gaussian-posterior mixture.
  • Mean-IMM averages parameters, whereas mode-IMM seeks a maximum of the mixture of Gaussian posteriors.
  • IMM requires a smooth, convex-like loss path between posterior means, so the procedure applies weight transfer, parameter L2 regularization, and a dropout variant.

2 Previous Works on Catastrophic Forgetting

Prior approaches to catastrophic forgetting use ensembles, distributed information storage, or regularization, each with stated scalability, preservation, or assumption constraints. Sequential Bayesian regularization methods can work when their assumptions fit the domain.

  • Ensemble methods share representations across task-specific networks but incur increasing inference complexity as new tasks add networks.
  • Dropout, maxout, and neural modules distribute task information across network capacity, but many such methods fail to preserve old-task performance under extreme environmental changes.
  • Learning without Forgetting uses new-task inputs with old-network outputs as pseudo-labels while jointly optimizing pseudo-old-task and real-new-task data.
  • EWC assumes a diagonal posterior covariance with no node correlations, an assumption described as fragile despite good performance in some domains.
  • The paper distinguishes IMM from prior Bayesian moment-matching work by targeting continual learning with significantly different task statistics.

3 Incremental Moment Matching

IMM incrementally approximates sequential task posteriors with a Gaussian and merges them through mean- or mode-based moment matching. Its mode-based variant incorporates covariance information through Fisher-based approximations and Laplacian mode finding.

  • IMM approximates the posterior for K sequential tasks with a Gaussian q1:K parameterized by combined mean µ1:K and covariance Σ1:K.The true posterior p1:K is defined over the neural-network parameter vector θ and task datasets.
  • The two IMM algorithms generate different Gaussian moments by optimizing different objectives on the same dataset.
  • Mean-based Incremental Moment Matching (mean-IMM): Mean-IMM averages layerwise parameters using mixing ratios αk whose sum is one and minimizes a weighted KL-divergence objective.
  • Mean-based Incremental Moment Matching (mean-IMM): Mean-IMM needs only the sequence of task-specific means because its optimal mean does not require covariance information.The paper frames its contribution as identifying when and how mean-IMM applies to modern deep networks and improves with transfer techniques.
  • Mode-based Incremental Moment Matching (mode-IMM): Mode-IMM uses covariance information and Laplacian approximation to estimate a mode of the mixture of Gaussian posteriors.Its candidate modes lie on a (K−1)-dimensional hypersurface formed by weighted combinations of the component means.
  • Mode-based Incremental Moment Matching (mode-IMM): Mode-IMM adds εI before matrix inversion, assumes diagonal covariance to reduce storage from O(D^2) to O(D), and estimates covariance using inverse Fisher information.The Fisher approximation treats squared parameter gradients as indicators of precision, the inverse of variance.

4 Transfer Techniques for Incremental Moment Matching

The section presents weight-transfer, L2-transfer, and drop-transfer as techniques that smooth the loss landscape for IMM, then evaluates their effects experimentally.

  • Weight-Transfer: Weight-transfer initializes each new task’s parameters from the previous task’s parameters, and the experiments use it by default.The paper reports that weight-transfer was critical to continual-learning performance.
  • Weight-Transfer: Weight-transfer is intended to create a smooth loss surface between task solutions, making an interpolated point potentially optimal for both tasks.The approach is motivated by the existence of low-barrier paths from initialization to solutions.
  • Weight-Transfer: 18,000 CIFAR-10 instances were split into three 6,000-instance subsets for sequential CNN training, with θ2 and θ3 initialized by weight-transfer.Loss and accuracy were evaluated along parameter-space paths connecting the three trained networks.
  • Weight-Transfer: The mean-IMM point, 1/3(θ1 + θ2 + θ3), performed better than θ1, θ2, or θ3; without weight-transfer, the convex-like shape disappeared.The no-transfer case exhibited a high cost barrier between θ1 and θ2.
  • L2-Transfer: L2-transfer adds a distance penalty between successive task parameters, using small λ to smooth the loss surface before IMM merges the parameters.Unlike conventional uses with large λ, this procedure does not primarily minimize the distance between successive parameters.
  • Drop-Transfer: Drop-transfer uses the previous task’s parameter µk−1 as dropout’s zero point, while preserving the current parameter as the expectation of the transformed weight.The transformed weight is defined piecewise according to whether a node is turned off.
  • Experiments: Dropout improved sampled-point and averaged-point accuracy in the MNIST loss-space experiment from 0.450 (± 0.084) to 0.950 (± 0.009) and from 0.757 (± 0.065) to 0.974 (± 0.003).The experiment investigated whether dropout regularization smooths the search space.

5 Experimental Results

IMM is evaluated across sequential-task benchmarks, where transfer techniques improve performance and enable competitive results on challenging continual-learning settings.

  • The evaluation covers disjoint MNIST, shuffled MNIST, ImageNet2CUB, and Lifelog experiments, using both untuned and tuned hyperparameter settings.The Lifelog dataset contains continuously changing class distributions and newly appearing classes.
  • Transfer Techniques: Both L2-transfer and drop-transfer boost IMM performance on disjoint MNIST, shift optimal α above 1/2, and drop-transfer produces a smoother accuracy curve.The comparison is shown for IMM with various transfer techniques.
  • Disjoint MNIST Experiment: The disjoint MNIST task requires joint 10-class categorization, making inference harder than separate 5-class task evaluation.The model must infer whether each instance belongs to the first or second task.
  • Disjoint MNIST Experiment: Less than 50% accuracy is obtained by usual SGD-based optimizers, while mode-IMM is robust to α and mean-IMM performs best with α greater than 1/2.The output-layer biases for the old task are pushed toward large negative values under the experimental setting.
  • Shuffled MNIST Experiment: 96.86 (± 0.21) is achieved by drop-transfer alone, while L2-transfer (λ = 1e-4) + drop-transfer (p = 0.4) achieves 97.61 (± 0.15), both competitive with EWC around 97.0.These results are reported for shuffled MNIST.
  • Lifelog Dataset: Mean-IMM and mode-IMM are competitive with the dual-memory architecture on Lifelog despite using a single network.Results are reported across location, sub-location, and activity classes for subjects A, B, and C.

6 Discussion

The discussion examines how transfer techniques affect IMM's weighting and robustness, and identifies limitations when task statistics or posterior assumptions differ.

  • Transfer techniques can shift the optimal mean-IMM α above 1/k because they violate the assumption that each network is mainly affected by its own task.Mode-IMM is more robust than mean-IMM when transfer techniques are applied.
  • Figure 4 links applied transfer techniques with changes in accuracy curves and shifts in the optimal α for mean-IMM and mode-IMM.
  • Using only the diagonal covariance in Bayesian neural networks is described as too naïve in general, motivating more complex priors such as matrix Gaussians.The discussion connects this limitation to EWC's failure on disjoint MNIST.
  • IMM can adjust task importance in real time by changing α_t, distinguishing it from regularization approaches such as LwF and EWC.For mean-IMM, the combined solution weights task-specific parameters through α_t.

7 Conclusion

The paper develops mean-IMM and mode-IMM for continual learning, combining posterior moment matching with transfer techniques to balance old and new task information. Mode-IMM outperforms mean-IMM and comparative models across datasets.

  • Mean-IMM applies posterior moment matching to continual learning and balances information between old and new networks.
  • Mode-IMM extends mean-IMM by interpreting parameter merging as mode-finding in a mixture of Gaussian posteriors.
  • Mode-IMM outperforms mean-IMM and comparative models across various datasets.
  • Drop-transfer alone performs well and is similar to EWC without dropout where EWC rarely forgets.
  • Applying transfer techniques often boosts performance and helps make the Gaussian-posterior assumption more reasonable.

APPENDIX A. Modes in the Mixture of Gaussian

The appendix characterizes modes of Gaussian mixtures and motivates mode-IMM through a precision-weighted parameter combination. It also describes diagonal covariance approximations and related Bayesian continual-learning methods.

  • Critical points of a two-component Gaussian mixture lie on a curve parameterized by the mixing weight α, with 0 < α < 1.
  • The optimal mode lies among precision-weighted combinations of the component means, motivating mode-IMM.
  • IMM uses diagonal covariance matrices, reducing covariance parameters from O(D2) to O(D).
  • For a two-component Gaussian mixture in K dimensions, the number of modes can be at most K + 1, making exhaustive mode-finding difficult in high dimensions.
  • EWC approximates posterior precision with the diagonal Fisher matrix, whereas its third-task penalty uses both earlier networks as a heuristic.
  • Mean-IMM and mode-IMM are combined with weight-transfer, L2-transfer, and drop-transfer as continual-learning algorithms.

D.1 Disjoint MNIST Experiment

In the disjoint MNIST experiment, IMM is evaluated under tuned and untuned hyperparameter settings with transfer-based implementations. Its strongest configuration exceeds 94% accuracy, while comparative models remain below 90%.

  • The untuned setting uses natural hyperparameters, including αk = 1/K for mean-IMM and λ = 1 for EWC.
  • The tuned setting uses heuristic hand-tuned hyperparameters rather than each algorithm’s natural parameter choice.
  • Hyperparameters λ and αK balance information between old and new tasks, with endpoint values reproducing either the first or last task network.
  • The MLP uses a [784-800-800-10] architecture with ReLU activations and vanilla SGD; the full IMM model uses weight-transfer.
  • 94%+: weight-transfer + L2-transfer + drop-transfer + mode-IMM performs best, while comparative models fail to exceed 90%.Accuracy values were averaged over 10 experiments.

D.2 Shuffled MNIST Experiment

The shuffled MNIST experiment tests IMM on three sequential tasks and examines how transfer techniques affect the loss landscape. IMM performs similarly to EWC where EWC works well, while transfer references that minimize parameter distance can worsen performance.

  • IMM is evaluated on three sequential shuffled-MNIST tasks, tuning α3 while setting α1 and α2 equally.
  • IMM performs similarly to EWC in the shuffled-MNIST setting where EWC performs well.
  • Weight-transfer, L2-transfer, and drop-transfer use the preceding model µk−1 as the training reference for µk.
  • Transfer references based on previous-task averages or modes worsen performance in the shuffled-MNIST experiment.
  • IMM transfer techniques target a smooth, convex-like loss space between successive models rather than merely minimizing their parameter distance.

D.3 ImageNet to Other Image Datasets

For tasks with different class output layers, IMM uses last-layer fine-tuning, while mode-IMM requires additional Fisher-matrix handling. In ImageNet2CUB, re-scaled mode-IMM performs similarly to mean-IMM, whereas naïve mode-IMM performs poorly.

  • Different class output layers: IMM adds last-layer fine-tuning to match the new task's output-layer moments when tasks use different class output layers.The fine-tuning model updates only the last layer for each new task, avoiding first-task performance loss but often learning insufficiently for new tasks.
  • ImageNet2CUB results: LwF is included in ImageNet2CUB moment matching because it is known to perform well when the new dataset is small relative to the old dataset.In this experiment, moments of the last-layer fine-tuning model and the LwF model are matched.
  • Different class output layers: Naïve mode-IMM uses a zero Fisher matrix for the first network's second-last layer, effectively retaining that layer from the second network.This assumption produces poor performance in the comparison shown in Figure 5.
  • ImageNet2CUB results: Re-scaled mode-IMM performs similarly to mean-IMM in the ImageNet2CUB experiment.The re-scaling uses the defined ˆα parameterization.

D.4 Lifelog Dataset

The Lifelog experiments use temporally ordered egocentric video from three participants and merge networks trained at successive points. IMM achieves competitive results with an ensemble approach without additional inference or learning cost.

  • Dataset: The Lifelog dataset contains 660,000 seconds of egocentric video recorded over 46 days from three participants.Each participant has 10 days of training data and 4 days of test data ordered in time.
  • Experimental setup: Networks are created on the 3rd, 7th, and 10th days, then merged with previously trained networks using IMM.The initial network is AlexNet pretrained on ImageNet.
  • Results: IMM achieves competitive results with an ensemble network without additional inference or learning cost.The reported Lifelog results evaluate classification accuracies across classes and subjects.
Loading 1703.08475v3…