Source-linked AI summary
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
Hippolyt Ritter, Aleksandar Botev, David Barber
TL;DR
Catastrophic forgetting degrades performance on earlier tasks as neural networks learn new ones, and simple parameter initialization does not reliably prevent it. The paper introduces a Bayesian online learning method that recursively applies Gaussian Laplace approximations with block-diagonal Kronecker factored curvature. It achieves over 90% average test accuracy across 50 permuted MNIST tasks and substantially improves over related diagonal methods, although the approximation still requires some regularization.
Problem
Catastrophic forgetting reduces performance on older tasks during sequential learning, while simple initialization from previous optima does not reliably preserve earlier-task performance.
Method
The method recursively approximates each task’s posterior with a Gaussian and uses block-diagonal Kronecker factored curvature to scale the Laplace approximation.
Results
Over 90% average test accuracy is achieved across 50 permuted MNIST tasks, nearly matching joint training and outperforming simpler diagonal methods.
Takeaways & Limitations
Accounting for interactions between parameters improves performance over diagonal approximation methods, particularly across long sequences of datasets.
Takeaways & Limitations
Some regularization remains necessary, suggesting that the Kronecker factored approximation still overestimates posterior variance.
Abstract
from arXiv · showhide
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.
1 Introduction
Catastrophic forgetting causes performance on earlier tasks to decline as neural networks learn new ones. The paper combines Bayesian online learning with a Kronecker factored Laplace approximation to address this sequential-learning problem.
- Catastrophic forgetting is the decline in performance on older tasks when a model learns new tasks.
- Initializing new-task parameters from an old optimum does not reliably preserve performance on previous datasets.
- Bayesian online learning uses the posterior after one task as prior information for the next, but neural-network posteriors require approximation.
- The method combines Bayesian online learning with a Kronecker factored Laplace approximation to update a quadratic penalty for each task.
2 Bayesian online learning for neural networks
The method performs Bayesian online posterior updates with Gaussian approximations, using modes and curvature estimates to make sequential learning tractable for neural networks. Kronecker factored curvature captures within-layer parameter interactions while remaining more scalable than a full Hessian.
- The goal is to optimize one network across sequentially arriving datasets while retaining performance across all tasks.
- Bayesian online learning: Bayesian online learning updates an approximate posterior by using the previous posterior as the next task’s prior.
- Bayesian online learning: The procedure alternates an update step using the previous approximation as prior and a projection step back into the chosen parametric family.
- Online Laplace approximation: For neural networks, Laplace approximation makes Bayesian online learning tractable by finding a mode and locally approximating the posterior with a Gaussian.
- Online Laplace approximation: The Gaussian approximation produces a quadratic penalty that encourages weights to remain near the previous approximate posterior’s mean.
- Approximation boundary: The online procedure is a valid Laplace approximation only at task-specific modes; evaluating all Hessians at the latest mode introduces additional approximation error.
- Kronecker factored curvature: Modern networks make full Hessians impractical, while diagonal Fisher approximations ignore parameter interactions.
- Kronecker factored curvature: Kronecker factored curvature represents each layer’s Hessian block as a product of two smaller matrices and approximates dataset expectations through factor independence.
3 Related work
The paper relates its approach to Bayesian online learning, Laplace propagation, EWC, and other Kronecker factored posterior approximations. Its distinctions center on posterior updating, curvature structure, and avoiding some ad-hoc regularization.
- Unlike closed-form Bayesian online learning, the method uses gradient-based mode finding followed by a quadratic Gaussian approximation.
- EWC repeatedly adds penalties and uses a diagonal Hessian approximation, whereas the proposed method approximates the posterior with richer curvature structure.
- Other Kronecker factored posterior methods add an identity multiple as an ad-hoc regularizer, while this method uses the approximate posterior as the prior for later tasks.
- Additional related methods without a Bayesian motivation are discussed in the appendix.
4 Experiments
Experiments evaluate online Laplace variants across permuted MNIST, disjoint MNIST, and five sequential vision datasets. Kronecker factored curvature generally improves accuracy, memory–flexibility trade-offs, and performance on long task sequences.
- Experimental setup: The experiments compare online and per-task Laplace methods, EWC, SI, and diagonal versus block-diagonal Kronecker factored curvature approximations.The permuted MNIST evaluation uses 50 tasks with a smaller two-hidden-layer network, making capacity and long-sequence learning challenging.
- Permuted MNIST: Over 90% average test accuracy across 50 permuted MNIST tasks nearly matches a network trained jointly on all observed data.Recalculating curvature for each task instead of retaining previous estimates does not significantly affect performance.
- Permuted MNIST: Kronecker factored curvature gives higher accuracy than diagonal curvature on the first task, most recent task, and average across displayed λ values.With λ = 3, it produces similar performance on the first and most recent tasks; large λ improves retention but impairs learning new tasks.
- Disjoint MNIST: On disjoint MNIST, the Kronecker factored approximation slightly improves over the diagonal approximation and outperforms SI while requiring weaker regularization.The Laplace approximation performs significantly better than previously reported, although it does not match the method developed in [20].
- Vision datasets: Across five sequential vision tasks, online Laplace with Kronecker factored curvature achieves the highest test accuracy across all tasks.The difference among Laplace-based methods is smaller than the improvement associated with using a better Hessian approximation.
- Vision datasets: With Kronecker factored curvature, the network remembers easier early tasks almost perfectly while learning difficult later tasks better than diagonal methods.The diagonal approximation mostly remembers the first three tasks but struggles with the fifth, whereas SI learns the fifth task nearly as well as the Kronecker method.
5 Conclusion
The online Laplace approximation improves continual learning by modeling parameter interactions rather than only individual-parameter sensitivity. This produces higher test accuracy, especially across long sequences of datasets.
- The online Laplace approximation substantially improves over EWC and SI, which also use quadratic regularization for new tasks.
- Considering interactions and correlations between parameters is necessary for a more complete response to continual learning.
- The method achieves considerable increases in test accuracy on the investigated problems, particularly for long sequences of datasets.
A Derivation of the Kronecker factorization of the diagonal blocks of the Hessian
The derivation represents neural-network curvature through block-diagonal Kronecker factors and recursively computes pre-activation Hessians. This yields a positive semi-definite curvature approximation that captures within-layer weight interactions.
- Block-diagonal Kronecker factored approximations provide positive semi-definite approximations of the Hessian through the Fisher or Gauss-Newton matrix.
- A neural network maps input a0 = x through linear pre-activations hl = Wlal−1 and elementwise activations al = fl(hl).
- The gradient with respect to an individual weight is obtained using the chain rule, followed by differentiation with respect to another weight in the same layer.
- The resulting within-layer Hessian blocks can be expressed as Kronecker products.
- The pre-activation Hessian is calculated recursively, with the recursion initialized by the Hessian of the error with respect to network outputs.
- Diagonal matrices Bl and Dl define the terms used in the curvature recursion.
B Visualization of the effect of λ for a Gaussian prior and posterior
The multiplier λ controls whether the joint Gaussian mode favors the new-task likelihood or the previous-task prior. Its optimal value depends on posterior-approximation quality, while experiments use one shared λ across tasks.
- Values below 1 shift the joint maximum θ∗ toward the likelihood representing the new task, whereas values above 1 shift it toward the prior representing previous tasks.
- A small λ enables learning the new task despite underestimated uncertainty, while increasing λ improves retention of previous parameters.
- The optimal λ depends on the true posterior and how closely the approximation represents it.
- Experiments keep λ constant across tasks because the posterior-approximation family remains the same and task-specific tuning would grow linearly with the number of tasks.
C Additional related work
Related work includes methods that address catastrophic forgetting without a Bayesian motivation, including Synaptic Intelligence and gradient-projection approaches.
- Synaptic Intelligence adds a quadratic penalty using a heuristic importance measure based on update path length during the previous task.
- Lopez-Paz and Ranzato formulate a quadratic program that projects gradients to address performance on previous tasks.
D Optimization details
The permuted MNIST methods are evaluated under multiple optimizers and tuned hyperparameters, with momentum-based optimizers performing better for Laplace methods while Adam performs better for SI.
- The compared methods mildly depend on optimizer choice in the permuted MNIST experiment.
- The experiments use Adam for 20 epochs per dataset with learning rate 10^-3, alongside specified SGD-momentum and Nesterov schedules.
- Laplace-based methods perform better with momentum-based optimizers, whereas SI performs better with Adam.
- The optimizer, hyperparameter, and decay-rate combination is selected using the best validation error across all tasks at training’s end.
E Numerical results of the vision experiment
Table 1 reports per-dataset test accuracy at the end of training across a suite of vision datasets, comparing several continual-learning methods and Laplace variants.
- Table 1 reports per-dataset test accuracy at the end of training on the vision-dataset suite.
- The compared methods include SI, EWC, Per-Task Laplace, Approximate Laplace, and Online Laplace.
- The table abbreviates notMNIST, FashionMNIST, and CIFAR10 as nMNIST, fMNIST, and C10, respectively.
F Additional figures for the vision experiment
Figure 6 shows test accuracy as a convolutional network learns a sequence of vision datasets, comparing methods for preventing catastrophic forgetting against separately trained networks.
- The figure displays test accuracy on each dataset after training on that dataset begins.
- Datasets are trained separately in the displayed top-to-bottom order.
- The dotted black line represents the same architecture trained separately on each task.