Source-linked AI summary
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell
TL;DR
Neural networks struggle to learn tasks sequentially without catastrophic forgetting, limiting continual learning. This paper introduces elastic weight consolidation, which protects task-important weights, and shows sequential learning across MNIST classification and Atari games while retaining older abilities.
Problem
Neural networks need to learn consecutive tasks without forgetting previous ones, but continual learning remains a barrier to artificial general intelligence.
Method
Elastic weight consolidation selectively slows learning on weights important to previous tasks, using a soft quadratic constraint to preserve them.
Results
EWC learns sequential supervised and reinforcement-learning tasks while preserving earlier performance, unlike SGD, fixed L2 regularization, and dropout, which fail to scale comparably.
Takeaways & Limitations
Continual learning without catastrophic forgetting is achievable in deep neural networks across multiple challenging task domains.
Takeaways & Limitations
Earlier quadratic-penalty approaches were limited to small models, motivating EWC’s more scalable formulation.
Abstract
from arXiv · showhide
The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks which they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on the MNIST hand written digit dataset and by learning several Atari 2600 games sequentially.
1 Introduction
Continual learning is essential for agents that must learn and remember many tasks despite unpredictable task switches and long gaps between repetitions. Neural networks suffer catastrophic forgetting in sequential learning, motivating elastic weight consolidation, which protects important weights for previously learned tasks.
- Motivation: Continual learning requires acquiring consecutive tasks without forgetting them when task identities, switching patterns, and recurrence intervals are uncertain.The task sequence may be unlabeled, switch unpredictably, and contain long intervals before a task recurs.
- Problem: Sequentially training neural networks can abruptly erase knowledge of earlier tasks as information relevant to a current task is incorporated.This phenomenon is termed catastrophic forgetting.
- Problem: Supporting continual learning remains a key barrier to developing artificial general intelligence.
- Contribution: The authors propose task-specific synaptic consolidation as a solution and develop elastic weight consolidation for neural networks.EWC slows learning on weights according to their importance for previously seen tasks.
- Contribution: EWC is presented as applicable to supervised learning.
2 Elastic weight consolidation
Elastic weight consolidation (EWC) preserves earlier task performance by constraining important parameters near their previous values while learning a new task. It uses a parameter-specific quadratic penalty, with importance estimated from the diagonal Fisher information matrix, and extends across multiple tasks.
- Core mechanism: EWC constrains parameters to remain near the solution learned for an earlier task, protecting performance while searching for a solution to the new task.Over-parameterization makes it likely that a good solution for task B lies near the solution for task A.
- Core mechanism: The constraint is a quadratic penalty whose spring-like stiffness is greater for parameters that matter most to the earlier task.Using the same coefficient for every parameter is too restrictive because it protects all weights equally.
- Importance estimation: EWC approximates the earlier task’s posterior with a Gaussian centered at θ∗A, using the diagonal Fisher information matrix as its precision.The Fisher matrix is equivalent to the loss’s second derivative near a minimum, computable from first-order derivatives, and positive semi-definite.
- Importance estimation: The resulting objective combines the new-task loss with parameter-wise penalties weighted by Fisher importance, where λ controls the old task’s influence.The index i labels each parameter, and LB(θ) denotes the loss for task B alone.
- Multiple tasks: For a third task, EWC keeps parameters close to the learned solutions for both earlier tasks using separate penalties or their equivalent summed quadratic penalty.The sum of two quadratic penalties is itself a quadratic penalty.
3 Discussion
The discussion presents elastic weight consolidation (EWC) as a scalable continual-learning algorithm that protects prior knowledge by selectively reducing weight plasticity. It frames EWC Bayesianly and connects its success in deep networks to neurobiological theories of synaptic consolidation.
- Algorithmic contribution: EWC protects previous-task knowledge during new learning by selectively decreasing the plasticity of important weights through a soft quadratic constraint.This mechanism is presented as a way to avoid catastrophic forgetting and as parallel to synaptic consolidation.
- Bayesian interpretation: Under a Bayesian interpretation, EWC uses previous-task posteriors as priors, enabling fast learning for poorly constrained parameters and slow learning for crucial ones.The parameter prior is the posterior conditioned on data from earlier tasks.
- Prior limitations: Earlier quadratic-penalty approaches were limited to small models; one was slow because it recomputed curvature at each sample, while ELLA required computing and inverting matrices.The passage contrasts these computational limitations with the discussion’s presentation of EWC.
- Neurobiological connections: EWC has considerable parallels with two computational theories of synaptic plasticity and with the proposal that synapses represent uncertainty about their weights.The discussion links variable synaptic responses and plasticity to uncertainty about the weight posterior.
- Empirical significance: Combining a neurobiologically inspired continual-learning algorithm with deep neural networks achieves successful performance across challenging domains and demonstrates that synaptic-consolidation theories scale.The discussion identifies sequential learning without forgetting as important to biological and artificial intelligence.
4 Appendix
The appendix details the MNIST and Atari experimental implementations, including network architectures, training settings, evaluation design, and automatic task recognition. It also defines Fisher-matrix overlap as a measure of how similarly two tasks use network weights.
- Atari implementation: Atari agents use an 84x84 grayscale representation of four concatenated observations and a three-convolution network followed by a 1024-unit fully connected layer.The original 210x160 RGB frames are downsampled using bilinear interpolation, converted to YUV, and represented using the grayscale channel.
- Evaluation design: 10-game experiments randomly sampled games from 19 Atari titles, averaged results over 10 game sets, and used 4 random seeds per set to compare baseline, EWC, and plain SGD training.The selected games were those for which standalone DQN could reach human-level performance in 50 × 10^6 frames.
- Automatic task determination: Atari task recognition models each task as a categorical hidden context in a generative environment model, with the highest-probability context selected as the current task label.The implementation uses factored multinomial pixel distributions with parametrized Dirichlet priors updated through Bayesian evidence.
- Weight overlap analysis: Fisher-matrix overlap is defined as 1 − d^2 after normalizing two task Fishers to unit trace, with values from zero for non-overlapping weights to one for proportional matrices.The distance used is the Fréchet distance on positive-semidefinite matrices and is bounded between zero and one.