Source-linked AI summary
Progress & Compress: A scalable framework for continual learning
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, Raia Hadsell
TL;DR
Continual learning systems must acquire sequential tasks without forgetting earlier skills, while transferring knowledge and scaling across tasks. Progress & Compress alternates active learning in an active column with distillation into an EWC-protected knowledge base, achieving a good trade-off across challenging domains. The authors report positive transfer on Atari and overall performance that can outperform online EWC and finetuning under the recommended setup.
Problem
Continual learning requires adapting to changing environments while retaining earlier task performance, transferring prior knowledge, scaling across tasks, and supporting backward transfer.
Method
Progress & Compress alternates learning new tasks in an active column with distilling its behavior into an EWC-protected knowledge base, using lateral connections for feature reuse.
Results
The framework achieves a good trade-off between transfer and reduced catastrophic forgetting across challenging domains, with positive transfer demonstrated on Atari.
Takeaways & Limitations
Progress & Compress provides a scalable framework for sequential problem solving that combines knowledge reuse with protection of previously learned skills.
Takeaways & Limitations
Original EWC is not scalable because its regularization terms grow linearly with the number of tasks, motivating the online EWC modification.
Abstract
from arXiv · showhide
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters and is designed to preserve performance on previously encountered tasks while accelerating learning progress on subsequent problems. This is achieved by training a network with two components: A knowledge base, capable of solving previously encountered problems, which is connected to an active column that is employed to efficiently learn the current task. After learning a new task, the active column is distilled into the knowledge base, taking care to protect any previously acquired skills. This cycle of active learning (progression) followed by consolidation (compression) requires no architecture growth, no access to or storing of previous data or tasks, and no task-specific parameters. We demonstrate the progress & compress approach on sequential classification of handwritten alphabets as well as two reinforcement learning domains: Atari games and 3D maze navigation.
1. Introduction
Continual learning requires systems to learn sequential tasks without forgetting, while also transferring prior knowledge, scaling across tasks, and supporting backward transfer. Progress & Compress addresses these competing goals with alternating active learning and consolidation phases.
- Continual learning systems must retain performance on earlier tasks while learning consecutive problems.
- The main desiderata also include positive forward transfer, scalability to many tasks, and positive backward transfer.
- Existing methods trade off these goals: finetuning transfers knowledge but forgets, EWC protects prior tasks but can impede new learning, and Progressive Networks lack scalability.
- Progress & Compress combines a knowledge base with an active column trained in alternating phases to unify these desiderata.
- During progress, lateral layerwise connections reuse knowledge-base features; during compress, the active column is distilled into the knowledge base while EWC protects prior skills.
2. The Progress and Compress Framework
The framework separates learning new tasks from consolidating them into a protected knowledge base. Feature reuse supports transfer during progress, while distillation and EWC preserve and integrate acquired behavior during compression.
- The architecture contains a knowledge base and an active column, alternating between progress and compress phases.
- Progress: Layerwise adaptor connections let the active column reuse features from the knowledge base during new-task learning.The adaptors are multilayer perceptrons; convolutional implementations use 1 × 1 convolutions.
- Progress: Parameters can be reinitialized for diverse tasks, whereas retaining them may improve positive transfer when tasks are very similar.
- Compress: Compression transfers newly learned behavior by minimizing cross-entropy between the active column’s predictions or policy and the knowledge base’s output.In reinforcement learning, distillation loss does not depend on reward scale.
- Compress: EWC protects previously acquired knowledge, but original EWC accumulates one regularizer per task and therefore scales linearly with task count.The framework uses online EWC to avoid this linear growth in computational requirements.
- Compress: During consolidation, the knowledge-base parameters are optimized while the active-column parameters remain fixed at the policy learned for the current task.
3. Related Work
Related work addresses continual learning through regularization, task-specific modules, replay, and transfer-oriented methods. These approaches differ in their handling of forgetting, transfer, scalability, task labels, and access to earlier data.
- Continual learning is also discussed under labels such as lifelong learning and never-ending learning, while transfer learning for reinforcement learning has a different focus.
- Finetuning can provide positive transfer when tasks are sufficiently similar, but it is mainly an alternative initialization strategy.
- Task-specific architectures share features or connect modules, but their resource demands can make scaling to many tasks difficult and unstable.
- Progressive Networks prevent forgetting by freezing learned columns, but this construction does not permit positive backward transfer.
- Learning Without Forgetting regularizes shared parameters using old-task outputs on current-task data, but it is not immediately applicable to reinforcement learning.
- Episodic-memory methods store prior examples or gradients to reduce forgetting, yet memory capacity can limit large-scale use.
- Replay approaches move toward multitask learning, whereas Distral transfers behavior through a distilled policy shared across tasks.
- PLAiD resembles Progress & Compress in its two-stage structure but assumes access to all tasks and consolidates through multitask distillation.
4. Online EWC
Online EWC reduces continual-learning consolidation to a constant-size running approximation, avoiding the linear growth of task-specific regularisers. Its re-centring and Fisher updates improve scalability but introduce approximation and retention concerns for nonlinear networks.
- Bayesian formulation: EWC approximates the posterior over knowledge-base parameters with a Gaussian centered at each task’s MAP solution and using diagonal Fisher information as precision.The Fisher serves as a positive-semidefinite surrogate for the negative-log-likelihood Hessian.
- Scalability: The standard formulation stores a mean and Fisher for every task, making computational cost linear in the number of tasks.The taskwise likelihood terms are incorporated sequentially through the posterior.
- Scalability: Online EWC instead applies Laplace’s approximation to the whole posterior, retaining only the latest MAP parameter and a running sum of Fishers.This reduces storage and computation to a constant-size representation, though the Fisher approximation remains local.
- Limitations: Re-centring previous likelihood approximations at the latest MAP parameter may weaken retention of older tasks in nonlinear neural networks.The authors state that the effect of re-centring is unclear and report that older tasks receive less direct constraint.
- Task revisitation: When tasks can be revisited, stochastic EP maintains one overall approximation and partially updates it instead of storing an explicit approximation for each task.A down-weighting factor γ is used when removing the approximation associated with a previous presentation of a task.
- Task revisitation: The approach avoids identifying task labels because it treats tasks equivalently, while task-boundary detection can rely on changes in low-level statistics such as reward statistics.The paper contrasts this with EWC/EP, which require task identities.
- Fisher normalisation: Normalising each task’s Fisher matrix makes updates depend on relative weight importance rather than arbitrary differences in the original Fisher scale.This is intended to treat tasks equally despite variable Fisher norms and reward schemes.
5. Experiments and Results
Experiments evaluate P&C on sequential Omniglot classification, 3D mazes, and Atari games against continual-learning desiderata including forgetting, transfer, and scalability. P&C shows strongest benefits when tasks permit positive transfer, while performance depends on task diversity and active-column reinitialisation.
- Experimental setup: Experiments span 50 sequential Omniglot alphabets, six Atari games, and eight 3D navigation tasks with varying difficulty and task similarity.The reinforcement-learning experiments use a distributed actor-critic architecture with four random seeds.
- Resilience against catastrophic forgetting: P&C retains initial Omniglot performance comparably to EWC and online EWC, whereas LwF degrades over many tasks and finetuning lacks forgetting protection.The comparison evaluates accuracy on an initial alphabet while learning the remaining 49 alphabets.
- Assessing forward transfer: P&C matches finetuning on similar random mazes, with both showing positive transfer in generalisation and data-efficient learning; online EWC transfers less effectively.Rewards are evaluated on a held-out maze after seven previous mazes and normalised against dedicated models trained from scratch.
- Assessing forward transfer: On diverse Atari tasks, EWC variants reduce final performance by over 40% on average, while P&C maintains significant positive transfer when active-column parameters are reinitialised.Without reinitialisation, P&C shows the same loss of transfer as finetuning, suggesting limited use of knowledge-base connections.
- Evaluating overall performance: Across Omniglot, no method improves generalisation over dedicated per-task models, but P&C achieves faster learning; it nevertheless attains the highest mean performance among compared methods.Online EWC is competitive, while Progressive Networks incur scalability costs from connections to previous columns.
- Evaluating overall performance: Atari learning curves show significant P&C improvements on several games and comparable or slightly worse performance on the remaining tasks.For P&C, the plotted rewards are collected during compression because the active-column parameters remain unchanged while learning a new task.
- Conclusions: P&C is best suited to domains allowing positive transfer, where it can improve over methods primarily designed to prevent forgetting.The authors also report that allowing some forgetting during repeated visits can increase overall performance by weakening capacity constraints.
6. Summary & Discussion
Progress & Compress facilitates transfer in sequential problem solving while minimizing catastrophic forgetting, combining a knowledge-preservation mechanism with an online EWC variant. The framework assumes the learner detects changes in task distribution, rather than identifying the current task.
- Progress & Compress achieves a good trade-off between transfer and minimizing catastrophic forgetting across challenging domains.
- The learner is assumed to recognize changes in task distribution so a new posterior approximation can be computed.
- This assumption relaxes the stricter requirement that the learner know the identity of the current task, with gradual drift identified as a future challenge.
- The framework uses an online EWC variant with an explicit forgetting mechanism and empirical evidence of practical performance.
Progress & Compress: A scalable framework for continual learning. Supplementary material
Figure 5 tracks retention of performance on an initial permuted-MNIST task while subsequent tasks are trained.
- The figure plots test accuracy on Task A while training proceeds on Tasks B-E.
A. Retention of task performance for EWC and online EWC
The permuted-MNIST experiments show that EWC preserves earlier-task accuracy better with limited training, whereas retaining initial parameters can hinder recent-task retention with more training.
- EWC, online-EWC, and pure SGD are compared across five sequential permuted-MNIST tasks, with Task A retention tracked during later-task learning.
- EWC retains higher accuracy on the initial Task A than online-EWC and pure SGD during subsequent training.
- Figure 6 reports final task accuracies by training-step budget, using ten sessions per budget with shared data across methods.
- For 500 and 1000 training steps, EWC accuracy is higher than online-EWC across the plotted tasks.
- With 10,000 training steps, preserving initial parameters makes retaining the most recent tasks more difficult than at 500 steps.
B. Faster learning on Omniglot
Progress & Compress does not improve final Omniglot generalization across alphabets, but it can improve data efficiency and accelerate learning after pre-training on earlier tasks.
- All considered methods fail to obtain higher sequential-learning accuracy in Omniglot, while Progress & Compress still shows improved data efficiency.
- Figure 7 presents averaged learning curves for Omniglot experiments.
- Progress & Compress was evaluated on ten unique Omniglot alphabets after pre-training on up to four different tasks.
- Pre-training on a small number of tasks can greatly improve data efficiency, with the effect plateauing beyond four tasks.
C.1. Omniglot
The Omniglot experiments used a common convolutional architecture and standardized data preparation across methods. Training and hyperparameters were kept consistent, with validation-based tuning for continual-learning methods.
- Network architecture: All methods used convolutional networks with four 3 × 3 convolutional blocks, 64 filters, ReLU nonlinearities, and 2 × 2 max-pooling.P&C and Progressive Nets used this architecture for every network column.
- Data preparation: The dataset was split 60/20/20% for training, validation, and testing, with images resized to 28 × 28 and augmented using 20 random permutations per image.Train and test alphabets were not distinguished because the setup was not few-shot learning.
- Optimization: Each alphabet was trained for 2500 updates with batch size 32 and stochastic gradient descent at learning rate 0.1, reduced to 0.05 during distillation.These settings were considered sufficient to learn each alphabet separately from scratch.
- Hyperparameter selection: Regularization and distillation hyperparameters were selected by grid search to maximize averaged validation performance across tasks.P&C and EWC variants varied λ and γ, while LwF varied λ; distillation used temperature τ = 2.0.
- Experimental consistency: The same network and optimization settings were used throughout, except isolated positive-transfer and forgetting comparisons fixed λ across EWC methods.This exception was intended to provide a fair comparison.
C.2. Atari & Navigation tasks
The Atari and navigation experiments used a shared actor-critic network derived from prior Atari work, with common optimization and reinforcement-learning settings. EWC variants received separate regularization tuning, including Fisher estimation from 100 equal-sized minibatches.
- Network and optimization: Atari and navigation tasks used a shared network with linear value and policy outputs adapted for actor-critic optimization.The network architecture followed Mnih et al. (2013).
- Network and optimization: Optimization used batch size 20, unroll length 20, RMSProp with ϵ = 0.1, and a learning rate annealed linearly to zero.The supplied passage also specifies an initial navigation learning rate of α = 0.004, but the sentence is truncated.
- Reinforcement-learning settings: Rewards were clipped to a maximum absolute value of 1.0, the policy-gradient baseline cost was 0.5, and the discount factor was 0.99.These settings were used for the reinforcement-learning experiments.
- EWC tuning: EWC regularization strengths were tuned separately, with online EWC using a distinct λ range because the loss scales differed.The EWC λ grid was 500–3000, while online EWC used 25, 75, 125, or 175.
- EWC tuning: The diagonal Fisher information was estimated using 100 minibatches of equal size.