PaperlayerView exact revision on arXiv

Source-linked AI summary

A continual learning survey: Defying forgetting in classification tasks

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, Tinne Tuytelaars

arXiv:1909.08383v3cs.CVstat.ML

TL;DR

Catastrophic forgetting makes static neural networks poorly suited to sequential learning, motivating continual learning without retraining from scratch. The paper surveys task-incremental classification, proposes a current-task-only stability-plasticity framework, and compares methods broadly. PackNet is the strongest overall performer in the studied multi-head setting, while performance varies substantially across datasets and methods.

  • Problem

    Static neural networks catastrophically forget earlier concepts when learning new tasks, while existing continual-learning evidence lacks comprehensive comparisons across settings.

  • Method

    The paper develops a taxonomy, a current-task-only framework for dynamically setting stability-plasticity hyperparameters, and experiments comparing 11 continual-learning methods with four baselines.

  • Results

    PackNet performs best overall across the three datasets, while method strengths vary with task similarity, distribution shift, model capacity, and regularization.

  • Takeaways & Limitations

    Parameter isolation is comparatively robust on challenging distributions, whereas LwF and EBLL collapse under the substantial distribution shifts in RecogSeq.

  • Takeaways & Limitations

    The evidence is largely confined to classification, multi-head evaluation, and the task-incremental setting, leaving broader continual-learning settings for future research.

Abstract

from arXiv · show

Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting. Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch. We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries. Our main contributions concern 1) a taxonomy and extensive overview of the state-of-the-art, 2) a novel framework to continually determine the stability-plasticity trade-off of the continual learner, 3) a comprehensive experimental comparison of 11 state-of-the-art continual learning methods and 4 baselines. We empirically scrutinize method strengths and weaknesses on three benchmarks, considering Tiny Imagenet and large-scale unbalanced iNaturalist and a sequence of recognition datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time, and storage.

1 INTRODUCTION

Continual learning addresses catastrophic forgetting by enabling neural networks to accumulate knowledge across sequential tasks without retraining from scratch. This survey focuses on task-incremental classification and provides a taxonomy, a fair hyperparameter framework, and broad empirical comparisons.

  • Static neural networks require restarting training when new data arrives, which becomes impractical for streams or constrained storage and privacy settings.
  • Continual learning seeks to extend acquired knowledge over an infinite data stream while avoiding catastrophic forgetting of previously learned concepts.
  • The study focuses on task-incremental classification, where each task arrives as a batch, is trained offline, and previous or future task data remain inaccessible.
  • The survey addresses limited consensus and the absence of a comprehensive experimental comparison across architectures, task combinations, and hyperparameters.
  • Its framework dynamically determines the stability-plasticity trade-off using current-task data while comparing 11 representative continual learning approaches.
  • Methods generalize across balanced Tiny Imagenet but less consistently to unbalanced iNaturalist and uneven RecogSeq, with parameter-isolation methods more resistant in challenging settings.

2 THE TASK INCREMENTAL LEARNING SETTING

The task-incremental setting trains on one task at a time while evaluating performance across all seen tasks despite unavailable old-task data. Tasks may differ in classes, domains, or output spaces, and the paper studies the task-incremental classification case.

  • Task-incremental learning receives one task’s training data at a time and trains until convergence before proceeding to later tasks.
  • Each task contains samples X^(t) and labels Y^(t) drawn from its task distribution D^(t).
  • The objective is to control statistical risk across all seen tasks despite limited or no access to previous task data.
  • Tasks can represent new class groups, domains, or output spaces, distinguishing class-incremental, task-incremental, and domain-incremental settings.
  • In the studied task-incremental setup, task labels identify isolated output nodes for each task.

3 CONTINUAL LEARNING APPROACHES

The survey organizes continual learning methods by how task-specific information is stored and used: replay, regularization, and parameter isolation. These families respectively reuse past information, constrain updates, or allocate separate parameters to tasks.

  • The taxonomy distinguishes replay, regularization-based, and parameter-isolation families according to how task-specific information is stored and used.
  • Replay methods: Replay methods store raw or generated samples and reuse them during new-task learning to alleviate forgetting.
  • Replay methods: Rehearsal retrains on stored exemplars, whereas GEM constrains new-task gradients to avoid interference with previous tasks.
  • Replay methods: Generative replay avoids storing raw inputs but adds continual generative-model training complexity and risks mode collapse.
  • Regularization-based methods: Regularization methods avoid storing raw inputs by adding loss penalties that consolidate previous knowledge, using data-focused or prior-focused strategies.
  • Parameter isolation methods: Parameter-isolation methods assign different parameters or masked subnetworks to tasks, preventing interference by freezing or excluding previous task components.

4 CONTINUAL HYPERPARAMETER FRAMEWORK

The hyperparameter framework selects stability-plasticity settings without previous-task data by first measuring maximal new-task plasticity, then decaying protection until the tolerated performance threshold is reached.

  • Grid-searching forgetting-related hyperparameters with validation data from all tasks violates the no-previous-data assumption and may produce unreproducible results.
  • The framework uses only current-task data and dynamically seeks a stability-plasticity trade-off for each continual learning method.
  • Maximal Plasticity Search: Maximal Plasticity Search finetunes a model copy on new-task data and selects a learning rate by maximizing held-out new-task accuracy.
  • Stability Decay: Stability Decay begins with maximal forgetting protection, then decreases method-specific hyperparameters when new-task performance fails to remain within the tolerated threshold.

5 COMPARED METHODS

The paper compares representative replay, regularization-based, and parameter-isolation methods for continual learning, alongside a framework for selecting stability–plasticity hyperparameters. The methods differ in how they preserve prior knowledge, allocate capacity, and trade memory or computation for retention.

  • Compared method families: The comparison covers representative methods from replay, regularization-based, and parameter-isolation families.The study describes selected methods before conducting a comprehensive comparison across the three families.
  • Hyperparameter selection: The framework dynamically selects learning-rate and stability–plasticity settings using only current-task data, without access to previous or future tasks.Its maximal-plasticity search evaluates candidate learning rates, while hyperparameter decay reduces stability constraints when accuracy falls below a margin.
  • Replay methods: Replay methods store class exemplars, but fixed memory limits can reduce exemplar representativeness while raw samples add computation, storage, and privacy costs.iCaRL selects samples near each class feature mean and combines new-class loss with distillation on previously learned classes.
  • Regularization-based methods: Regularization methods preserve prior knowledge by constraining parameter changes or outputs, but their effectiveness depends on importance estimation and regularization strength.The survey compares seven regularization methods; EWC estimates importance with the Fisher Information Matrix, while SI accumulates importance online during training.
  • Regularization-based methods: LwF distills previous-task knowledge from new-task data, but distribution shifts can cause gradual error accumulation and require storing additional outputs.Its success depends heavily on how closely the new-task data relates to earlier tasks.
  • Parameter merging: IMM merges task-specific models, but independent local minima can make simple parameter averaging increase loss; transfer variants initialize or regularize around previous parameters.The study compares mean-IMM and mode-IMM with weight-transfer and L2-transfer, with mode-IMM consistently outperforming mean-IMM.
  • Parameter isolation methods: HAT uses task-specific attention masks and sparsity regularization to preserve earlier task representations while retaining capacity for additional tasks.Per-layer embeddings are converted into increasingly near-binary masks during training.

6 EXPERIMENTS

Experiments compare continual-learning methods using accuracy and forgetting across model choices, regularization settings, replay schemes, and task orders. On Tiny Imagenet, PackNet and iCaRL achieve the strongest overall retention and accuracy, while results expose method-specific trade-offs in capacity, replay memory, and storage.

  • Evaluation: Accuracy and forgetting are evaluated per task after each task is trained, with forgetting defined relative to accuracy when that task was first learned.Figures track accuracy as additional tasks are added, while tables report average accuracy and forgetting.
  • Common baseline: 55.70% average accuracy is achieved by joint training on Tiny Imagenet, providing a soft upper bound for continual-learning methods.Tiny Imagenet tasks are balanced in classes, training data, and approximate difficulty.
  • Baselines: 21.30% average accuracy and 26.90% average forgetting characterize finetuning, whose performance drops sharply when new tasks are added.The baseline initially performs well on each task but suffers severe catastrophic forgetting afterward.
  • Parameter isolation: 49.13% average accuracy makes PackNet the strongest final performer, although compression limits capacity for new tasks while masking preserves prior knowledge without forgetting.PackNet’s flat curves reflect complete retention through the task sequence.
  • Regularization: 46.90% average accuracy makes MAS superior to the other two regularization methods under continual hyperparameter selection, while LwF and EBLL remain competitive.LwF and EBLL obtain 41.91% and 45.34%, respectively, versus 42.43% for EWC.
  • Replay: 48.76% average accuracy is obtained by iCaRL with a doubled replay buffer, improving over its 47.27% standard result and approaching PackNet.iCaRL also shows −1.11% average forgetting, indicating backward transfer for about half the tasks; its baseline comparison is 47.27% versus 37.31% for R-FM.
  • Replay baselines: R-FM significantly outperforms R-PM for tasks 1–7, but the two replay baselines converge for the final tasks as their memory allocations become equivalent.The comparison isolates the effect of using full available memory from the start versus preallocating memory across all tasks.
  • Capacity and regularization: Dropout is generally beneficial but can increase forgetting for replay methods, whereas weight decay mainly improves wide-network accuracy and often harms smaller models.Model capacity also matters: SI is vulnerable to overfitting, PackNet mitigates it through compression, and replay baselines are broadly model agnostic except for DEEP.

7 LOOKING AHEAD

The paper contrasts its relaxed task-incremental setting with ideal continual learning and identifies desiderata for more general systems. These differences motivate examining how continual learning should extend beyond clearly separated offline tasks.

  • The study’s task-incremental setting requires known task boundaries and permits offline training over shuffled batches until convergence.This makes it a relaxation of the desired continual learning system.
  • An ideal continual learner would use constant memory, learn online without task boundaries, and support forward and backward transfer.
  • Further desiderata include problem-agnostic learning, adaptation from unlabeled data, no test-time task oracle, task revisiting, and graceful forgetting.
  • Comparing continual learning with related fields is necessary because they share properties such as knowledge sharing, adaptation, and transfer.

8 RELATED MACHINE LEARNING FIELDS

Related machine learning fields share aspects of continual learning but differ in simultaneity, adaptation, task directionality, and data assumptions. The paper positions continual learning as sequential knowledge accumulation rather than one-time transfer or joint learning.

  • Multi-Task Learning: Multi-Task Learning jointly learns related tasks through shared parameters, whereas continual learning adapts after deployment.
  • Transfer Learning: Transfer Learning uses source-task knowledge for a target task but does not continuously adapt or evaluate retained source-task performance.
  • Domain Adaptation: Domain Adaptation transfers between different input domains and remains unidirectional without accumulating knowledge.
  • Online Learning: Online Learning processes sequential data instances, but typically assumes i.i.d. sampling within a single task domain.
  • Open World Learning: Open World Learning detects new classes at test time and becomes incremental learning when those classes are integrated into the model.

9 CONCLUSION

The paper evaluates state-of-the-art continual learning methods within task-incremental, multi-head classification and introduces a continual hyperparameter framework for fair comparison. Experiments across three datasets examine method behavior, architecture, regularization, and task ordering, while the authors identify broader scope limitations.

  • The study compares recent continual learning methods in task-incremental classification with a multi-head setup.
  • A continual hyperparameter framework determines the stability-plasticity trade-off using only current-task data.
  • Experiments on Tiny Imagenet, iNaturalist, and RecogSeq assess method strengths and weaknesses under unbalanced and varying task sequences.
  • The study examines model capacity, dropout, weight decay, and task ordering, finding task order minimally influential for the general method trends.
  • The authors conclude that future work should move beyond classification, multi-head evaluation, and task-incremental settings.

A.3 Methods Setup

The experiments use standardized optimization, initialization, stopping, and temperature settings, while configuring replay, regularization, parameter-isolation, and timing procedures separately. These choices define how the compared methods are trained and measured.

  • General setup: All methods use PyTorch, momentum SGD with momentum 0.9, batch size 200, and early stopping with learning-rate annealing.Training stops after 10 unimproved validation iterations; the learning rate decays after 5.
  • General setup: Baselines and parameter-isolation methods start from scratch, while other methods continue from the model trained on the first task.
  • General setup: Knowledge-distillation temperatures are fixed at 2, and initial hyperparameters are set near empirically observed upper bounds with an inverse-decay margin.
  • Replay setup: GEM divides memory equally among tasks, while iCARL incrementally divides total exemplar memory equally across seen tasks.
  • Regularization-based setup: Regularization strengths start at EWC 400, SI 400, MAS 3, and distillation strength 10 for LwF and EBLL.
  • Parameter isolation setup: PackNet begins with 90% pruning per layer, while HAT uses c = 2.5 and smax = 800 after first-task warmup.
  • Timing experiments: Timing experiments use Tiny Imagenet with the BASE model, batch size 200, 10 epochs for iterative phases, and 450 exemplars per task.

B.1 Complete Mean-IMM Results

Mean-IMM is consistently outperformed by mode-IMM on Tiny Imagenet, while regularization can worsen mean-IMM performance. Results also include comparisons on iNaturalist and across task orderings.

  • Mode-IMM consistently surpasses mean-IMM on Tiny Imagenet across random, easy-to-hard, and hard-to-easy task orderings.Mode-IMM tracks per-parameter importance for each task, whereas mean-IMM averages models optimized for different tasks.
  • Dropout increases forgetting for mean-IMM in the evaluated Tiny Imagenet configurations.Table 12 also measures weight decay and dropout effects across four model configurations.
  • Mean-IMM and mode-IMM are compared on iNaturalist using pretrained AlexNet, alongside baseline methods.The manuscript reports iNaturalist and RecogSeq results for all three task orderings.

B.2 Synaptic Intelligence (SI): Overfitting and Regularization

SI exhibits substantial overfitting without regularization, while dropout and weight decay reduce the training-validation discrepancy. EWC and LwF reduce overfitting intrinsically, whereas finetuning benefits less clearly from regularization.

  • 48.8% average training-validation discrepancy shows substantial SI overfitting without regularization on Tiny Imagenet.Training accuracy tends to reach 100% while validation accuracy attains about half of that level.
  • Dropout reduces SI’s discrepancy to 20.6%, while weight decay reduces it to 30.8%.These results are reported for the BASE model on randomly ordered Tiny Imagenet.
  • Increasing finetuning weight decay up to λ = 0.0001 reduces average training accuracy to 85.5% but leaves a significant validation discrepancy.Without regularization, SI and finetuning have similar discrepancies of 48.8% and 48.0%, respectively.
  • EWC and LwF reduce overfitting without traditional weight decay or dropout, with discrepancies of 38.4% and 28.9%, respectively.The comparison concerns SI, finetuning, EWC, and LwF on the BASE model.
  • GEM’s comparison with iCARL and rehearsal baselines is sensitive to the number of training epochs.The study adds an experiment because the original GEM setup assumes one epoch per task, unlike the multi-epoch experiments used elsewhere.

B.4 Reproduced RecogSeq Results

RecogSeq results broadly reproduce the original setup, but they reveal strong learning-rate sensitivity: regularization-based methods favor low rates, whereas PackNet performs better with higher rates.

  • RecogSeq results are similar to the original proposed setup under the study’s coarse learning-rate grid.The experiments consistently use the same coarse grid of learning rates.
  • Using softer learning rates reproduces the original results but relies on hyperparameters selected through validation data from all tasks.This selection violates the continual-learning assumption that only current-task validation data is available.
  • Regularization-based approaches show less forgetting at low learning rates, while PackNet accuracy decreases in that regime.The findings indicate differing learning-rate preferences across methods.
  • The results expose significant hyperparameter sensitivity: regularization-based methods favor low learning rates, whereas PackNet thrives only with higher learning rates.This contrast complicates fair continual-learning comparisons when task-wide validation data is unavailable.

B.5 HAT Analysis: Asymmetric Capacity Allocation

HAT’s capacity allocation is highly asymmetric across layers and model settings, limiting learning on large or difficult task sequences. Capacity constraints are especially severe for early layers in DEEP, iNaturalist, and RecogSeq configurations.

  • Asymmetric Capacity Allocation: Unequal capacity distribution creates difficulties for HAT on large-scale iNaturalist and RecogSeq tasks and for DEEP on Tiny Imagenet.These tasks require much of the low-level feature capacity from the first task onward.
  • Hyperparameters: Current-task validation data makes HAT’s sparsity and gate-slope hyperparameters difficult to set for controlled capacity allocation.The original HAT procedure searches validation sets from all tasks, unlike the study’s continual framework.
  • Asymmetric Capacity Allocation: The first two Tiny Imagenet tasks consume over 80% of DEEP’s Conv0 capacity, leaving virtually none after task 3.This prevents subsequent tasks from learning effectively.
  • Asymmetric Capacity Allocation: In SMALL, Conv0 saturates only at the last task, while other layers retain 50%–80% capacity.The layer-wise usage pattern differs substantially from DEEP.
  • Capacity mitigation: Increasing sparsity regularization or reducing smax can free capacity, but subsequent tasks may still require all remaining early-layer capacity.Additional sparsity schemes did not resolve the capacity problem in the large-scale setups.
  • Asymmetric Capacity Allocation: RecogSeq cannot attain more than 30% remaining capacity, while iNaturalist can retain more free capacity under the reported AlexNet analysis.Table 18 reports first-task validation accuracy together with minimum, maximum, and total remaining capacity.
  • Capacity Analysis: PackNet’s capacity saturation rate remains tied to pruning percentage, so reducing model size assigns fewer parameters per task without changing that rate.The study also increases the number of tasks to identify when free capacity is depleted.
  • Capacity Analysis: Reducing BASE to SMALL lowers PackNet’s gain to 1.55%, compared with 2.23% for BASE on Tiny Imagenet.SMALL reaches 46.68% accuracy versus 45.13% for EWC, while BASE reaches 49.13% versus 46.90% for the best other method.

B.6.2 Long Task Sequence Experiment

The 40-task Long Tiny Imagenet experiment stresses continual learners by extending the task sequence while keeping model capacity fixed. PackNet performs strongly on similar tasks despite capacity saturation, but saturation severely limits plasticity on a later, different task.

  • Extending the sequence to 40 tasks keeps model capacity fixed while increasing the number of tasks that must receive allocated capacity.
  • HAT deteriorates before completing the 10-task Tiny Imagenet sequence because asymmetric allocation rapidly saturates early-layer capacity.
  • 70.32% versus 68.80% average accuracy after task 21 marks iCaRL superseding PackNet midway through the Long Tiny Imagenet sequence.
  • PackNet’s capacity saturates after task 11, yet it retains the best average accuracy and forgetting in the Long Tiny Imagenet experiment.
  • 26.62% SVHN accuracy shows PackNet’s plasticity severely deteriorates after 40 tasks, whereas other methods approach 90% on the new task.
  • PackNet’s fixed weights preserve prior performance, with accuracy dropping only 1%, but prevent learning tasks that depart from the Long Tiny Imagenet distribution after saturation.
Loading 1909.08383v3…