PaperlayerView exact revision on arXiv

Source-linked AI summary

Three scenarios for continual learning

Gido M. van de Ven, Andreas S. Tolias

arXiv:1904.07734v1cs.LGcs.AIcs.CVstat.ML

TL;DR

Catastrophic forgetting and inconsistent evaluation protocols make continual-learning methods difficult to compare. The paper defines three task-identity scenarios and compares methods on split and permuted MNIST, finding that regularization methods fail when task identity must be inferred while replay-based methods perform well.

  • Problem

    Inconsistent evaluation protocols hinder direct comparison of continual-learning methods for mitigating catastrophic forgetting.

  • Method

    The paper defines three continual-learning scenarios by whether task identity is provided or must be inferred, then compares proposed methods on split and permuted MNIST.

  • Results

    Regularization-based methods fail in Class-IL, whereas only replay-based methods obtain good or acceptable performance across the more challenging scenarios.

  • Takeaways & Limitations

    Continual-learning performance depends substantially on the scenario, and replay-based approaches are currently the most reliable across scenarios when task identity must be inferred.

  • Takeaways & Limitations

    The study uses relatively easy-to-generate MNIST images, leaving generative replay’s success on more complicated input distributions unresolved.

Abstract

from arXiv · show

Standard artificial neural networks suffer from the well-known issue of catastrophic forgetting, making continual or lifelong learning difficult for machine learning. In recent years, numerous methods have been proposed for continual learning, but due to differences in evaluation protocols it is difficult to directly compare their performance. To enable more structured comparisons, we describe three continual learning scenarios based on whether at test time task identity is provided and--in case it is not--whether it must be inferred. Any sequence of well-defined tasks can be performed according to each scenario. Using the split and permuted MNIST task protocols, for each scenario we carry out an extensive comparison of recently proposed continual learning methods. We demonstrate substantial differences between the three scenarios in terms of difficulty and in terms of how efficient different methods are. In particular, when task identity must be inferred (i.e., class incremental learning), we find that regularization-based approaches (e.g., elastic weight consolidation) fail and that replaying representations of previous experiences seems required for solving this scenario.

1 Introduction

Continual learning remains difficult because standard neural networks catastrophically forget previously learned tasks, while inconsistent evaluation protocols obscure comparisons among proposed methods. The paper introduces three increasingly difficult scenarios and finds that task-identity inference exposes failures of regularization methods, whereas replay-based approaches can work across scenarios.

  • 1 Introduction: Standard neural networks forget most information about previously learned tasks when trained sequentially, a phenomenon called catastrophic forgetting.
  • 1 Introduction: Varied experimental protocols make continual-learning results difficult to compare, with methods sometimes performing well in one setting and failing dramatically in another.
  • 1 Introduction: The report defines three continual-learning scenarios of increasing difficulty based on whether task identity is provided at test time or must be inferred.
  • 1 Introduction: Extensive comparisons across the three scenarios show substantial differences in both scenario difficulty and method effectiveness.
  • 1 Introduction: Regularization-based approaches, including elastic weight consolidation, completely fail when task identity must be inferred, even under simple MNIST classification protocols.
  • 1 Introduction: Replay-based approaches are the only methods reported to have potential to perform well across all three scenarios.

2 Three Continual Learning Scenarios

The paper defines three continual learning scenarios based on whether task identity is provided at test time and, if not, whether the model must infer it. Task-IL is easiest, Domain-IL omits task identity without requiring inference, and Class-IL requires both task solving and task identification.

  • Problem setup: Continual learning assumes sequentially learned, clearly separated tasks with only current-task data available during training.The framework is intended to structure comparisons among methods addressing catastrophic forgetting.
  • Task-incremental learning: Task-IL provides task identity at test time, permits task-specific components, and is the easiest scenario.A typical architecture uses a multi-headed output layer with separate output units for each task.
  • Domain-incremental learning: Domain-IL withholds task identity at test time, but models only need to solve the current task rather than infer which task it is.Typical protocols retain task structure while changing the input distribution.
  • Class-incremental learning: Class-IL requires models to solve every task seen so far and infer which task they are presented with.This scenario includes incrementally learning new object classes.
  • Relation to prior distinctions: The categorization extends multi-headed versus single-headed layouts by separating cases without provided task identity according to whether task inference is required.The paper states that these resulting scenarios substantially differ in difficulty.
  • Task protocols: The scenarios can be applied to both split MNIST and permuted MNIST, although split MNIST is commonly used for Task-IL and Class-IL while permuted MNIST is most natural for Domain-IL.Each protocol can nevertheless be performed according to all three scenarios.

3 Strategies for Continual Learning

The section organizes continual-learning strategies around restricting task-specific parameter updates, regularizing parameters across tasks, replaying pseudo-data or stored data, and using stored exemplars during execution. Replay includes distillation, generative replay, hybrid replay, and exact replay, while exemplar-based classification uses learned features and class means.

  • Task-specific subnetworks: Defining a different sub-network for each task can alleviate catastrophic forgetting by avoiding optimization of the entire network on every task.The strategy is motivated by new-task optimization making network parameters no longer suitable for previous tasks.
  • Parameter regularization: When task identity is unavailable at test time, EWC and SI differently regularize network parameters during training while using the entire network for execution.Both methods estimate the importance of all network parameters.
  • Replay: Replay complements each new task’s training data with pseudo-data representing previous tasks, including current inputs with model-generated targets and generated inputs from a sequential generative model.Learning without Forgetting uses soft targets rather than only the most likely hard target; Deep Generative Replay uses generated inputs paired with hard targets.
  • Replay: DGR+distill combines generative replay with distillation by pairing generated input samples with soft targets.The hybrid method is included in the comparison under the name DGR+distill.
  • Stored data and exemplars: Exact replay stores data from previous tasks and replays it to improve classification, but privacy concerns or memory constraints can make storing prior data impossible.Stored data can also serve as exemplars during execution, as in iCaRL, which classifies in a learned feature space using stored-data class means.

4 Experimental Details

The experiments compare continual-learning methods across three scenarios using split and permuted MNIST, with standardized architectures, baselines, and training protocols. The protocols differ in task structure, input transformations, and model configuration.

  • Evaluation: Methods were evaluated across all three continual-learning scenarios on both split and permuted MNIST protocols.The comparison targeted differences between scenarios and performances of recently proposed methods.
  • Task protocols: Split MNIST uses five two-way classification tasks with unprocessed 28x28 grayscale images and standard 60,000/10,000 training/test splits.The dataset contains approximately 6,000 training and 1,000 test images per digit.
  • Task protocols: Permuted MNIST uses ten ten-way classification tasks, with images zero-padded to 32x32 and independently permuted across 1024 pixels.No other preprocessing was performed, and the standard training/test split was retained.
  • Model and outputs: All methods used the same ReLU multilayer perceptron, with two hidden layers of 400 or 1000 nodes for split or permuted MNIST, respectively.Task-IL used multi-headed output layers, while all methods except iCaRL used a final softmax layer.
  • Compared methods: The study included task-gating, parameter-regularization, replay-based methods, iCaRL, and None and Offline baselines representing lower- and upper-bound training.XdG requires test-time task identity, iCaRL is restricted to Class-IL, and the None and Offline baselines correspond to fine-tuning and joint training.
  • Optimization: Split and permuted MNIST models were trained with ADAM for 2000 and 5000 iterations per task, using learning rates 0.001 and 0.0001, respectively.DGR and DGR+distill additionally used a sequentially trained variational autoencoder with generative replay.

5 Results

Results show that continual-learning difficulty and method effectiveness depend strongly on the scenario and task protocol. Replay-based methods were especially important when task identity had to be inferred, while EWC performed competitively in split-MNIST Task-IL under broader hyperparameter tuning.

  • Split MNIST: Split MNIST showed clear scenario differences: all methods performed well in Task-IL, whereas regularization-based methods struggled in Domain-IL and failed in Class-IL.LwF also struggled in Domain-IL, while EWC, Online EWC, and SI completely failed in Class-IL.
  • Split MNIST: Only replay-based methods—DGR, DGR+distill, and iCaRL—achieved good performance above 90% in split-MNIST Domain-IL and Class-IL.The result identifies replay as the successful strategy in the more difficult scenarios.
  • Split MNIST: Across all split-MNIST scenarios, replaying current-task images prevented forgetting previous tasks better than regularization-based methods.For example, replaying ‘2’s and ‘3’s helped preserve recognition of ‘0’s and ‘1’s.
  • Split MNIST: EWC and Online EWC achieved competitive split-MNIST Task-IL performance because the study explored a much wider hyperparameter range than prior reports.The selected hyperparameter values were several orders of magnitude larger than those typically considered.
  • Permuted MNIST: On permuted MNIST, all methods except LwF performed well in Task-IL and Domain-IL, but regularization-based methods failed in Class-IL and only replay-based methods performed well.The Task-IL–Domain-IL difference was small, possibly because task identity was generally used only in the output layer.

6 Discussion … A.2.1 EWC

The paper structures continual-learning comparisons around three scenarios distinguished by task-identity availability, finding class-incremental learning especially difficult for regularization methods and requiring replay-based approaches. It also specifies the losses, distillation targets, and EWC penalty and Fisher-information calculations used in these comparisons.

  • 6 Discussion: Three continual-learning scenarios differ according to whether task identity is provided at test time or must be inferred.These scenarios support more structured comparisons of methods using split and permuted MNIST protocols.
  • 6 Discussion: In class-incremental learning, only replay-based methods produce acceptable results, whereas EWC and SI completely fail even on simple MNIST protocols.The scenario requires inferring task identity rather than receiving it at test time.
  • 6 Discussion: Generative replay performs well on MNIST, but whether it remains successful for more complicated input distributions is an open question.The study notes that MNIST images are relatively easy to generate, while improving generative models are promising for future protocols.
  • A.1.1 Classification: The classification loss for hard target y is Lclassification (x, y; θ) = −log pθ (Y = y|x), with normalization potentially restricted to active output nodes.The active nodes are the classes included in the relevant output distribution.
  • A.1.1 Classification: DGR uses scenario-specific hard-target class sets: replayed classes are task-specific in Task-IL, all classes are available in Domain-IL, and only previous-task classes are available in Class-IL.The supplied passage describes these distinctions when generating hard targets for replayed inputs.
  • A.1.2 Distillation: LwF and DGR+distill use distillation loss by matching the trained model’s probabilities to soft targets produced by a copy saved after the latest task.The soft target is a probability vector over active classes; scenario-specific class distributions are used for replay.
  • A.2.1 EWC: EWC penalizes parameter changes for previous tasks, weighting each quadratic term by parameter importance estimated with the task’s Fisher Information matrix.The calculation can be time-consuming, so using a subset of training data may trade accuracy for speed through NFisher.

A.2.2 Online EWC

Online EWC addresses original EWC’s task-scaling limitation by replacing its growing collection of quadratic penalties with a single penalty anchored at the latest task’s optimum and weighted by accumulated Fisher information. A decay hyperparameter controls how much previous tasks contribute to this running Fisher-information sum.

  • Motivation: Original EWC’s regularization term grows linearly in the number of tasks, increasing computational cost over lifelong learning.The passage identifies this growth as an important limitation for true lifelong learning.
  • Online EWC: Online EWC uses a single quadratic penalty anchored at parameters optimized after the most recent task.The formulation follows from a stricter approximate Bayesian treatment of continual learning.
  • Online EWC: The penalty weights parameters using a running sum of previous tasks’ diagonal Fisher Information matrices, with γ ≤1 controlling gradual decay of earlier contributions.The current task’s diagonal Fisher Information is calculated separately according to equation 6.

A.2.3 SI … A.4.1 Feature Extractor

The methods specify how Synaptic Intelligence estimates parameter importance, how generative replay models are configured and trained, and how iCaRL’s feature extractor handles expanding class outputs and mixed targets. Together, these sections describe regularization, generative replay, and rehearsal-based distillation mechanisms.

  • A.2.3 SI: SI penalizes parameter changes quadratically, with each penalty weighted by the parameter’s estimated importance across previously learned tasks.Importance is estimated from per-parameter contributions to loss changes during training on each task.
  • A.2.3 SI: SI normalizes each task’s parameter contribution by squared parameter change plus dampening ξ = 0.1, then sums contributions across tasks.The dampening term bounds normalized contributions when a parameter’s total change approaches zero.
  • A.3 Generative Model: DGR and DGR+distill use a VAE whose encoder and decoder are multilayer perceptrons with task-dependent hidden-layer widths, a 100-unit latent layer, and a standard normal prior.The hidden layers contain 400 units for split MNIST or 1000 for permuted MNIST, with ReLU nonlinearities.
  • A.3 Generative Model: The VAE combines binary-cross-entropy reconstruction with latent-variable regularization into the per-sample generative loss.The loss is defined as Lgenerative (x; φ, ψ) = Lrecon (x; φ, ψ) + Llatent (x; φ).
  • A.3 Generative Model: The generative model is trained with replay generated by a copy saved after completing the previous task.This mirrors the main model’s use of replay during continual training.
  • A.4 iCaRL: iCaRL uses the classifier architecture as a feature extractor after removing its softmax output layer, and trains its parameters with binary classification and distillation loss.During training only, a sigmoid output layer is appended for these objectives.
  • A.4.1 Feature Extractor: When a new class is encountered, iCaRL adds a corresponding parameter vector wc to its trainable parameter vector θ.The full parameter vector is θ = (φ, w1, ..., wNclasses so far).
  • A.4.1 Feature Extractor: Each task trains iCaRL on current data plus stored data from previous tasks, pairing hard labels with target vectors that combine old-task soft targets and new-task hard targets.The resulting per-sample loss uses these extended target vectors during training on task K.

A.4.2 Selection of Stored Data · A.4.3 Nearest-Class-Mean Classification

iCaRL stores a fixed memory budget by distributing exemplars evenly across seen classes, selecting new exemplars through herding and reducing old class sets to the same per-class limit. It classifies new inputs using a nearest-class-mean rule in feature space based on stored exemplars.

  • A.4.2 Selection of Stored Data: iCaRL assumes that up to B data-points, called exemplars, may be stored in memory.The memory budget is evenly distributed over classes seen so far.
  • A.4.2 Selection of Stored Data: The available memory is evenly divided among seen classes, yielding m stored exemplars per class.The supplied passage defines m as the per-class exemplar allocation.
  • A.4.2 Selection of Stored Data: After training on a task finishes, iCaRL updates the data selected for storage in memory.This update follows the exemplar-set procedures described below.
  • A.4.2 Selection of Stored Data: For each new class, iCaRL iteratively selects m exemplars using herding over extracted feature vectors.Each selected example makes the mean feature vector of selected examples as close as possible to the mean over all available examples in that class.
  • A.4.2 Selection of Stored Data: The exemplar selection procedure defines the new class exemplar sequence as the selected examples for each new class c.The supplied notation identifies the selected exemplars as 1, ..., p_c^m for each new class c.
  • A.4.2 Selection of Stored Data: For each old class, iCaRL discards the last selected exemplars until only m exemplars per class remain.This reduction applies when an existing exemplar-set contains more than m exemplars.
  • A.4.3 Nearest-Class-Mean Classification: iCaRL classifies inputs using a nearest-class-mean rule in feature space based on stored exemplars.The class means are computed from exemplar feature vectors.
  • A.4.3 Nearest-Class-Mean Classification: For each class seen so far, iCaRL computes a mean feature vector from its stored exemplar set before predicting a new input’s label.The supplied notation defines µ_c from exemplar features for c = 1, ..., N_classes so far.

B Using Task Identity in Hidden Layers

For permuted MNIST, Task-IL and Domain-IL results differed only slightly, suggesting test-time task identity was not important for this protocol. This was attributed to most methods using task identity only in the output layer, unlike XdG.

  • Task-IL and Domain-IL showed only small differences on permuted MNIST, suggesting test-time task identity was not important for this protocol.
  • Most methods used task identity only in the network’s output layer, with XdG as the exception.
  • Table B.1 compares separate task-specific output layers with combining each method with XdG in the Task-IL permuted MNIST protocol.

C Replay of Stored Data

Stored data can be replayed during training, used during execution through feature-space nearest-class-mean classification, or used in both ways. In Class-IL, exact replay methods surpassed regularization with minimal storage but required substantially more stored data to match generative replay, and DGR+distill remained superior on permuted MNIST even with 50,000 examples.

  • Ways of Using Stored Data: Stored data supports replay during training, nearest-class-mean classification during execution, or both.The nearest-class-mean rule operates in feature space with class means calculated from stored data.
  • Class-IL Results: In Class-IL, storing one example per class was enough for every exact replay method to outperform all regularization-based methods on both task protocols.This result held for both split MNIST and permuted MNIST.
  • Class-IL Results: Exact replay required substantially more stored data to match generative replay.The passage reports this requirement across the evaluated Class-IL task protocols.
  • Class-IL Results: 50,000 stored examples still left exact replay variants consistently outperformed by DGR+distill on permuted MNIST.This comparison specifically concerns the permuted MNIST protocol.

D Hyperparameters

The section argues that continual-learning hyperparameters should not be tuned with validation data from tasks seen throughout training, because this can create an unfair advantage. Nevertheless, generative-replay methods receive grid searches, whose combinations are evaluated on task test sets rather than validation sets.

  • Hyperparameter selection: Hyperparameters are typically selected by training models across values on the training set and choosing the best performance on a separate validation set.The section adapts this standard strategy to continual learning by considering training across the full task protocol.
  • Hyperparameter selection: Continuously using each task’s validation data during incremental training can produce an unfair advantage.The cited example sets a bias-removal parameter after each task to optimize validation performance across all tasks seen so far.
  • Hyperparameter selection: Generative-replay methods undergo grid searches to give them the best possible chance and assess hyperparameter influence.The searches are reported for both split MNIST and permuted MNIST protocols.
  • Hyperparameter selection: All tested hyperparameter combinations are evaluated using task test sets rather than validation sets.Each grid-search experiment is run once; the passage explains this choice follows the concern about validation-set use.
Loading 1904.07734v1…