PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative replay with feedback connections as a general strategy for continual learning

Gido M. van de Ven, Andreas S. Tolias

arXiv:1809.10635v2cs.LGcs.AIcs.CVstat.ML

TL;DR

Continual-learning systems can catastrophically forget earlier tasks, while varied evaluation protocols hinder direct comparison. The paper defines three task-identity scenarios, compares methods on MNIST protocols, and integrates generative replay into the main model. Generative replay with distillation performs best across scenarios, while Replay-through-Feedback reduces training time with no or negligible performance loss.

  • Problem

    Artificial neural networks quickly forget earlier tasks, and differing continual-learning evaluation protocols make methods difficult to compare directly.

  • Method

    The paper defines three task-identity scenarios, compares continual-learning methods on split and permuted MNIST, and integrates generative feedback connections into the main model.

  • Results

    Generative replay with distillation achieved superior performance across all three scenarios, while Replay-through-Feedback substantially reduced training time with no or negligible performance loss.

  • Takeaways & Limitations

    Generative replay is presented as a promising general strategy for lifelong learning, with Replay-through-Feedback addressing its computational cost.

  • Takeaways & Limitations

    The study’s inputs are relatively easy MNIST digits, leaving scalability to more complicated task protocols for future empirical work.

Abstract

from arXiv · show

A major obstacle to developing artificial intelligence applications capable of true lifelong learning is that artificial neural networks quickly or catastrophically forget previously learned tasks when trained on a new one. Numerous methods for alleviating catastrophic forgetting are currently being proposed, but differences in evaluation protocols make it difficult to directly compare their performance. To enable more meaningful comparisons, here we identified three distinct scenarios for continual learning based on whether task identity is known and, if it is not, whether it needs to be inferred. Performing the split and permuted MNIST task protocols according to each of these scenarios, we found that regularization-based approaches (e.g., elastic weight consolidation) failed when task identity needed to be inferred. In contrast, generative replay combined with distillation (i.e., using class probabilities as "soft targets") achieved superior performance in all three scenarios. Addressing the issue of efficiency, we reduced the computational cost of generative replay by integrating the generative model into the main model by equipping it with generative feedback or backward connections. This Replay-through-Feedback approach substantially shortened training time with no or negligible loss in performance. We believe this to be an important first step towards making the powerful technique of generative replay scalable to real-world continual learning applications.

1 Introduction

The paper frames catastrophic forgetting and inconsistent evaluation as central obstacles to continual learning. It compares methods across defined scenarios and proposes Replay-through-Feedback to reduce generative replay’s computational cost.

  • Artificial neural networks rapidly forget previously learned tasks when trained on new ones, a phenomenon called catastrophic forgetting.
  • Different experimental protocols make direct comparison of continual-learning methods difficult.
  • The paper identifies three continual-learning scenarios distinguished by whether task identity is provided or must be inferred.
  • Generative replay, especially combined with distillation, performs well across all three scenarios but can be computationally costly.
  • Replay-through-Feedback merges the generative model into the main model using feedback or backward connections, substantially reducing training time with no or negligible performance loss.

2 Continual learning scenarios

The paper distinguishes three continual-learning scenarios by the availability and role of task identity at test time. They range from task-informed learning to learning that also requires task inference.

  • Continual learning involves sequentially learning tasks without storing raw data, but evaluation protocols differ in task-identity requirements.
  • Task-incremental learning (Task-IL): Task-incremental learning provides task identity and permits task-specific components such as multi-headed output layers.
  • Domain-incremental learning (Domain-IL): Domain-incremental learning withholds task identity, while requiring only solution of the task at hand.
  • Class-incremental learning (Class-IL): Class-incremental learning requires models to solve previously seen tasks and infer which task they are presented with.

3 Continual learning strategies

The paper groups continual-learning strategies into parameter constraints, task-specific subnetworks, replay-based data modification, and exact replay. Replay methods use pseudo-data or generated inputs because storing raw data is not allowed in this study.

  • Catastrophic forgetting can be addressed by restricting parameter optimization or modifying training data to represent previous tasks.
  • Not optimizing entire network: Task-specific subnetworks select different network components for each task but require task identity, limiting them to Task-IL.
  • Not optimizing entire network: EWC and SI regularize parameters according to their estimated importance for previously learned tasks, slowing future changes to them.
  • Modifying training data: Replay complements new-task data with pseudo-data representative of previous tasks, including current-task inputs labeled by a previous model in LwF.
  • Modifying training data: Deep Generative Replay trains a separate generative model to produce previous-task inputs, while DGR+distill pairs generated samples with soft targets.
  • Modifying training data: Exact replay stores previous-task examples, but privacy concerns or memory constraints can make this infeasible; this study disallows raw-data storage.

4 Experimental details

The experiments compare continual-learning methods on split and permuted MNIST across three scenarios, using common architectures and training protocols. Methods include regularization, replay, generative replay, and joint-training baselines.

  • Task protocols: Both split and permuted MNIST were evaluated under all three continual-learning scenarios to assess the role of task identity.Split MNIST used five two-digit tasks; permuted MNIST used ten tasks with independently permuted pixels.
  • Task protocols: The split MNIST protocol used five two-digit classification tasks from 28x28 grayscale images, with 60,000 training and 10,000 test images.
  • Task protocols: The permuted MNIST protocol used ten tasks containing all ten digits, with a different random permutation of zero-padded 32x32 image pixels per task.
  • Implementation: All methods used the same neural-network architecture within each protocol, with multi-headed outputs in the Task-IL scenario.Split MNIST used two hidden layers of 400 nodes; permuted MNIST used two hidden layers of 1000 nodes.
  • Training: Regularization methods optimized Ltotal = Lcurrent + λLregularization, while replay methods added replay examples to the current-task loss.Training used ADAM with 2000 iterations per split-MNIST task and 5000 per permuted-MNIST task.
  • Compared methods: The comparison included fine-tuning, XdG, EWC, online EWC, SI, LwF, DGR, DGR+distill, and offline joint training.DGR used hard targets from a previous model, whereas DGR+distill used soft targets.
  • Compared methods: Generative replay used a separately trained variational autoencoder, while Replay-through-Feedback was not part of these listed experimental methods.The generative model used a stochastic latent layer of size 100 and was itself trained with generative replay.
  • Baselines: Offline training used all data seen so far and served as an upper-bound reference, whereas fine-tuning served as a lower-bound reference.

5 Results

Across split and permuted MNIST, generative replay methods were the only approaches successful in Class-IL, with DGR+distill consistently outperforming DGR. Replay-through-Feedback remained competitive while reducing training cost, though the study notes scope limitations for more complex inputs.

  • Split MNIST: Only generative-replay methods retained good performance above 90% in split-MNIST Class-IL, and DGR+distill outperformed DGR in all scenarios.
  • Permuted MNIST: On permuted MNIST, regularization methods performed reasonably in Task-IL and Domain-IL but failed in Class-IL, while generative replay succeeded.DGR+distill again outperformed DGR in every scenario.
  • Method comparison: In Task-IL, XdG reduced catastrophic forgetting on both protocols but was outperformed by SI and therefore by DGR+distill.
  • Method comparison: EWC and online EWC could have performed better with an extremely large hyperparameter, but the authors state they would still remain substantially below DGR+distill.

6 Replay-through-Feedback (RtF)

Replay-through-Feedback integrates generative replay into the main model through feedback connections, addressing the computational cost of training a separate generative model. It substantially reduced training time while maintaining comparable performance.

  • Generative replay was computationally expensive partly because it required training a separate generative model.In the experiments, DGR and DGR+distill took roughly twice as long to train as SI.
  • RtF integrates the generative model into the main model using feedback connections trained for generative capability.The model adds feedback connections that reconstruct inputs from hidden representations and stochastic latent variables with a known sampling distribution.
  • RtF trains current-task data with generative and classification losses, while replayed data uses generative and distillation losses.The loss terms are weighted according to how many tasks the model has seen.
  • RtF slightly outperformed DGR+distill on split MNIST but performed slightly worse on permuted MNIST.The differences were relatively small, and RtF still outperformed all other tested methods.
  • RtF substantially reduced training time, almost halving it in most experiments, with no or negligible performance loss.Training time was consistently longest for DGR and DGR+distill and was reduced by RtF.

7 Discussion

The discussion emphasizes continual learning’s need for common benchmarks, efficient generative replay, and caution about whether MNIST-based evidence generalizes to more complex inputs.

  • Continual learning lacks common benchmarks despite reuse of the same datasets, making direct comparisons difficult.
  • Generative replay with distillation consistently outperformed other tested methods across the evaluated scenarios.
  • The study’s evidence is limited because generating MNIST digits is relatively easy, leaving scalability to more complicated inputs for future empirical work.
  • Generative replay can be computationally costly, so continual-learning methods should also be evaluated by training time, especially in lifelong or real-time settings.
  • Replayed samples need not be perfect: task 1 was not forgotten even when permuted-MNIST replay quality substantially declined.The authors hypothesize that distillation makes generative replay more robust to imperfect replayed inputs.
  • Replay-through-Feedback merges the generator into the main model and substantially reduces training time with no or negligible performance loss.

A.2.1 EWC

EWC protects parameters associated with previous tasks by penalizing changes according to estimated parameter importance. Its empirical Fisher calculation is an approximation and can be time-consuming.

  • EWC applies a quadratic penalty to parameter changes from values recorded after previously learned tasks.Each task contributes a separate regularization term.
  • Penalty strength depends on how important each parameter is estimated to be for each previous task.More important parameters receive higher penalties.
  • EWC estimates parameter importance using diagonal elements of each task’s Fisher Information matrix.
  • The implementation calculates these terms with the diagonal empirical Fisher using the task’s training data.This is presented as an approximation under a near-perfect-prediction assumption.
  • Fisher Information calculation is time-consuming, particularly when tasks contain much training data.Using only a subset of training data can trade accuracy for speed.

A.2.2 Online EWC

Online EWC replaces the original task-growing regularization with one quadratic penalty anchored at the most recent task, using accumulated Fisher information to preserve earlier knowledge.

  • A.2.2 Online EWC: Online EWC uses one quadratic penalty term rather than a separate term for every previous task.This avoids the original formulation’s linearly growing number of regularization terms.
  • A.2.2 Online EWC: The penalty is anchored at the parameters obtained after the most recent task.
  • A.2.2 Online EWC: Previous tasks’ parameter importance is represented by a running sum of diagonal Fisher Information values, with decay controlled by γ ≤1.
  • A.2.2 Online EWC: The Fisher Information can be calculated using model-predicted labels or by sampling multiple labels from the model’s conditional distribution.
  • A.2.2 Online EWC: Reported results do not depend much on the choice of Fisher Information calculation.

A.2.3 SI

Synaptic intelligence estimates parameter importance from each task’s loss changes and uses that importance to penalize later parameter deviations from the previous task’s solution.

  • A.2.3 SI: SI uses one quadratic regularization term that penalizes changes away from parameters’ values after the previous task.Penalty strength depends on each parameter’s estimated importance for tasks learned so far.
  • A.2.3 SI: SI computes per-parameter contributions to the loss change during every training task.
  • A.2.3 SI: These contributions are normalized by squared total parameter change plus a dampening term ξ set to 0.1, then summed across tasks.
  • A.2.3 SI: The total parameter change is defined as the difference between a parameter’s value after training and its value before training on task k.
  • A.2.3 SI: The resulting importance estimate is used to form SI’s regularization term during training on the next task.

B Hyperparameters

The paper argues that conventional hyperparameter validation can violate continual-learning constraints, so it uses test-set grid searches for generative-replay methods instead.

  • B Hyperparameters: Selecting hyperparameters with validation data from all tasks violates the principle that each task may be visited only once and sequentially.
  • B Hyperparameters: The paper therefore questions using task validation sets continuously throughout incremental training.
  • B Hyperparameters: Generative-replay hyperparameters were grid-searched to give those methods the best possible chance and assess hyperparameter influence.
  • B Hyperparameters: Each grid-search experiment was run once, with performance evaluated using the task test sets rather than validation sets.
  • B Hyperparameters: Figure 7 reports average test accuracy over 5 split-MNIST tasks for tested hyperparameter combinations.
  • B Hyperparameters: Figure 8 reports average test accuracy over 10 permuted-MNIST tasks for tested hyperparameter combinations.

C Additional discussion: similar methods & storing data

The discussion relates Replay-through-Feedback to FearNet and clarifies that the paper deliberately excludes methods storing previous data, despite their potential performance benefits.

  • C Additional discussion: similar methods & storing data: FearNet’s mPFC network also combines a generative autoencoder with classification capability and generates pseudo-examples of earlier tasks.
  • C Additional discussion: similar methods & storing data: Replay-through-Feedback differs from FearNet because the paper describes its generative model as a variational autoencoder rather than FearNet’s autoencoder.
  • C Additional discussion: similar methods & storing data: The paper considers only methods that do not store data because storage may be impossible under privacy or memory constraints.
  • C Additional discussion: similar methods & storing data: When storage is possible, methods such as iCaRL and FearNet can substantially boost performance, especially for class-incremental learning.
  • C Additional discussion: similar methods & storing data: FearNet stores previous-task data in the hippocampus until the model sleeps, reportedly every ten tasks for its main results.
Loading 1809.10635v2…