PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Continual Learning with Maximally Interfered Retrieval

Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, Tinne Tuytelaars

arXiv:1908.04742v3cs.LGstat.ML

TL;DR

Online continual learning must address catastrophic forgetting in a never-ending data stream, while replay methods commonly sample memories randomly. MIR instead retrieves the most interfered samples for replay, improving performance and reducing forgetting across multiple settings.

  • Problem

    Online continual learning must overcome catastrophic forgetting while learning from a non-stationary, never-ending data stream.

  • Method

    MIR retrieves samples most interfered by an estimated incoming update, with variants for stored memories and generative models.

  • Results

    MIR reduces forgetting and improves performance over random sampling and standard baselines across multiple online continual-learning settings.

  • Takeaways & Limitations

    Interfered-sample retrieval is a relevant replay criterion for online continual learning, while generative modeling remains challenging on harder datasets.

  • Takeaways & Limitations

    Generative replay remains unsuitable for CIFAR-10 under the current state of generative modeling.

Abstract

from arXiv · show

Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work, we consider a controlled sampling of memories for replay. We retrieve the samples which are most interfered, i.e. whose prediction will be most negatively impacted by the foreseen parameters update. We show a formulation for this sampling criterion in both the generative replay and the experience replay setting, producing consistent gains in performance and greatly reduced forgetting. We release an implementation of our method at https://github.com/optimass/Maximally_Interfered_Retrieval.

1 Introduction

Continual learning seeks systems that accumulate knowledge across an unbounded sequence of tasks, but neural networks suffer catastrophic forgetting. This work targets the harder online, non-i.i.d. setting with Maximally Interfered Retrieval, replaying memories whose loss is expected to increase most after an update.

  • Motivation: Continual learning aims to extend neural networks beyond narrow-task performance by enabling continual learning and accumulation of knowledge across an unlimited number of tasks.The motivation is framed relative to human intelligence and lifelong learning.
  • Problem: Catastrophic forgetting of previously seen data remains a central challenge, while many solutions simplify continual learning into task-incremental settings with clear task boundaries and offline training.Examples include learning handwritten digit recognition and vehicle recognition as separate tasks.
  • Setting: The paper studies online continual learning where non-i.i.d. samples are seen only once, a harder and more realistic setting than incremental tasks, and focuses on replay-based methods.The authors note that this setting can arise in applications such as social media.
  • Contribution: Maximally Interfered Retrieval retrieves replay samples whose loss is expected to increase under the model’s estimated parameter update, with variants for stored memories and generative models.The method is evaluated against random sampling and strong baselines.

2 Related work

Continual learning is challenged by catastrophic forgetting and the stability/plasticity dilemma. Related work addresses forgetting through replay or prior-based regularization, while this paper focuses on improving rehearsal sample selection beyond random sampling.

  • Challenges: Catastrophic forgetting of previous knowledge after acquiring new knowledge is a central continual-learning challenge linked to the stability/plasticity dilemma.These problems occur in both biological and artificial neural networks.
  • Method families: Under fixed architectures, methods primarily replay past or generated samples, or encode prior-task knowledge to regularize learning of new tasks.The passage identifies replay-based and prior-focused families as the two main streams.
  • Rehearsal: Replayed samples support either constraining parameter updates or rehearsal; this work adopts rehearsal because it is cheaper and effective compared with constraint optimization.Rehearsal uses samples from previous history, including buffer samples or pseudo-samples from a generative model.
  • Replay selection: Random replay selection is contrasted with choosing candidates according to interference with the estimated update to improve learning behavior and reduce interference.The proposed strategy targets better replay candidates than random sampling.
  • Generative modeling: Prior work studied continual generative modeling and compressed representations, while this work considers generative modeling for the first time in online continual learning.Riemer et al. used an autoencoder to store compressed representations instead of raw samples.

3 Methods

MIR retrieves replay samples whose losses are predicted to increase most after the incoming-batch update, rather than selecting previous samples randomly. The method is instantiated for experience replay and generative replay, with diversity and confidence mechanisms supporting retrieval.

  • Core principle: MIR selects previous samples expected to suffer the greatest loss increase under the virtual parameter update caused by incoming data.The objective is to learn on new samples without increasing loss on previously observed samples.
  • Experience replay: In experience replay, MIR scores memory samples using sMI-1(x) = l(fθv(x), y) − l(fθ(x), y) or sMI-2(x) = l(fθv(x), y) − min.The virtual update is θv = θ − α∇L(fθ(Xt), Yt), and the top-k samples are retrieved by score.
  • Experience replay: ER-MIR first randomly samples C memory items with C > B, then applies the interference criterion to retrieve the budget B samples.The initial subset encourages diversity and reduces the computational cost of searching the full memory.
  • Generative replay: Generative MIR optimizes feature-space points to maximize the loss difference before and after the estimated classifier update, while enforcing diversity among retrieved points.The points are decoded into input samples, and retrieval is initialized from the encoder representation of incoming data.
  • Generative replay: For generated samples, prior-model predictions provide pseudo labels and KL divergence between prior and updated predictions serves as an interference proxy.The method also applies an entropy penalty to encourage generated points on which the previous model is confident, reducing noise from blurry or mixed-category generations.
  • Generative replay: MIR can retrieve samples that would interfere with the generator itself by evaluating the change in the VAE’s evidence lower bound under virtual encoder and decoder updates.This extends interference-based retrieval from the classifier to generator parameters (φ, γ).

4 Experiments

Experiments evaluate MIR in shared-classifier online continual learning across experience and generative replay, using standard datasets and accuracy and forgetting as primary metrics. MIR improves replay performance, including on longer task sequences and generative modeling, while challenging CIFAR-10 generative replay motivates a hybrid approach.

  • Experimental setup: Experiments compare MIR with random sampling in experience and generative replay across MNIST Split, Permuted MNIST, CIFAR-10 Split, and MiniImagenet Split.The evaluations use shared classifiers, where predictions span classes from all tasks without task metadata at inference time.
  • Experience and generative replay: 2.8% accuracy and 2.5% forgetting are the MNIST Split improvements achieved by MIR over the baseline.For MNIST, generative replay methods were hyperparameter-searched for additional iterations so GEN and GEN-MIR could reach their optimal performance.
  • Experience and generative replay: GEN-MIR achieves the best performance over all models on Permuted MNIST, despite generative replay requiring more tuning and being unable to store past data.The result is reported as an advantage over the generative replay baselines.
  • Generative modeling: The generator’s loss is significantly lower when rehearsing on maximally interfered samples rather than random samples on both evaluated datasets.This extends the reported benefit of MIR beyond supervised classification to online continual generative modeling.
  • Ablation study: Minimizing label entropy H(ypre) is identified as the most important ablation component and essential for outperforming the baseline.The criterion encourages confidence in the previous classifier’s prediction for each retrieved sample.
  • Limitations: Generative replay is not viable for CIFAR-10 under the current state of generative modeling, motivating the hybrid approach introduced next.The authors report reaching the same conclusion as prior work on challenging continual generative modeling datasets.

5 Conclusion

The paper proposes retrieving interfered memories for online continual learning, showing reduced forgetting and improved performance over random sampling and standard baselines. It also examines generative modeling and encoded memories in this setting.

  • 5 Conclusion: The paper proposes and studies a criterion for retrieving relevant memories in online continual learning.The criterion targets memories affected by interference.
  • 5 Conclusion: Retrieving interfered samples reduces forgetting and significantly improves performance over random sampling and standard baselines across multiple settings.The conclusion reports consistent benefits across a number of settings.
  • 5 Conclusion: The results and analysis illuminate the feasibility and challenges of generative modeling for online continual learning.The paper explicitly frames generative modeling as both feasible and challenging in this setting.
  • 5 Conclusion: The paper reports a first result using encoded memories for online continual learning.The supplied conclusion passage truncates immediately after introducing this result.

A Full Results on ER-MIR

Full ER-MIR results cover varying buffer sizes on Permuted MNIST and MNIST Split, plus CIFAR-10 with 1000 samples per task. Gains are smaller in this reduced-data CIFAR-10 setting than with 9750 samples per task, suggesting greater effectiveness in more challenging settings.

  • A Full Results on ER-MIR: ER-MIR is evaluated across buffer sizes on Permuted MNIST and MNIST Split, and on CIFAR-10 with 1000 samples per task.The CIFAR-10 setting follows prior study.
  • A Full Results on ER-MIR: Margins of gain are lower for ER-MIR with 1000 CIFAR-10 samples per task than with 9750 samples per task, indicating greater effectiveness in the more challenging setting.The passage contrasts the reduced-data and full CIFAR-10 settings.

B.1 Experience Replay Experiments

Experience replay experiments use established base settings, with dataset-specific learning rates and architectures. ER-reservoir-MIR additionally tunes an initial sampling-size hyperparameter.

  • ER and ER-MIR use batch sizes of 10 for incoming samples and 10 for buffered samples, following the base settings in [8].
  • MNIST experiments use a learning rate of 0.05, while CIFAR-10 experiments select 0.1 by validation.
  • ER-reservoir-MIR selects the initial sampling size C from 30, 50, 100, and 150, choosing 50.
  • MNIST uses a 2 layer MLP with 400 hidden nodes, whereas CIFAR-10 uses a standard Resnet-18.

B.2 Generative Modeling Experiments

The generative replay experiments used the same classifier and incoming-data batch size as the experience replay experiments, with an MLP VAE and matched computational budgets for the baseline and proposed method.

  • The experiments reused a 2-layer MLP classifier with 400 hidden nodes to facilitate comparison with experience replay.
  • Incoming data were processed with batch size 10, and the VAE was also an MLP.
  • Hyperparameter search varied learning rate among 5e-1, 1e-1, and 1e-2 and updates per incoming datapoint among 2, 5, 10, and 15.
  • The baseline and proposed method received equal numbers of runs, while settings were chosen so additional computation did not improve performance.The passage notes that more training can lead to more forgetting.

C Further Description of Hybrid Approach

The hybrid approach is specified by an AE-MIR algorithm and evaluated with an offline-trained generator in an incremental setting. Ablations show that training the classifier only on autoencoded images is essential, while autoencoding test inputs provides an additional performance gain.

  • Algorithm 3 specifies AE-MIR with learning rate α, subset size C, budget B, and a generator as inputs.
  • The generator is trained offline for 5 epochs in the incremental setting, using batch size 10 and averaging results over 5 runs.
  • Ablation Study: The ablation replaces autoencoded test images with real images in “- test AE” and also replaces reconstructed training images with real stream images in “- train & test AE”.
  • Ablation Study: Never training the classifier on real images is essential, because “- train & test AE” performs badly.
  • Ablation Study: Autoencoding inputs at test time adds a small but noticeable performance increase from “- test AE” to “AE-Random”.
Loading 1908.04742v3…