PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Experience Replay: a Bag of Tricks for Continual Learning

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Simone Calderara

arXiv:2010.05595v1cs.LGstat.ML

TL;DR

Continual Learning must accommodate shifting data streams while retaining previously acquired knowledge under bounded resources, but catastrophic forgetting makes this difficult. The paper enhances Experience Replay with five modifications and finds that the resulting method outperforms state-of-the-art rehearsal methods across four Class-IL settings despite limited additional computational requirements.

  • Problem

    Continual Learning seeks to learn from shifting data streams while retaining old knowledge under bounded computational and memory resources, but fine-tuning on new classes quickly disrupts previously acquired knowledge.

  • Method

    The paper enhances Experience Replay, which interleaves past examples with current training batches, using five modifications that address shortcomings of naive replay.

  • Results

    ER equipped with these tricks outperforms state-of-the-art rehearsal methods on four distinct Class-IL experimental settings.

  • Takeaways & Limitations

    Despite limited growth in computational requirements, the enhanced ER provides a reliable baseline for Continual Learning practitioners and can benefit some prior-based methods.

Abstract

from arXiv · show

In Continual Learning, a Neural Network is trained on a stream of data whose distribution shifts over time. Under these assumptions, it is especially challenging to improve on classes appearing later in the stream while remaining accurate on previous ones. This is due to the infamous problem of catastrophic forgetting, which causes a quick performance degradation when the classifier focuses on learning new categories. Recent literature proposed various approaches to tackle this issue, often resorting to very sophisticated techniques. In this work, we show that naive rehearsal can be patched to achieve similar performance. We point out some shortcomings that restrain Experience Replay (ER) and propose five tricks to mitigate them. Experiments show that ER, thus enhanced, displays an accuracy gain of 51.2 and 26.9 percentage points on the CIFAR-10 and CIFAR-100 datasets respectively (memory buffer size 1000). As a result, it surpasses current state-of-the-art rehearsal-based methods.

I. INTRODUCTION

Continual Learning addresses catastrophic forgetting when class distributions shift over time. The paper argues that five simple modifications can substantially strengthen Experience Replay and outperform sophisticated rehearsal methods.

  • Continual Learning must learn sequentially arriving classes while retaining prior knowledge under bounded computational resources and memory.
  • Experience Replay interleaves stored past examples with current batches, offering a straightforward rehearsal baseline for mitigating forgetting.
  • Naive rehearsal can overfit its small buffer and become biased toward newer classes during incremental learning.
  • Reservoir sampling may omit classes when the buffer is small, limiting the representativeness of replayed data.
  • The paper introduces five easily applied modifications and evaluates their effects across four Class-IL experimental settings.
  • Enhanced Experience Replay outperforms state-of-the-art rehearsal methods while providing a reliable baseline for Continual Learning practitioners.

II. RELATED WORKS

Prior Continual Learning methods either protect past parameters or rehearse stored examples, often with substantial complexity or resource demands. The paper returns to Experience Replay and argues that simple enhancements can outperform more elaborate alternatives.

  • Experience Replay stores previous examples and interleaves them with new batches, yet earlier work often lacked direct experimental comparison against it.
  • Continual Learning research broadly separates methods that store past network parameters from methods that store past examples.
  • Parameter-based approaches limit interference through separate models, architectural adaptation, or losses that constrain later weight changes.
  • Rehearsal methods retain exemplars from earlier tasks, using strategies such as nearest-mean classification or memory-based constraints.
  • Generative alternatives avoid a literal memory buffer but require online generator training, while the paper describes ER as more effective and viable.
  • The paper tests whether ER equipped with its proposed tricks can perform significantly better than other approaches.

III. BASELINE METHOD

The baseline method trains on sequential class partitions by approximating joint-data learning with rehearsal. Experience Replay stores exemplars from prior tasks and merges sampled memories with each current batch.

  • Class Incremental Learning trains a classifier on samples x with class labels y from a dataset D by minimizing a classification loss.
  • In Continual Learning, only one task-specific class partition is observed at a time, preventing direct optimization over the complete dataset.
  • Experience Replay stores examples and labels from previous tasks in a buffer B, then merges sampled items with the current training batch.
  • The replay objective serves as a surrogate for the loss that would be optimized using the full dataset.
  • The method adds buffer size and replay-sample count as two hyperparameters; reservoir sampling gives each input exemplar probability |B|/|D| of entering the buffer.

IV. TRAINING TRICKS

The section presents training and replay-buffer tricks that address augmentation overfitting and task-induced prediction bias in Class-IL. Independent buffer augmentation varies replayed examples, while Bias Correction adjusts outputs using replay-buffer exemplars.

  • A. Independent Buffer Augmentation: Independent Buffer Augmentation stores unaugmented examples so replayed items can receive distinct transformations later.This reduces overfitting to the memory and increases variety in rehearsal examples.
  • A. Independent Buffer Augmentation: Storing augmented examples and replaying them unchanged can limit the benefit of rehearsal diversity.The authors contrast this practice with applying distinct transformations when replay examples are shown.
  • B. Bias Control (BiC): Class-IL predictions become biased toward classes from the current task, reflecting a whole-model imbalance rather than only a faulty final layer.Zeroing the final classification layer is therefore not beneficial.
  • B. Bias Control (BiC): Bias Correction uses a two-parameter linear model downstream of the classifier to compensate task-related output bias.The parameters α and β are trained after each task using a limited amount of replay-buffer exemplars and cross-entropy loss.
  • B. Bias Control (BiC): The authors report that this simple linear model is effective for countering the identified prediction bias.They adopt the modular solution proposed by Wu et al.

C. Exponential LR Decay (ELRD)

Exponential LR Decay progressively lowers the learning rate throughout training rather than restarting the schedule at each task. The schedule is intended to reduce forgetting while adding a regularization effect without the overhead of explicit interference-prevention losses.

  • C. Exponential LR Decay (ELRD): Exponential LR Decay progressively decreases the learning rate at each iteration to help preserve previous knowledge.The authors describe decreasing the learning rate as an explicit training trick for continual learning.
  • C. Exponential LR Decay (ELRD): The proposed decay runs for the whole training duration, unlike task-wise schedules that restart at each task.The authors report that this continuous schedule relieves catastrophic forgetting.
  • C. Exponential LR Decay (ELRD): The learning rate follows lr_j = lr_0 · γ^N_ex, where N_ex counts examples seen, lr_0 is the initial rate, and γ is tuned for the target final value.The target is approximately 1/6 of the initial learning rate at the end of training.
  • C. Exponential LR Decay (ELRD): Decreasing the learning rate also regularizes training by penalizing weight changes between successive steps.The authors relate this effect to non-rehearsal methods but state that ELrD avoids their additional overhead.

D. Balanced Reservoir Sampling (BRS)

Balanced Reservoir Sampling modifies reservoir replacement to encourage equal class representation in a fixed-size buffer. On a six-class toy dataset, it more closely approaches the target allocation and achieves lower sampling error than plain reservoir.

  • D. Balanced Reservoir Sampling (BRS): Plain reservoir sampling gives each streamed exemplar equal representation probability but can omit classes when the buffer is small.This omission becomes especially critical when buffer capacity is limited.
  • D. Balanced Reservoir Sampling (BRS): When |B| is approximately C, the probability of leaving at least one class out rises from 0.25 for C = 2 to approximately 0.367 as C approaches infinity.For C = 10, the reported probability is 0.349.
  • D. Balanced Reservoir Sampling (BRS): BRS retains buffer capacity more effectively than ring-buffer and herding alternatives that reserve class-specific slices.The cited alternatives can leave capacity unused or incur computational overhead.
  • D. Balanced Reservoir Sampling (BRS): Balanced Reservoir Sampling replaces an exemplar from the most represented class instead of selecting a random exemplar for removal.This small reservoir modification encourages balance in the number of exemplars per class.
  • D. Balanced Reservoir Sampling (BRS): On a 1020-item, six-class toy dataset with a 12-item buffer, BRS achieves Mean Squared Error 0.28 versus 1.64 for plain reservoir.The target is exactly 2 items per class, and BRS is reported to be closer to that ideal.

E. Loss-Aware Reservoir Sampling (LARS)

Loss-Aware Reservoir Sampling uses stored training losses to retain examples that remain difficult for the model, while combining this criterion with class balancing. The combined method forms a normalized score from class-balance and loss-based terms.

  • E. Loss-Aware Reservoir Sampling (LARS): Loss-aware replacement discards less important examples using their recorded loss score.The method uses training loss as a simpler criterion than gradient comparisons for modeling exemplar importance.
  • E. Loss-Aware Reservoir Sampling (LARS): Figure 3 compares reservoir, Balanced Reservoir, and Loss-Aware Reservoir graphically.The caption identifies the three sampling strategies as the figure’s comparison set.
  • E. Loss-Aware Reservoir Sampling (LARS): The expected buffer loss can be computed without backpropagation and maximized to retain exemplars that have not been fit.This provides the stated rationale for preferring higher-loss examples.
  • E. Loss-Aware Reservoir Sampling (LARS): Loss values are stored with buffer examples and refreshed when those examples are replayed.Because the loss is a scalar, the resulting memory overhead is described as negligible relative to storing the example.
  • E. Loss-Aware Reservoir Sampling (LARS): Loss-Aware Reservoir and BRS are combined because they address separate, complementary issues.The combined algorithm constructs balance and loss score terms, normalizes them for equal contribution, and sums them into a single score.

V. EXPERIMENTAL RESULTS

The proposal is evaluated on four Class-IL benchmarks of increasing difficulty, using dataset-specific task structures and standard training protocols. Results are reported as average accuracy over all tasks after training, averaged across ten runs.

  • Evaluation settings: The evaluation covers Split Fashion-MNIST, Split CIFAR-10, Split CIFAR-100, and Split CORe-50, ordered by increasing difficulty.The benchmarks contain five, five, ten, and nine tasks respectively.
  • Evaluation settings: Split Fashion-MNIST and Split CIFAR-10 each contain five two-class tasks, with 6000 and 5000 exemplars per class respectively.
  • Evaluation settings: Split CIFAR-100 contains ten tasks with ten classes each and 500 exemplars per class.
  • Evaluation settings: Split CORe-50 contains 50 classes organized into nine tasks, beginning with ten classes and continuing with five classes per task.
  • Evaluation protocol: All results use average accuracy over tasks at the end of training, with ten repetitions averaged and equal batch sizes across methods.

B. Comparison with State of the Art

ER equipped with the proposed tricks is compared with rehearsal-based state-of-the-art methods across several Class-IL datasets and buffer sizes. It consistently surpasses competitors on Fashion-MNIST, harder CIFAR settings, and CORe-50, while some baselines show dataset-specific weaknesses.

  • Comparison setup: ER+T is compared with iCaRL, GEM, A-GEM, and HAL using average final accuracy across several datasets and buffer sizes.ER uses reservoir sampling, while GEM, A-GEM, and HAL use ring buffers and iCaRL uses herding.
  • Split Fashion-MNIST: On Split Fashion-MNIST, ER+T consistently surpasses all competitors.Plain ER is already strong; GEM and A-GEM perform worse, while iCaRL’s advantage appears only at memory size 200.
  • Split CIFAR datasets: On Split CIFAR-10 and Split CIFAR-100, ER+T gains an edge over the competition despite naive ER being surpassed by GEM and iCaRL.iCaRL performs strongly on these harder datasets, while HAL encounters failures on Split CIFAR-100.
  • Split CORe-50: On CORe-50, the tricks boost ER over naive replay and outperform the other approaches.The authors attribute iCaRL’s unreliable performance to fine-grained class differences that challenge nearest-mean-of-exemplars classification.

C. Influence of Each Trick

The experiments examine the contribution of individual tricks, their combinations, their effects on different methods, and their extension beyond rehearsal. The largest reported gains occur on CIFAR-10 and CIFAR-100, while bias correction has a stated applicability condition.

  • Individual tricks: IBA provides a meaningful accuracy boost on CIFAR-10 and CIFAR-100, almost doubling the initial accuracy on CIFAR-10.
  • Individual tricks: Bias Control and Exponential learning rate Decay have positive effects, especially on the two most difficult settings.
  • Sampling strategies: The combined BRS and LARS sampling strategies are particularly beneficial on Fashion-MNIST and CIFAR-10.
  • Overall effect: The proposed tricks produce performance boosts of +146% on CIFAR-10 and +120% on CIFAR-100.
  • Applicability to other methods: Data augmentation on buffer points benefits methods inconsistently: HAL improves consistently, whereas GEM suffers severe degradation on Split CIFAR-10.The authors conjecture that augmented examples may not align with GEM’s buffer-based inequality constraints.
  • Applicability to other methods: BiC and Exponential learning rate Decay can be extended to oEWC and Synaptic Intelligence, but BiC is effective only when previous tasks are equally biased.This condition holds for rehearsal methods that replay equal amounts from previous tasks, unlike oEWC and SI.

F. Execution Times

Execution-time comparisons show that plain ER is substantially faster than more sophisticated rehearsal methods, while several alternatives incur overhead at task boundaries. The experiments use Split Fashion-MNIST with buffer size 200 under identical GPU conditions.

  • Experimental setup: The runtime experiment evaluates methods on Split Fashion-MNIST with a memory buffer of 200 using identical conditions and an Nvidia GeForce RTX 2080 GPU.
  • Runtime comparison: Plain ER is remarkably faster than all other evaluated methods.GEM is the slowest because its quadratic programming constraint is solved for each task, whereas A-GEM applies one constraint throughout.
  • Runtime comparison: A-GEM is clearly faster than GEM because it applies one similar constraint at all times instead of a quadratic programming constraint for each task.
  • Runtime comparison: Task-boundary computations contribute to the execution times of iCaRL, HAL, and ER+T.These include herding over previous-task examples, computing class anchor points, and training the Bias Control module.
  • Overall trade-off: The paper reports that enhanced ER outperforms more sophisticated approaches despite a limited growth in computational requirements.
Loading 2010.05595v1…