Source-linked AI summary
On Tiny Episodic Memories in Continual Learning
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, Marc'Aurelio Ranzato
TL;DR
Continual learning must retain performance on past tasks while learning from a task stream with limited supervision. This paper evaluates Experience Replay with tiny episodic memories under single-pass supervised learning and finds strong gains across four benchmarks. It also reports that repeated memory training can improve generalization rather than harm it.
Problem
Catastrophic forgetting makes it difficult for continual-learning models to recall previously learned tasks, motivating evaluation of small episodic memories under realistic single-pass conditions.
Method
The paper empirically studies Experience Replay, which jointly trains on current-task examples and examples stored in a tiny episodic memory across four supervised continual-learning benchmarks.
Results
Experience Replay outperforms continual-learning approaches with and without episodic memory, while tiny-memory replay does not harm generalization and can improve it.
Takeaways & Limitations
ER with tiny episodic memories provides a strong, simple baseline at very low additional computational cost, and subsequent-task training acts as a data-dependent regularizer.
Takeaways & Limitations
The study focuses on supervised continual-learning tasks with fixed architectures, single-pass examples, and small episodic memories.
Abstract
from arXiv · showhide
In continual learning (CL), an agent learns from a stream of tasks leveraging prior experience to transfer knowledge to future tasks. It is an ideal framework to decrease the amount of supervision in the existing learning algorithms. But for a successful knowledge transfer, the learner needs to remember how to perform previous tasks. One way to endow the learner the ability to perform tasks seen in the past is to store a small memory, dubbed episodic memory, that stores few examples from previous tasks and then to replay these examples when training for future tasks. In this work, we empirically analyze the effectiveness of a very small episodic memory in a CL setup where each training example is only seen once. Surprisingly, across four rather different supervised learning benchmarks adapted to CL, a very simple baseline, that jointly trains on both examples from the current task as well as examples stored in the episodic memory, significantly outperforms specifically designed CL approaches with and without episodic memory. Interestingly, we find that repetitive training on even tiny memories of past tasks does not harm generalization, on the contrary, it improves it, with gains between 7\% and 17\% when the memory is populated with a single example per class.
1 Introduction
Continual learning seeks rapid adaptation across task streams, but catastrophic forgetting undermines recall of previously learned tasks. The paper evaluates tiny episodic memories and finds that simple Experience Replay is highly effective and efficient.
- Catastrophic forgetting makes it difficult for continual-learning models to recall how to perform previously seen tasks.
- The study evaluates continual-learning methods with supervised tasks, single-pass examples, fixed architectures, and small memories of past-task examples.Restricting memory distinguishes continual learning from multi-task learning, where complete task datasets remain available.
- Experience Replay jointly trains on current-task examples and examples from a tiny episodic memory, outperforming specifically designed continual-learning approaches with and without memory.The evaluation covers Permuted MNIST, Split CIFAR, Split miniImageNet, and Split CUB.
- ER with tiny episodic memories delivers very strong performance at a very small additional computational cost over fine-tuning.
2 Related Work
Related continual-learning methods address forgetting through regularization or episodic memory. This work distinguishes its supervised, single-pass, tiny-memory Experience Replay setting from prior replay studies.
- Regularization-based methods reduce forgetting by penalizing feature drift or discouraging changes to parameters important for past tasks.These methods may store activations for distillation or parameter-importance measures whose best-case memory complexity matches the network size.
- Memory-based methods store subsets of past-task data and can constrain optimization so losses on past tasks do not increase.
- This study examines supervised continual learning with a single pass through data and a much smaller episodic memory than earlier Experience Replay studies.It also analyzes why repeatedly training on tiny memories does not lead to overfitting.
3 Learning Framework
The learning framework uses separate cross-validation and evaluation task streams, with evaluation examples observed once and performance measured throughout the sequence using average accuracy and forgetting.
- The protocol separates a replayable cross-validation stream from an evaluation stream containing T tasks.The cross-validation stream contains T_CV tasks, while the evaluation stream is D_1 through D_T.
- Cross-validation tasks may be replayed as needed, whereas evaluation tasks are played only once.
- The learner observes evaluation examples sequentially, is tested during learning, and receives final evaluation on held-out test data.
- Each task dataset consists of examples containing an input, task descriptor, and task-specific target vector.
- Performance on the evaluation stream is measured with average accuracy and average forgetting.Average accuracy uses held-out performance on task j after training through task i; average forgetting is defined separately.
4 Experience Replay
Experience Replay directly combines current-task and episodic-memory examples during optimization, while using compact memory-writing strategies to preserve useful past-task representatives.
- Optimization: Under single-pass learning, ER directly trains on episodic-memory examples instead of using memory only for gradient projection.A-GEM projects the current gradient when it conflicts with a memory gradient, whereas ER averages gradients from current and memory minibatches.
- Optimization: ER doubles the minibatch by stacking current-task examples with a memory minibatch before a single gradient update.
- Training procedure: The ER procedure samples a current-task minibatch and a memory minibatch, updates parameters jointly, then updates the episodic memory.
- Memory writing: Reservoir sampling selects a random subset from an unknown-length data stream, while ring buffers guarantee equal class representation.Ring buffers retain recent observations per class and can leave memory underused early in training.
- Memory writing: k-Means and Mean of Features select examples near class centroids or average feature vectors to populate memory.The k-Means strategy uses representations before the final classification layer and seeks feature-space coverage.
5 Experiments
The experiments evaluate continual-learning methods on four supervised benchmarks under single-pass training, comparing episodic-memory strategies and analyzing experience replay’s accuracy, efficiency, and generalization.
- Experimental setup: The evaluation covers Permuted MNIST, Split CIFAR, Split miniImageNet, and Split CUB using common architectures, single-pass training, and held-out DEV metrics.The benchmarks use task-specific classifier heads and stochastic-gradient optimization; DCV is used for cross-validation.
- Accuracy results: Experience Replay substantially outperforms FINETUNE, EWC, A-GEM, and MER, including with very small episodic memories.On CIFAR, ER gains 1.7% over A-GEM with one example per class and more than 5% with 13; gains over FINETUNE and EWC are 15% and about 28%, respectively.
- Accuracy results: Average accuracy increases with memory size and does not saturate at the tested limit of 13 examples per class.This result is reported for the first experiment’s average-accuracy evaluation on DEV.
- Memory strategies: Reservoir sampling performs best across larger memories, whereas balanced strategies such as ring buffer, k-Means, and Mean of Features work best below three examples per class.With one example per class on CIFAR, reservoir ER is 3.5% worse than ER K-Means, while the three balanced methods are within 0.5% of one another.
- Memory strategies: A hybrid reservoir-to-ring-buffer strategy improves over reservoir sampling as memory per task shrinks and remains at least similar to ring buffer.The strategy switches when some classes have too few stored samples; the example uses 85 slots, averaging one sample per class by the end.
- Efficiency and generalization: Experience Replay combines strong accuracy and lower forgetting with only slight computational overhead over fine-tuning and lower training cost than A-GEM and MER.The compute comparison uses ring-buffer ER on Split CIFAR and Split CUB because the ER variants have the same computational complexity.
- Efficiency and generalization: Training on current-task data together with memory examples improves Task 1 generalization across task relatedness, although nearly adversarial tasks can reduce accuracy from 40% to 30%.The analysis attributes the regularization strength to the amount of current-task data and its effectiveness to task relatedness.
Conclusions
The study finds that experience replay with tiny episodic memories performs strongly across supervised continual-learning streams at marginal computational cost. It also identifies memorization of tiny memories as useful for generalization, while leaving memory-selection strategies for future work.
- ER methods with tiny episodic memories provide a very large performance boost over fine-tuning at a very marginal computational cost.
- The proposed hybrid memory strategy balances randomized memory contents with sufficient representatives for each class.
- Memorizing tiny memories can improve generalization because subsequent-task training acts as a data-dependent regularizer.
- Future work includes identifying inputs that best mitigate expected forgetting and strategies for removing samples when memory is full.
Appendix
The appendix collects algorithms, detailed experimental results, further analysis, and baseline hyper-parameters supporting the main paper.
- The appendix provides memory-update algorithms described in the main paper.
- It reports detailed results underlying Figure 1 and Table 1.
- It further describes the analysis from Section 5.5 and lists baseline hyper-parameters across datasets.
A Memory Update Algorithms
The appendix specifies several strategies for updating the episodic memory, including reservoir sampling, ring buffers, K-means, and feature-mean selection.
- Reservoir sampling: Reservoir sampling fills available memory and probabilistically overwrites existing slots as more stream examples arrive.The algorithm uses the number of observed examples and a random index to decide replacements.
- Ring buffer: Ring buffers store examples by appending them to fixed-size first-in, first-out stacks.
- K-Means: K-means memory updates assign each example to its nearest centroid and retain examples closest to those centroids.
- Mean of Features: Mean-of-features selection stores examples closest to a running average feature vector.
- Mean of Features: A class-aware update maintains examples near class feature centers by replacing stored examples that are farther from the current center.
B Detailed Results
The appendix reports detailed accuracy and forgetting results for four continual-learning benchmarks and shows how average-accuracy values support the main paper’s figures.
- Evaluation context: The appendix notes that the MULTI-TASK baseline retains all task datasets and therefore does not follow the continual-learning definition.
- Permuted MNIST: Detailed results report average accuracy and forgetting for Permuted MNIST under different samples-per-class settings.
- Split CIFAR: Detailed results report average accuracy and forgetting for Split CIFAR under different samples-per-class settings.
- miniImageNet: Detailed results report average accuracy and forgetting for miniImageNet under different samples-per-class settings.
- CUB: Detailed results report average accuracy and forgetting for CUB under different samples-per-class settings.
- Figures: Figures 4–7 show average-accuracy evolution as new tasks are learned for MNIST, CIFAR, miniImageNet, and CUB.The figures compare low-memory settings with larger samples-per-class settings.
C Further Analysis
Table 7 compares ER-RINGBUFFER and A-GEM on train, memory, and test performance after similar and dissimilar task rotations. ER-RINGBUFFER reaches perfect memory performance, while A-GEM requires more training iterations to match it.
- Table 7 reports MNIST Rotation performance on task 1 after training on task 2.
- The comparison covers train, memory, and test performance for similar 10°-rotation and dissimilar 90°-rotation tasks.
- ER-RINGBUFFER always achieves perfect performance on the memory across the evaluated task configurations.
- A-GEM must train for more iterations to achieve the same memory-performance effect as ER-RINGBUFFER.
D Hyper-parameter Selection
Table 8 lists the selected hyper-parameters for the four benchmark datasets, including learning rate and method-specific training coefficients.
- Table 8 provides hyper-parameter selections for the four benchmark datasets.
- The listed parameters include learning rate, EWC synaptic strength, MER within-batch meta-learning rate, and MER current-example learning-rate multiplier.