Source-linked AI summary
Rainbow Memory: Continual Learning with a Memory of Diverse Samples
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, Jonghyun Choi
TL;DR
Blurry-CIL is a realistic continual-learning setting in which tasks share classes while memory access to previous data is limited. Rainbow Memory uses uncertainty-based exemplar selection and data augmentation to increase memory diversity, and it achieves strong performance across blurry-CIL benchmarks. The paper also reports comparable performance in disjoint and offline CIL setups.
Problem
Existing continual-learning evaluations often use artificial disjoint tasks, while blurry-CIL involves shared classes and requires retaining old-class information with few stored samples.
Method
Rainbow Memory selects diverse exemplars using per-sample perturbation-based uncertainty and further increases their diversity with data augmentation.
Results
RM consistently outperforms other methods by large margins in blurry-CIL across MNIST, CIFAR10, CIFAR100, and ImageNet evaluations.
Takeaways & Limitations
Diversity-enhancing memory management is effective for blurry-CIL and remains comparable to other methods in disjoint and offline CIL setups.
Takeaways & Limitations
The method estimates feature-space relative location through classification uncertainty rather than exact sample-to-sample distances, which would require O(N^2) computation.
Abstract
from arXiv · showhide
Continual learning is a realistic learning scenario for AI models. Prevalent scenario of continual learning, however, assumes disjoint sets of classes as tasks and is less realistic rather artificial. Instead, we focus on 'blurry' task boundary; where tasks shares classes and is more realistic and practical. To address such task, we argue the importance of diversity of samples in an episodic memory. To enhance the sample diversity in the memory, we propose a novel memory management strategy based on per-sample classification uncertainty and data augmentation, named Rainbow Memory (RM). With extensive empirical validations on MNIST, CIFAR10, CIFAR100, and ImageNet datasets, we show that the proposed method significantly improves the accuracy in blurry continual learning setups, outperforming state of the arts by large margins despite its simplicity. Code and data splits will be available in https://github.com/clovaai/rainbow-memory.
1. Introduction
The paper targets blurry-CIL, a more realistic setting where tasks share classes and only limited previous-task data is accessible. Rainbow Memory improves memory diversity through uncertainty-aware sampling and data augmentation, yielding strong blurry-CIL performance.
- Motivation: Blurry-CIL shares classes across sequential tasks, unlike the more artificial disjoint-CIL setup.Tasks arrive as a stream, their majority classes differ, and only a small portion of previous-task data is available.
- Motivation: Fixed-size episodic memories must preserve sufficient old-class information despite storing far fewer samples than incoming tasks.Simply increasing memory size would violate the limited fixed-memory constraint.
- Approach: Rainbow Memory selects exemplars using perturbation-induced, per-sample classification uncertainty to capture both robust and fragile samples.The method samples according to a distribution corresponding to estimated robustness, helping preserve class-discriminative boundaries.
- Approach: Rainbow Memory further enhances exemplar diversity with label-mixing and composed data augmentations applied to stored samples.The paper names the resulting method Rainbow Memory, or RM.
- Results: RM outperforms state-of-the-art baselines by significant margins in blurry-CIL while remaining comparable to other methods in disjoint-CIL.Evaluations cover MNIST, CIFAR10, CIFAR100, and ImageNet.
2. Related Work
Related work covers continual-learning setups, imbalance mitigation, and episodic-memory management. Unlike approaches focused mainly on efficiency or weight updates, this paper emphasizes the quality and diversity of stored samples.
- Class Incremental Learning Setups: CIL research includes rehearsal methods that replay stored exemplars and regularization methods that retain prior-task information without storing old samples.The paper focuses on class-incremental learning without task identity at inference.
- Class Incremental Learning Setups: Recent work distinguishes realistic blurry task streams from conventional disjoint tasks and online learning from offline learning with temporary current-task buffers.The paper focuses on the more difficult and practical online setup.
- Class Incremental Learning Setups: The paper identifies memory management as a contribution for class-incremental learning.
- Class Imbalance: Rehearsal methods can suffer catastrophic forgetting when class-imbalanced exemplars make models vulnerable to frequent classes.Prior methods address this with gradient-based updates or bias calibration.
- Episodic Memory Management: Episodic-memory selection methods include uniform sampling, herding, discriminative sampling, and entropy-based sampling, with some offering marginal gains over random selection.Other work also targets representative coresets and perturbed or generated memories.
- Episodic Memory Management: Generative, compressed, feature-reduced, and batch-distillation methods improve memory utilization or efficiency, whereas this paper focuses on stored-sample quality.
3. Class Incremental Learning Setups
CIL assigns class subsets to sequential tasks, with blurry-CIL allowing classes to recur across tasks while disjoint-CIL keeps task classes separate. These setups formalize changing class distributions and differing exposure to previously seen classes.
- Task formulation: CIL represents each task with a class subset T_t assigned by a stochastic function ψ over the set of all classes.The formulation also defines sample spaces for classes and tasks, while the total number of classes is not assumed known or bounded.
- Blurry versus disjoint CIL: Blurry-CIL intersects task sample spaces so that classes can appear in multiple tasks, whereas disjoint-CIL does not.The distinction is expressed through whether task sample spaces overlap across tasks.
- Blurry versus disjoint CIL: Disjoint-CIL exaggerates catastrophic forgetting by never exposing seen classes in successive tasks, unlike the more realistic recurring-class structure of blurry-CIL.Blurry-CIL gives each task a small number of classes also present in other tasks, producing faint task boundaries.
4. Approach
Rainbow Memory manages an episodic memory for blurry continual learning by selecting diverse, class-balanced exemplars using perturbation-based uncertainty and augmenting samples to further diversify training data.
- Rainbow Memory selects exemplars that are both representative of their class and discriminative to other classes by targeting diversity in feature space.The strategy associates central samples with representativeness and boundary-near samples with discriminativeness.
- Perturbation-based uncertainty estimates a sample’s relative location without computing O(N^2) sample-to-sample distances.The method assumes more certain samples lie nearer the class center and uses model-output variability under augmentation instead.
- Uncertainty is computed from the variance of model outputs across color jitter, shear, and cutout perturbations, with lower values indicating stronger model confidence.The estimate uses a Monte Carlo approximation, and consistency of the top-1 prediction across perturbations yields lower uncertainty.
- The memory update assigns equal slots to seen classes, sorts each class’s samples by uncertainty, and selects interval-spaced exemplars.This fills memory with samples spanning robust to fragile perturbation responses, imposing perturbation-based diversity.
- Data augmentation further diversifies exemplars through single-image, mixed-label, and automated augmentations, including mixing new-task images with old-class exemplars.Mixed-label augmentation addresses distribution changes across tasks, while AutoAugment composes augmentations with controlled numbers and magnitudes.
5. Experiments
The experiments evaluate Rainbow Memory across blurry and disjoint class-incremental settings, datasets, memory sizes, and training protocols. RM consistently excels in online blurry-CIL, while its advantage is smaller offline and comparable in disjoint-CIL.
- Experimental setup: RM is evaluated on MNIST, CIFAR10, CIFAR100, and ImageNet across blurry-CIL benchmarks, memory sizes, metrics, and online or offline protocols.The evaluation compares RM with standard CIL baselines and analyzes its memory-management and augmentation components.
- Blurry online results: RM consistently outperforms all other methods in CIFAR10-Blurry10-Online, with larger gains as the number of classes increases.On MNIST, RM without data augmentation performs best; on other datasets, augmentation improves the metrics substantially.
- Memory-size analysis: RM outperforms baselines on A5, F5, and I5 across episodic memory sizes in CIFAR10-Blurry10-Online.Its performance gaps shrink with larger memories, and similar trends hold on CIFAR100 and ImageNet.
- Blurry levels and protocols: RM performs comparably in disjoint-CIL without regularization while outperforming rehearsal-based methods such as iCaRL, GDumb, and BiC.Regularization methods including EWC and Rwalk remain competitive in disjoint-CIL.
- Blurry levels and protocols: RM’s gain diminishes offline, while in online Blurry10 and Blurry30 it surpasses baselines by remarkable margins and exceeds 70% accuracy with data augmentation.The authors conjecture that large incoming buffers dilute exemplar-sampling sensitivity offline.
6. Conclusion
The paper targets realistic blurry-CIL, where tasks share classes, by increasing episodic-memory diversity through uncertainty-based sampling and data augmentation. RM outperforms state-of-the-art methods in blurry-CIL while remaining comparable in disjoint and offline settings.
- Conclusion: Blurry-CIL models realistic class-incremental learning by allowing tasks to share classes.The paper identifies this setting as more realistic and practical than disjoint tasks.
- Conclusion: RM enhances episodic-memory diversity using per-sample perturbation-based uncertainty and diverse data augmentation.The resulting exemplars aim to improve representativeness and discriminativeness.
- Conclusion: On CIFAR10, CIFAR100, and ImageNet blurry-CIL, RM outperforms state-of-the-art methods by large margins and remains comparable in disjoint and offline CIL.The paper also studies different blurry levels and individual method components.
A. Accuracy Over the Tasks in Various CIL Setups
Accuracy trajectories differ between disjoint and blurry CIL because their evaluation sets and class exposure evolve differently. RM is strongest in online settings, whereas its advantage reduces or becomes negative offline.
- Evaluation dynamics: Disjoint-CIL accuracy tends to decrease as task iterations add more seen classes to the evaluation set.Each disjoint task introduces exclusive classes, increasing the evaluation-set size over time.
- Evaluation dynamics: Blurry-CIL accuracy can increase in later tasks because the evaluation covers all classes and the model sees more data for each class.As the blurry ratio rises from Blurry10 to Blurry30, accuracy flattens across tasks.
- Protocol effects: RM and RM without data augmentation outperform other methods online, but their margin reduces or becomes negative offline.Blurry-online exposes current-task samples once, making memory diversity more important than in offline training.
B. Metrics Details
The paper evaluates continual-learning performance with last accuracy, last forgetting, and intransigence. These metrics respectively capture final accuracy, retained performance relative to prior best accuracy, and distance from a non-CIL upper bound.
- Metrics: Last accuracy reports model accuracy after training ends over all classes exposed during training.It is denoted A in the metric description.
- Metrics: Last forgetting averages how much each task’s accuracy degraded from its best value in earlier training phases.It is denoted F and is measured after training ends.
- Metrics: Intransigence measures each task’s achieved accuracy relative to a non-CIL upper bound and averages the values across tasks.The metric is denoted I.
C. Class Distribution over Tasks
CIFAR10 and CIFAR100 classes were randomly split into five tasks, while ImageNet classes were split into ten tasks to create the CIL benchmark.
- CIFAR10 and CIFAR100 classes were randomly split into five tasks, containing 2 and 20 classes per task, respectively.
- ImageNet classes were split into ten tasks to generate the CIL benchmark.
- Experiments were repeated three times with different class splits from three random seeds, except for ImageNet.
- The CIFAR10 CIL-benchmark class splits used in the experiments are summarized in Table 7.
D. Data Augmentation (K = 1,000)
The appendix reports accuracy comparisons for data-augmentation methods in CIFAR10-Blurry10-Online at K = 1,000 and illustrates accuracy changes across online and offline task settings.
- Data Augmentation (K = 1,000): 78.0%: RM achieves the best last accuracy among methods with data augmentations in CIFAR10-Blurry10-Online at K = 1,000.The reported tendency matches Table 4 for K = 200 and K = 500.
- Data Augmentation (K = 1,000): Table 8 compares last accuracy A5 (↑, %) across methods with data augmentations in CIFAR10-Blurry10-Online at K = 1,000.
- Online Settings: Figure 5 illustrates accuracy changes while learning tasks in CIFAR10-Disjoint-Online, CIFAR10-Blurry10-Online, and CIFAR10-Blurry30-Online settings.
- Offline Settings: Figure 6 illustrates accuracy changes while learning tasks in CIFAR10-Disjoint-Offline, CIFAR10-Blurry10-Offline, and CIFAR10-Blurry30-Offline settings.