Source-linked AI summary
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
Zheda Mai, Ruiwen Li, Hyunwoo Kim, Scott Sanner
TL;DR
Online class-incremental learning must learn new classes from a non-stationary stream while limiting catastrophic forgetting, but replay with Softmax remains vulnerable to recency bias. The paper replaces Softmax with NCM and introduces SCR, which improves class structure in embedding space through supervised contrastive replay. SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art methods across datasets and memory settings.
Problem
Replay-based online class-incremental learning remains challenged by Softmax-induced task-recency bias under old–new class imbalance.
Method
SCR combines supervised contrastive loss with replay and an NCM classifier to cluster same-class embeddings and separate different-class embeddings.
Results
SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art continual-learning methods across various datasets and memory settings.
Takeaways & Limitations
NCM is a simple, effective alternative to Softmax for online continual learning, while SCR provides a stronger replay-based approach within the evaluated settings.
Takeaways & Limitations
The reported scope is bounded by the evaluated datasets and memory settings; combining SCR with other memory-management methods may yield better performance.
Abstract
from arXiv · showhide
Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods. To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.
1. Introduction
Online class-incremental learning requires models to learn new classes from single-pass streams without task IDs while retaining prior knowledge. The paper identifies Softmax recency bias as a replay challenge, advocates NCM, and introduces SCR to improve embedding geometry during replay.
- Online class-incremental learning requires learning new classes from a single-pass stream and classifying all labels without task IDs.
- Replay methods are efficient, but old–new class imbalance makes Softmax classifiers favor classes from the most recent task.
- The NCM classifier addresses recency bias, avoids fully connected-layer changes for new classes, and improves five memory-buffer methods.
- SCR uses supervised contrastive loss to cluster same-class embeddings tightly and separate embeddings from different classes during replay.
- Across three benchmarks, SCR outperforms state-of-the-art methods by significant margins with three memory-buffer sizes.
2. Related Work
The paper formalizes online class-incremental learning around an encoder, classifier, bounded memory, and single-head evaluation. It situates replay among major continual-learning approaches and motivates supervised contrastive representations for distance-based classification.
- 2.1. Continual Learning: The data stream contains non-overlapping task classes, and each sample is seen only once in online class-incremental learning.
- 2.1. Continual Learning: The model uses an encoder that maps inputs to d-dimensional embeddings and a classifier that predicts among classes observed so far.
- 2.1. Continual Learning: A bounded memory stores selected prior data, while single-head evaluation requires choosing among all labels without task IDs.
- 2.2. Approaches: Continual-learning methods are categorized as regularization, parameter-isolation, or replay approaches.
- 2.3. Metrics: Average accuracy measures performance across observed tasks using held-out test-set accuracies after sequential training.
- 2.2. Contrastive Learning: Supervised contrastive learning treats same-class samples as similar and different-class samples as dissimilar in embedding space.
3. Method
The method replaces the Softmax classifier with NCM to reduce recency bias and avoid FC-layer changes, then uses supervised contrastive replay to improve embedding structure for NCM classification.
- Softmax Classifier vs. NCM Classifier: Softmax classification is vulnerable to new-old class imbalance, producing task-recency bias and requiring FC-layer changes when new classes arrive.The FC weights for new classes can exceed those for old classes, increasing probability mass for recent classes.
- Softmax Classifier vs. NCM Classifier: NCM classifies samples by comparing their embeddings with class-mean prototype vectors, without requiring an additional FC layer.Prototypes are computed from class embeddings, although updated encoders make exact old-class prototypes unavailable without previous training data.
- Softmax Classifier vs. NCM Classifier: NCM classifiers significantly improve average accuracy over Softmax classifiers across five replay methods and three datasets.The comparison uses memory sizes of 2,000 for Mini-ImageNet and CIFAR-100 and 500 for CIFAR-10.
- Supervised Contrastive Replay: The method argues that NCM performance depends on embeddings that cluster same-class samples and separate different-class samples.Binary cross-entropy in iCaRL and categorical cross-entropy may not create sufficiently discernible embedding patterns.
- Supervised Contrastive Replay: Supervised contrastive learning uses same-label samples as positives and a temperature parameter to shape representation separation.The loss operates on augmented batch representations and excludes each sample itself from its comparison set.
- Supervised Contrastive Replay: SCR combines current and memory-buffer batches, encodes them with shared encoder and projection networks, and applies supervised contrastive loss during replay.At testing, the projection network is discarded; buffered samples are encoded to compute NCM prototypes.
4. Experiment
Experiments evaluate SCR and NCM-augmented baselines across three datasets, memory sizes, runtime, and ablations. SCR consistently achieves higher accuracy, while NCM improves replay methods and SCR remains computationally efficient.
- NCM evaluation: NCM variants significantly improve AGEM, ER, GSS, MIR, and ASERµ across three datasets, with larger gains generally appearing at smaller memory sizes.For example, MIR on Mini-ImageNet improves relatively by 66.4% with M=1k versus 27.7% with M=5k.
- SCR evaluation: SCR consistently outperforms compared methods across CIFAR10, CIFAR100, and Mini-ImageNet, with all plotted methods using NCM.Figure 6 reports average accuracy on observed tasks with M=0.2k for CIFAR10 and M=2k for CIFAR100 and Mini-ImageNet.
- SCR evaluation: 35.4% (13.3%↑), 37.8% (8.2%↑), and 65.7% (15.4%↑) are SCR’s reported results for Mini-ImageNet, CIFAR100, and CIFAR10, respectively.These are reported as improvements over the second-best methods across the evaluated settings.
- Efficiency: SCR’s combined training and inference runtime is shorter than ASERµ and only slightly longer than MIR.The runtime comparison indicates that SCR’s accuracy does not require sacrificing computation efficiency.
- Ablation study: Memory batch sizes below 200 show more obvious accuracy improvement as batch size increases, but performance drops when the batch grows further.The authors suspect overfitting when batches of 500 or 1,000 comprise 25% or 50% of the memory buffer.
- Ablation study: Random retrieval with reservoir update performs better than GSS and slightly better than other tested memory-management options on CIFAR100.The authors attribute GSS’s weaker result to some classes having few or zero memory samples, and suggest other methods may help on imbalanced datasets.
- Ablation study: SCR remains stable for τ from 0.02 to 0.16, while performance deteriorates when temperature is too low or too high.The projection-network choice among MLP, linear, and none is reported as insignificant in online CL.
5. Conclusion
The paper identifies the NCM classifier as an effective replacement for Softmax in online continual learning and proposes SCR to improve its use through supervised contrastive embedding structure. SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art methods across datasets and memory settings.
- The NCM classifier is a simple yet effective substitute for Softmax in online continual learning, resolving several Softmax deficiencies.
- SCR encourages same-class embeddings to cluster tightly and different-class embeddings to separate during experience replay-based training.
- SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art continual-learning methods across various datasets and memory settings.
- Combining randomized experience replay, supervised contrastive loss, and an NCM classifier brings continual learning closer to offline-training performance.