Source-linked AI summary
SS-IL: Separated Softmax for Incremental Learning
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, Taesup Moon
TL;DR
Class-incremental learning suffers catastrophic forgetting when imbalanced new-class and exemplar data create classification-score bias. The paper analyzes ordinary all-class softmax as a source of this bias and proposes SS-IL, combining separated softmax with task-wise distillation. Experiments report strong accuracy and more balanced scores without additional post-processing.
Problem
Exemplar-memory CIL lacks a systematic analysis of classification-score bias arising from imbalanced new-class and old-class data, although the bias contributes to forgetting.
Method
SS-IL combines a separated-softmax output layer with task-wise knowledge distillation to mitigate biased penalization and preserve task-wise knowledge.
Results
SS-IL achieves strong state-of-the-art accuracy with more balanced prediction scores across old and new classes, without additional post-processing.
Takeaways & Limitations
Separating softmax computation and distillation by task addresses score-bias concerns in exemplar-memory CIL within the evaluated large-scale benchmark scenarios.
Takeaways & Limitations
The exemplar memory is much smaller than current-task data, and examples stored per old class decrease linearly as incremental tasks grow.
Abstract
from arXiv · showhide
We consider class incremental learning (CIL) problem, in which a learning agent continuously learns new classes from incrementally arriving training data batches and aims to predict well on all the classes learned so far. The main challenge of the problem is the catastrophic forgetting, and for the exemplar-memory based CIL methods, it is generally known that the forgetting is commonly caused by the classification score bias that is injected due to the data imbalance between the new classes and the old classes (in the exemplar-memory). While several methods have been proposed to correct such score bias by some additional post-processing, e.g., score re-scaling or balanced fine-tuning, no systematic analysis on the root cause of such bias has been done. To that end, we analyze that computing the softmax probabilities by combining the output scores for all old and new classes could be the main cause of the bias. Then, we propose a new method, dubbed as Separated Softmax for Incremental Learning (SS-IL), that consists of separated softmax (SS) output layer combined with task-wise knowledge distillation (TKD) to resolve such bias. Throughout our extensive experimental results on several large-scale CIL benchmark datasets, we show our SS-IL achieves strong state-of-the-art accuracy through attaining much more balanced prediction scores across old and new classes, without any additional post-processing.
1. Introduction
The paper identifies classification-score bias from imbalanced exemplar-memory training as a source of catastrophic forgetting in class-incremental learning. It proposes SS-IL, combining separated softmax with task-wise knowledge distillation, and reports balanced scores and strong accuracy without post-processing.
- Class incremental learning trains on newly arriving classes while testing across all classes learned so far, making catastrophic forgetting a central challenge.
- Exemplar-memory methods face severe imbalance between abundant new-class data and limited old-class exemplars, skewing predictions toward new classes.
- Prior bias-correction methods improve accuracy but lack systematic analysis of the bias source, and some use knowledge distillation without adequate justification.
- The paper attributes the bias to ordinary softmax combining all class scores, which heavily penalizes old-class outputs during imbalanced training; GKD may preserve this bias.
- SS-IL uses a separated-softmax layer and task-wise knowledge distillation to balance old- and new-class scores, achieving strong state-of-the-art accuracy without additional post-processing.
2. Related Work
Related work includes regularization-, architecture-, and exemplar/replay-memory approaches to continual learning. In CIL, exemplar memory, bias correction, feature distillation, and knowledge distillation form key comparison points, with setting differences affecting fairness.
- Continual-learning methods are commonly grouped into regularization-based, dynamic-architecture-based, and exemplar/replay-memory-based approaches.
- Exemplar-memory CIL methods such as iCaRL and EEIL mitigate forgetting using stored old-class examples, with EEIL adding balanced fine-tuning.
- LUCIR and PODNet use feature distillation in a setting with many base classes, which the paper treats as more limited than pure CIL.
3. Preliminaries
The preliminaries formalize CIL as sequential learning of disjoint new-class tasks with a fixed exemplar memory and consolidated all-class prediction. They define global and task-wise knowledge-distillation formulations for retaining prior knowledge.
- 3.1. Notations and problem setting: Each incremental task contains data for m previously unseen classes, and the total class count after task t is C_t = m · t.
- 3.1. Notations and problem setting: The exemplar memory stores floor(|M|/C_{t−1}) examples per previously learned class, so examples per class decrease linearly as tasks grow and |M| ≪ n_t.
- 3.1. Notations and problem setting: At task t, the CNN classifier parameters are learned from current-task data D_t combined with exemplar memory M.
- 3.1. Notations and problem setting: After learning, the consolidated final layer predicts among all classes learned through task t as an ordinary multiclass classifier.
- 3.2. Knowledge distillation: General KD uses a global softmax across past tasks, whereas task-wise KD computes separate softmax probabilities for each task before regularizing the previous model.
4. Motivation
The analysis attributes CIL prediction bias to ordinary softmax coupling across old and new classes under imbalanced training, with GKD potentially preserving that bias.
- 4.1. Bias caused by ordinary cross-entropy: A task-level confusion matrix shows past-task predictions becoming overly biased toward the most recent task.The cited experiment uses ImageNet with m = 100, |M| = 10k, and 10 total tasks.
- 4.1. Bias caused by ordinary cross-entropy: Ordinary cross-entropy combines old and new class scores, causing old-class scores to be continually penalized during abundant new-class updates.This mechanism links imbalanced gradient descent steps to score bias toward new classes.
- 4.1. Bias caused by ordinary cross-entropy: The toy illustration depicts scores for old classes continuing to decrease under imbalanced gradient descent steps.It considers m = 2 and T = 2 on imbalanced D2 ∪M.
- 4.2. Bias preserved by GKD: GKD can preserve bias because its soft targets, produced by the old model, are heavily skewed toward the most recent old task.This skew can further penalize probabilities for older tasks and intensify their forgetting.
- 4. Motivation: These observations motivate computing softmax probabilities separately rather than combining old and new tasks altogether.The proposed method is introduced as a response to this identified source of prediction bias.
5. Main Method
SS-IL combines a separated-softmax output layer with task-wise knowledge distillation to reduce imbalanced penalization and avoid preserving task-wise bias.
- 5. Main Method: SS-IL consists of a Separated-Softmax output layer and Task-wise KD.The two components are motivated by the preceding analysis of score bias and biased distillation targets.
- (1) Separated-Softmax (SS) layer: Separated softmax computes probabilities and cross-entropy separately for old-memory and new-task samples.Old and new samples use output scores from their respective class groups.
- (1) Separated-Softmax (SS) layer: For new-class samples, the separated-softmax gradient with respect to old-class scores is zero, preventing their excessive penalization.This directly blocks the new-class gradient from overly reducing old-class scores.
- (2) Task-wise KD: Task-wise KD computes soft targets within each task, avoiding the task-wise bias that may remain in the old model.The paper argues that this makes TKD particularly well-suited to the separated-softmax layer.
- 5. Main Method: The paper reports that SS balances scores between old and new classes and achieves state-of-the-art accuracy across large-scale benchmarks and incremental scenarios.The method is trained with mini-batch SGD using exemplar replay.
6. Experiments
The experiments evaluate SS-IL across large-scale datasets, varied incremental scenarios, and exemplar-memory sizes. SS-IL generally outperforms baselines, balances old and new class predictions, and benefits from task-wise knowledge distillation.
- Datasets and evaluation protocol: Table 1 reports Average Top-1 and Top-5 accuracy averaged over all incremental tasks, including the initial task.Baselines include iCaRL, Fine-Tuning, IL2M, EEIL, BiC, LUCIR, and PODNet; PODNet was not run on Landmark-v2 datasets.
- Datasets and evaluation protocol: The evaluation covers ImageNet-1K, Landmark-v2-1K, and Landmark-v2-10K across varying task counts and exemplar-memory sizes.The scenarios use T = {5, 10, 20}; memory sizes are {5k, 10k, 20k} for 1K datasets and {20k, 40k, 60k} for Landmark-v2-10K.
- Results: SS-IL dominates other baselines throughout almost every evaluation scenario, whereas baseline accuracy drops sharply as exemplar-memory size decreases.The results indicate that SS-IL is robust across incremental conditions and memory constraints.
- Results: In Figure 5, SS-IL mostly dominates baselines, and its performance gap widens as the incremental task increases.For ImageNet-1K, SS-IL is less affected by catastrophic forgetting than baselines relative to the jointly trained upper bound.
- Ablation study: SS produces overlapping old- and new-class scores, while SS-IL and SS yield more balanced predictions than models without SS.The ablation compares LCE, LCE + LTKD, LCE-SS, and LCE-SS + LTKD on ImageNet-1K with T = 10 and |M| = 10k.
- Analyses on KD: LCE-SS + LTKD achieves the highest Top-5 accuracy, and LTKD outperforms GKD in the analyzed comparisons.On the same biased teacher, θTKD,t has higher Top-5 accuracy than θGKD,t at every evaluated task.
7. Concluding Remarks
SS-IL addresses classification score bias in exemplar-memory CIL by analyzing ordinary softmax and balancing scores between old and new classes. Extensive experiments show outstanding performance in most scenarios.
- Ordinary softmax over all classes heavily penalizes old-class output probabilities, contributing to classification score bias.
- SS-IL addresses this bias in exemplar-memory based CIL.
- SS effectively balances prediction scores between old and new classes.
- SS-IL shows outstanding performance in most evaluated scenarios.