PaperlayerView exact revision on arXiv

Source-linked AI summary

Essentials for Class Incremental Learning

Sudhanshu Mittal, Silvio Galesso, Thomas Brox

arXiv:2102.09517v1cs.CVcs.LG

TL;DR

Sequentially learning evolving tasks can cause neural networks to forget previous concepts, and class-IL lacks commonly established best practices for its core components. The paper combines simple components with a compositional loss and studies representation quality; it reports state-of-the-art improvements and links better class-IL performance to preserving secondary class information.

  • Problem

    Neural networks can overwrite previous concepts when trained on evolving tasks, while class-IL lacks a common understanding of best practices for its core components.

    Source: §1 · p. 1

  • Method

    The paper combines random exemplar selection and plain knowledge distillation with a compositional loss that balances intra-task and inter-task learning, and studies representation properties and regularization.

    Source: §1 · p. 1, §3.3 · p. 3, §4 · p. 4

  • Results

    The proposed base model exceeds state-of-the-art results, and the paper reports that feature specialization correlates with forgetting while preserving or improving secondary class information has a positive influence on class-IL.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors conclude that representation learning, particularly boosting secondary information, is a promising direction for improving incremental performance and feature transfer between tasks.

    Source: §7 · p. 8

    The evaluation’s strictly separated class batches are a comparison protocol that covers key class-IL issues, but practical data may arrive in a more mixed-up fashion.

    Source: §3.1 · p. 2

Abstract

from arXiv · show

Contemporary neural networks are limited in their ability to learn from evolving streams of training data. When trained sequentially on new or evolving tasks, their accuracy drops sharply, making them unsuitable for many real-world applications. In this work, we shed light on the causes of this well-known yet unsolved phenomenon - often referred to as catastrophic forgetting - in a class-incremental setup. We show that a combination of simple components and a loss that balances intra-task and inter-task learning can already resolve forgetting to the same extent as more complex measures proposed in literature. Moreover, we identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL. We show that performance is correlated with secondary class information (dark knowledge) learned by the model and it can be improved by an appropriate regularizer. With these lessons learned, class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.

1. Introduction

Class-incremental learning must build a unified classifier as new classes arrive, while retaining knowledge of previous classes despite limited stored data. The paper proposes a simple compositional approach and examines how learned representations relate to forgetting.

1. Introduction

  • The paper focuses on class-IL, where neural networks can overwrite previous concepts when trained sequentially on evolving tasks.The authors frame catastrophic forgetting as an obstacle to continual or lifelong learning.Source: §1 · p. 1
  • Class-IL learns a unified classifier over incrementally arriving classes, but cannot retain all past data for unified training and must avoid forgetting old classes while learning new ones.The paper describes memory, a forgetting constraint, and a system balancing old and new classes as crucial components.Source: §1 · p. 1
  • The authors position their approach as complementary to Prabhu et al., whose GDumb approach outperforms specialized formulations in most continual-learning settings but finds class-IL particularly challenging.Their approach masks softmax outputs appropriately and uses data balancing.Source: §1 · p. 1
  • The paper proposes a compositional class-IL model using plain knowledge distillation, random exemplar selection, and changes to output normalization; its base model exceeds state-of-the-art results.The approach is intended to isolate the underlying reasons for catastrophic forgetting while combining simple, effective components.Source: §1 · p. 1
  • The paper reports that feature specialization correlates with forgetting, and that only regularization techniques preserving or improving secondary class information have a positive influence on class-IL.Secondary class information is also called dark knowledge in [11].Source: §1 · p. 1

2. Related Work

Related work addresses exemplar replay, forgetting constraints, and bias from class imbalance through varied techniques. This paper uses random exemplars and plain logit-based knowledge distillation, and targets bias through its learning system.

Class-incremental learning

  • iCaRL introduced class-IL as a formal problem and combines decoupled feature-representation and classifier learning with distillation and replay.Later work by Castro et al. [3] extended it to an end-to-end model combining distillation and cross-entropy.Source: §2 · pp. 1–2

Exemplar selection

  • Replay-based methods retain old-class exemplars in a memory buffer; this paper selects samples randomly from each class rather than using herding or learned exemplars.Other cited approaches optimize exemplars jointly with the model or store feature descriptors instead of images.Source: §2 · p. 2

Forgetting-constraint

  • Knowledge distillation has been used as a forgetting constraint in class-IL, while this paper uses plain logit-based distillation to avoid forgetting.Other cited methods preserve feature representations or feature-space topology.Source: §2 · p. 2

Bias removal methods

  • Prior bias-removal methods target last-layer bias, while the paper proposes resolving the bias without post-processing by addressing underlying issues.The cited approaches include validation-set correction, prediction-statistic adjustment, weight-norm adjustment, and cosine normalization; the authors connect class imbalance to overfitting on new tasks and deteriorating old-class performance.Source: §2 · p. 2

3. Class-Incremental Learning

The paper defines class-IL as learning a unified classifier from sequential class data with only a limited memory of past exemplars. It evaluates incremental performance, forgetting, and feature retention, then presents a framework built from exemplars, distillation, and a compositional learning system.

3.1. Problem Definition

  • Class-IL requires classifying all classes seen so far despite access to complete data only for new classes and a small retained exemplar set for old classes.The paper notes that strictly separated data batches simplify comparison while covering key class-IL issues.Source: §3.1 · p. 2

3.2. Evaluation Metrics for Class-IL

  • The evaluation uses average incremental accuracy, forgetting rate, and feature retention to measure performance across steps and retained feature information.Feature retention compares a frozen feature extractor with a newly learned classifier against a jointly trained model with full data access.Source: §3.2 · p. 2

3.3. A Basic Class-IL Framework

  • Exemplars are selected in equal numbers per class and sorted by distance from each class’s feature mean under a fixed memory budget.When the memory is full, samples farther from the mean are removed first.Source: §3.3 · p. 3
  • The framework uses knowledge distillation with KL-divergence to penalize changes from the old model’s outputs on exemplar and new-class samples.The final loss combines classification cross-entropy and distillation loss.Source: §3.3 · p. 3
  • The proposed learning system isolates inter-task and intra-task learning by normalizing output logits, addressing class weight bias while sharing task-agnostic parts.The figure contrasts a standard loss using softmax over all logits with a compositional system using separate old- and new-class softmax spans.Source: §3.3 · p. 3, Figure 1 · p. 3

4. Compositional Learning System

The compositional system separates learning within new classes from learning relationships across old and new classes. It combines separate losses, balanced exemplars, and a reduced incremental learning rate.

Compositional Learning System

  • The system computes separate losses for new-class data and exemplars instead of merging their batches.The updated exemplar set also includes an equal-size set of exemplars from current new classes.Source: §4 · p. 3

Intra-task Learning

  • Separate losses let new-class classification use a dedicated softmax over new classes while distillation compares old-class outputs with the previous model.This allows new-class weights to be learned independently while sharing the feature extractor.Source: §4 · pp. 3–4

Inter-task Learning

  • A balanced exemplar set supports inter-task learning by using a combined softmax over old and new classes.The set contains equal numbers of samples from all classes, including old and new ones.Source: §4 · p. 4

Compositional Learning System

  • The exemplar set is compiled before learning the incremental task, unlike prior works that compile it after the step.The loss uses separate softmax for new-class learning and a unified softmax for the exemplar loss.Source: §4 · p. 4

Transfer Learning

  • Reducing the learning rate during incremental steps is proposed to improve transfer from old to new classes and mitigate distribution-shift effects.The reduced rate is determined experimentally based on the scale and relevance of features learned in the base task.Source: §4 · p. 4

5. Improving Feature Representations for Incremental Learning

The paper links class-incremental forgetting to the quality of base-task representations, including overfitting and loss of secondary class information. Its analyses find that regularizers improve class-IL performance when they preserve or enhance that secondary information.

5. Improving Feature Representations for Incremental Learning

  • The paper reports that overfitting and loss of secondary information correlate with forgetting, and that regularization helps only when it enhances secondary information.The regularization study tests self-distillation, data augmentation, label smoothing, and mixup.Source: §5 · p. 4, §5.3 · p. 6

5.1. Measuring the Quality of Secondary Logits

  • Secondary information is the model’s non-maximum class scores, which reflect beliefs about relationships between the target and other classes.The proposed CIFAR-100 proxy uses superclass classification from non-maximum logits; SS-NLL and SS-Acc measure this information, with lower SS-NLL and higher SS-Acc indicating better quality.Source: §5.1 · p. 5

5.2. Forgetting starts before the incremental step

  • As base-task validation loss rises after about 100 epochs, incremental accuracy falls while forgetting and feature-retention measures worsen.Secondary information also deteriorates: SS-Acc decreases and SS-NLL increases; the worsening feature-retention metric points to representations rather than last-layer bias.Source: Table 1 · p. 5, Figure 2 · p. 5, §5.2 · p. 6

5.3. Analyzing Catastrophic Forgetting with Regularization

  • Regularizers that improve secondary information—self-distillation and augmentation—improve class-IL performance, while label smoothing and mixup harm it.Augmentation performs best across the reported metrics; all tested regularizers improve calibration, but ECE does not indicate class-IL performance well.Source: §5.3 · pp. 6–7

6. Results

Across CIFAR-100, ImageNet-100, and ImageNet, regularized CCIL achieves state-of-the-art performance, while the simpler CCIL model compares favorably with previous results. Ablations show that combining separate softmax with a lower learning rate largely resolves bias and produces the best class-IL results.

6.1. Training Details

  • Experiments use one base task followed by 5 or 10 incremental tasks, with exemplar memory K = 2k for 100-class datasets and K = 20k for ImageNet.CIFAR-100 and ImageNet-100 use 50 base classes; full ImageNet uses 500.Source: §6.1 · p. 7

6.2. Ablation Studies

  • Separate softmax and a lower learning rate together largely resolve the old/new class weight bias and produce the best class-IL results.Separate softmax alone substantially reduces the bias, while a lower learning rate also removes it and improves performance to a lesser extent.Source: §6.2 · pp. 7–8
  • Replacing combined softmax with separate softmax and reducing the learning rate yields a major improvement over the iCaRL++ baseline.iCaRL++ adds a cosine-normalized layer and adaptive weighting to iCaRL.Source: §6.2 · p. 8

6.3. Comparison to SOTA

  • CCIL-SD achieves state-of-the-art performance across CIFAR-100, ImageNet-100, and ImageNet, while CCIL compares favorably with previous results, especially on ImageNet-1k.Table 4 compares average incremental accuracy across the three datasets.Source: §6.3 · p. 8, Table 4 · p. 8

7. Conclusions

The paper presents a straightforward class-incremental learning system that exceeds the state of the art without sophisticated modules. It also concludes that feature representation quality before incremental learning considerably determines forgetting.

7. Conclusions

  • The straightforward class-incremental system exceeds the state of the art without integrating sophisticated modules.The authors present it as a base model for future class-incremental learning research.Source: §7 · p. 8
  • Feature representation quality before incremental learning considerably determines the amount of forgetting, so countering forgetting only during incremental steps is not enough.The authors identify representation learning as a promising direction for improving incremental performance.Source: §7 · p. 8

A. Experiment Details

The appendix reports evaluation metrics averaged over three trials to capture variability in class-IL training. It also describes adaptive weighting between classification and distillation losses, which increases the weight on preserving old knowledge over incremental steps.

A. Experiment Details

  • Evaluation metrics are averaged over three run trials to capture variance in class-IL training.The appendix notes exceptions may apply when mentioned otherwise.Source: A. Experiment Details · p. 9

A.1. Adaptive Weighting (AW)

  • Adaptive weighting λ balances classification and distillation losses, dynamically increasing the weight on preserving old knowledge as incremental training continues.The reported improvement is 0.45% for the 5-task CIFAR-100 experiment; λbase is 5 for CIFAR-100, 20 for ImageNet-100, and 600 for ImageNet.Source: §A.1 · p. 9

A.2. Experiment Details

The experiments use shuffled class orders with a fixed Numpy seed of 1993. They train ResNet models using dataset-specific schedules and SGD settings.

A.2. Experiment Details

  • CIFAR-100 and ImageNet class orders are shuffled using Numpy seed 1993, and ImageNet-100 uses 100 randomly sampled ImageNet classes.ImageNet-100 contains around 128K training images and 5K evaluation images.Source: §A.2 · p. 9
  • CIFAR-100 and ImageNet networks use SGD; CIFAR-100 base and subsequent tasks train for 120 and 240 epochs, while ImageNet-100 trains for 70 epochs and ImageNet base and incremental tasks for 70 and 40.Source: §A.2 · p. 9

A.3. Overfitting Experiment

The overfitting experiment tracks class-incremental performance, secondary information, and calibration across increasingly overfitted snapshots. The regularization appendix examines effects on first-task metrics and describes the self-distillation protocol and class-mean distance comparisons.

A.4. Regularization

  • Regularization effects on CIFAR-100 class-IL performance and secondary information are evaluated on the first-task network, with the largest improvement observed from regularization on the initial task.Table 6 compares regularizers with CCIL using average incremental accuracy, forgetting rate, SS-NLL, and SS-Acc; the supplied passages do not state individual values.Source: §A.4 · pp. 9–10, Table 6 · p. 10
  • Self-distillation uses 4 generations for CIFAR-100 and ImageNet-100 and 2 for ImageNet, with each snapshot becoming the next generation’s teacher.Generation counts are selected using validation performance, and training combines classification and distillation losses.Source: §A.4 · p. 9

A.3. Overfitting Experiment

  • As snapshots become more overfitted, accuracy decreases and SS-NLL increases monotonically, while forgetting rate correlates with overfitting.The experiment also reports that ECE increases monotonically with training epochs; results are averaged over five runs with standard deviations.Source: §A.3 · p. 9, Table 5 · p. 10

A.4. Regularization

  • Figure 4 reports class-mean distance ratios relative to baseline CCIL, marking similar classes in bold and distinguishing similar from dissimilar pairs.The dotted circle denotes the baseline ratio of 1.0.Source: Figure 4 · p. 10

B. Representations: Qualitative Analysis

The analysis compares class-mean representation distances for semantically similar and dissimilar CIFAR-100 classes. Self-distillation and heavy augmentation preserve or widen their distance contrast, whereas mixup and label smoothing reverse it.

B. Representations: Qualitative Analysis

B.1. Class-mean Representations

  • The analysis treats classes sharing a CIFAR-100 coarse label as similar, compares them with classes from different superclasses, and measures L2 distance between class means.The comparison expresses each method’s distances relative to default baseline CCIL.Source: §B.1 · p. 11
  • Self-distillation and heavy augmentation reduce distances between similar classes while leaving dissimilar-pair distances unchanged or increasing them; mixup and label smoothing show the opposite pattern.The analysis reports cup-can distance ratios of 0.9 with self-distillation and 0.8 with augmentation, versus 1.2 with label smoothing and 1.1 with mixup.Source: §B.1 · p. 11
  • The distance patterns agree with the paper’s account that mixup and label smoothing deteriorate secondary class information by dismantling the natural output distribution.Source: §B.1 · p. 11

Related papers

Loading 2102.09517v1…