PaperlayerView exact revision on arXiv

Source-linked AI summary

Class-Incremental Learning: A Survey

Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, Ziwei Liu

arXiv:2302.03648v2cs.CVcs.LG

TL;DR

Class-incremental learning must acquire novel classes while retaining earlier knowledge, but direct training on new classes causes catastrophic forgetting. This paper surveys CIL methods, evaluates 17 methods across benchmark image-classification tasks, and proposes memory-aligned and memory-agnostic evaluation. Under fair comparison, the gap between dynamic networks and other methods becomes much smaller, with DER’s CIFAR100 final-accuracy advantage over iCaRL decreasing from 9.07% to 1.74%.

  • Problem

    Class-incremental learning seeks to build a classifier over all seen classes despite streaming data and catastrophic forgetting of former classes during new-class training.

  • Method

    The paper taxonomically surveys CIL methods, compares them on CIFAR100 and ImageNet benchmarks, and evaluates methods using aligned memory budgets and budget-agnostic measures.

  • Results

    Under memory-aligned comparison, dynamic networks no longer have a large gap over other methods; DER’s CIFAR100 final-accuracy advantage over iCaRL falls from 9.07% to 1.74%.

  • Takeaways & Limitations

    Exemplars and knowledge distillation are effective against forgetting, model rectification can boost other methods, and pre-trained models show strong performance.

  • Takeaways & Limitations

    Comparisons can be unfair when dynamic networks use extra model-buffer memory, and template-based classifiers can fail under substantial domain gaps when the backbone is frozen.

Abstract

from arXiv · show

Deep models, e.g., CNNs and Vision Transformers, have achieved impressive achievements in many vision tasks in the closed world. However, novel classes emerge from time to time in our ever-changing world, requiring a learning system to acquire new knowledge continually. Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally and build a universal classifier among all seen classes. Correspondingly, when directly training the model with new class instances, a fatal problem occurs -- the model tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades. There have been numerous efforts to tackle catastrophic forgetting in the machine learning community. In this paper, we survey comprehensively recent advances in class-incremental learning and summarize these methods from several aspects. We also provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms empirically. Furthermore, we notice that the current comparison protocol ignores the influence of memory budget in model storage, which may result in unfair comparison and biased results. Hence, we advocate fair comparison by aligning the memory budget in evaluation, as well as several memory-agnostic performance measures. The source code is available at https://github.com/zhoudw-zdw/CIL_Survey/

1 INTRODUCTION

Class-Incremental Learning addresses sequentially arriving, non-overlapping classes by building a classifier over all seen classes while resisting catastrophic forgetting. This survey organizes recent methods, compares them across benchmarks, and argues that memory budgets must be aligned for fair evaluation.

  • CIL setting: CIL updates a classifier as new class instances arrive, then evaluates it across all classes seen so far.The setting requires learning new classes while retaining formerly learned knowledge.
  • Core challenge: Catastrophic forgetting occurs when optimizing on new classes erases former knowledge and causes irreversible performance degradation.Resisting this forgetting is the core problem in building CIL models.
  • Related settings: CIL is harder than TIL because inference must discriminate among all classes, whereas TIL classifies only within the corresponding task space.DIL instead uses the same classes across tasks while their domains or distributions change.
  • Survey scope: The survey reviews CIL methods taxonomically and chronologically and provides unified comparisons on CIFAR100 and ImageNet100/1000.The comparison includes traditional CNN-backed and modern ViT-backed methods.
  • Evaluation: The authors advocate aligning memory budgets and using memory-agnostic measures because storage costs can bias comparisons between CIL methods.These evaluations target both high-performance computers and edge-device settings.

2 PRELIMINARIES

CIL learns from sequential tasks with disjoint label spaces, accesses only current-task data by default, and is evaluated over all seen classes. The preliminaries formalize this objective and describe exemplar storage, management, and herding-based selection.

  • Problem Formulation: CIL receives sequential tasks with disjoint label spaces and learns a model evaluated over the union of all seen classes.Only the current task dataset is accessible during each incremental step under the standard formulation.
  • Problem Formulation: The formal objective minimizes expected classification risk over the current task distribution while preserving discrimination across all seen classes.The hypothesis space and indicator function define the risk formulation.
  • Baseline: Finetuning is a CIL baseline that focuses only on new concepts and therefore suffers severe forgetting.It pays no attention to formerly learned classes during optimization.
  • Exemplars and Exemplar Set: Exemplar sets store representative instances from former tasks so training can use them together with the current dataset.This relaxes the restriction to current-task data while retaining only a relatively small memory.
  • Exemplar Set Management: Exemplar management can fix the number per class or fix total memory, with the latter assigning approximately floor(M/|Y_b|) instances per class.The survey uses the fixed-total-memory strategy.
  • Exemplar Selection: Herding selects exemplars near each class center so their average feature vector remains close to the class mean.The method iteratively appends representative instances until the per-class memory bound is reached.

3 CLASS-INCREMENTAL LEARNING: TAXONOMY

The taxonomy organizes CIL methods by their key mechanisms, while emphasizing that categories can overlap. It covers replay, regularization, dynamic networks, knowledge distillation, rectification, and template-based classification, along with their trade-offs.

  • CIL methods are organized taxonomically around seven aspects, but the categories are not mutually exclusive because techniques are shared across algorithms.The taxonomy is intended to provide holistic understanding from different perspectives rather than strict boundaries.
  • Data Replay: Replay methods revisit stored exemplars during updates, while memory-efficient variants save features or low-fidelity images to reduce storage costs.These alternatives may require adaptation because their distributions can differ from raw images.
  • Data Replay: Generative replay synthesizes former-class instances for training, but its performance depends on generated-data quality and can fail on complex, large-scale inputs.Generating features rather than raw images is proposed as a less demanding alternative.
  • Dynamic Networks: Dynamic networks expand structures or prompts to improve representation, including task-specific blocks, new backbones, and prompt-based adaptation with frozen pretrained ViTs.Backbone expansion can preserve old features, whereas prompt expansion requires less memory than storing whole backbones.
  • Regularization and Distillation: Parameter regularization, knowledge distillation, and relational distillation constrain updates by preserving important parameters, outputs, features, or structural relationships.Parameter-importance estimates can make memory grow linearly with the number of tasks.
  • Template-Based Classification: Template-based classification performs inference through prototypes or generative models, but prototype recalculation may be impossible without exemplars and frozen backbones can lose adaptability under domain gaps.Generative templates also require more inference computation than prototype-based methods.

4 EXPERIMENTAL EVALUATION

The evaluation compares 17 CIL methods across benchmark datasets and examines performance, fairness under aligned memory budgets, and memory-agnostic evaluation. Results show that replay, distillation, model rectification, dynamic networks, and pre-trained models offer distinct benefits, while memory costs materially affect comparisons.

  • Experimental setup: The study evaluates 17 CIL methods on CIFAR100 and ImageNet100/1000 using unified benchmark settings and common training splits.Methods span replay, regularization, dynamic networks, distillation, model rectification, and pre-trained ViT approaches.
  • Benchmark comparison: Finetuning performs worst, while knowledge distillation and exemplar replay substantially reduce forgetting compared with parameter regularization.Replay, Coil, and iCaRL combine exemplar replay or distillation to improve performance over simpler baselines.
  • Benchmark comparison: Pre-trained L2P obtains the best CIFAR100 performance, but direct comparison is unfair because other methods train from scratch while L2P uses an ImageNet-21K-pretrained ViT.L2P is therefore not directly comparable to the other training-from-scratch methods.
  • Benchmark comparison: TFH requires greater stability than TFS because evaluating many base classes makes preserving old-class knowledge more critical.The comparison evaluates both settings across benchmark datasets.
  • Benchmark comparison: Dynamic networks consistently achieve the strongest benchmark performance, but their advantage is associated with storing multiple backbones and using more memory.In CIFAR100 Base0 Inc10, DER has ten times as many parameters as iCaRL.
  • Memory-aligned comparison: Under aligned memory costs, DER’s final-accuracy advantage over iCaRL on CIFAR100 Base0 Inc10 falls from 9.07% to 1.74%.The authors conclude that dynamic networks and other methods occupy different dominant domains depending on available memory, with intersections between them.

5 FUTURE DIRECTIONS

Future CIL research should address more complex inputs and less restrictive data streams, including online learning without task boundaries. It should also explore compatibility across models and incremental stages.

  • CIL with Complex Inputs: Future CIL systems should handle few-shot, imbalanced, weakly supervised, multimodal, concept-drift, novel-class, and hierarchical data settings.
  • CIL with General Data Stream: CIL methods should support fully online training without requiring stored exemplars, offline multi-epoch training, or explicit task boundaries.
  • CIL with Complex Inputs: Open-world recognition and novel class discovery can extend CIL models with detection ability for unknown test classes.
  • CIL with Complex Inputs: Incrementally training language-vision models and combining pre-trained models are identified as directions for handling multimodal streams and generalization.
  • CIL with Bidirectional Compatibility: Bidirectional compatibility is presented as a design direction for interoperability between models and incremental learning stages.

6 CONCLUSION

The paper surveys class-incremental learning methods, compares them across public datasets, and uses the results to summarize common rules and future research insights.

  • The paper provides a comprehensive survey of class-incremental learning organized into seven taxonomic and chronological categories.
  • It compares different methods holistically on several publicly available datasets.
  • The evaluation results are used to discuss insights and summarize common rules intended to inspire future research.

Supplementary Material

The supplementary material adds analyses, implementation details, visualizations, and experiments that extend the paper’s benchmark and memory-aligned evaluations. It also documents selected methods and related incremental-learning topics.

  • Benchmark comparison: Additional benchmark results cover CIFAR100, ImageNet100, and ImageNet top-5 accuracy, including performance across dataset splits.The supplementary benchmark section reports detailed results for CIFAR100 and ImageNet100 before presenting ImageNet top-5 accuracy.
  • Analysis and evaluation: Supplementary analyses visualize confusion matrices and weight norms, while reporting per-stage accuracy, forgetting, and intransigence.These materials extend the main evaluation with diagnostic visualizations and additional incremental-learning measures.
  • Memory-aware evaluation: The supplementary material reports extra settings for memory-aligned comparison and details the AUC-A/L measure across memory, backbone, and incremental-performance settings.The AUC-A/L implementation is described point by point, including memory and backbone configurations.
  • Taxonomy and related topics: It also discusses the paper’s taxonomy, related incremental object detection and semantic segmentation topics, and selected compared methods.These sections provide context for the taxonomy and fuller introductions to methods used in the main comparison.
  • Additional experiments: Further experiments examine exemplar influence, computational efficiency, and replay strategies using direct, feature, and generative replay.The replay analysis compares three replay types, while other experiments study exemplar effects and efficiency.

A.1 Detailed Results of CIFAR100

The detailed results show that benchmark rankings depend on model scale, pre-training, and memory budgets, motivating more equitable comparisons across datasets.

  • CIFAR100: L2P consistently achieves the best performance across CIFAR100 settings, but uses an 85M-parameter ImageNet-21K pre-trained Vision Transformer.
  • CIFAR100: Typical methods instead train ResNet32 from scratch with 0.46M parameters, making direct comparisons with pre-trained methods unfair.
  • CIFAR100: Dynamic networks such as DER and MEMO obtain better performance at the cost of additional parameters and memory budget.
  • ImageNet100: ImageNet100 results are consistent with smaller datasets, with dynamic networks performing better while requiring more memory budget.
  • ImageNet100: GEM and L2P are excluded from ImageNet comparisons because GEM requires a large quadratic-programming matrix and L2P uses a pre-trained backbone overlapping the target dataset.

APPENDIX B VISUALIZATION RESULTS

The visualization results compare forgetting, exemplar effects, bias, and dynamic-network performance across CIL methods. They indicate that distillation and exemplars mitigate forgetting, while dynamic networks remain highly competitive.

  • Finetune and EWC forget most former-class knowledge, whereas LwF reduces forgetting through knowledge distillation.
  • Exemplars brighten confusion-matrix diagonals, showing that replay alleviates forgetting and supports steadier performance improvement.
  • Data regularization with exemplars performs worse than knowledge distillation for using exemplars efficiently, as seen when comparing GEM with iCaRL.
  • BiC and WA improve iCaRL by rectifying model bias, indicating that rectification can complement other CIL methods.
  • Dynamic networks, including DER, FOSTER, MEMO, and DyTox, show the most competitive visualized performance and help retrieve old features.

APPENDIX C SUPPLIED RESULTS IN MEMORY-ALIGNED COMPAR-

This section evaluates CIL methods under aligned memory budgets and introduces memory-agnostic performance analysis. The results show that budget alignment can change method rankings and that AUC summarizes performance across budgets.

  • Memory alignment exchanges model capacity and exemplars, accounting for parameters, exemplar counts, and total memory size.
  • Aligning total memory budgets narrows performance gaps between methods, with iCaRL achieving the best average accuracy in CIFAR100 Base0 Inc5.
  • At 7.4 MB iCaRL has higher accuracy, whereas at 23.5 MB DER performs better, so no single method suits every budget.
  • AUC evaluates the area under the incremental performance-memory curve, avoiding dependence on one specific memory budget.

BiC WA

The CIFAR100 memory-scale experiments examine how aligned budgets distribute resources between exemplars and dynamic backbones. Dynamic methods become competitive or superior as stronger backbones become affordable.

  • At 7.6 MB, DER and MEMO use small ConvNets and perform poorly in the base session because their backbones limit learning ability.
  • At 12.4 MB, stronger ResNet14 backbones make DER and MEMO competitive and superior to other methods.
  • At 16.0 MB, DER and MEMO use ResNet20 and outperform other methods with stronger backbones.
  • At 19.8 MB, DER and MEMO use ResNet26 and continue to show competitive performance with stronger representation ability.

D.2.2 ImageNet100 Implementations

ImageNet100 implementations extend memory-aligned comparisons across six budgets, trading exemplar capacity against backbone strength. Dynamic networks are restricted at the smallest budget but outperform in the reported 493 MB setting.

  • At 329 MB, DER and MEMO use small four-layer ConvNets and are restricted by inferior backbones that perform poorly on the base task.
  • At 493 MB, dynamic networks switch to ResNet10 and outperform other methods with stronger backbones.
  • At 755 MB, SingleNet exchanges 2970 exemplars while dynamic networks switch to ResNet18 for stronger representation ability.
  • At 1273 MB, SingleNet exchanges 6574 exemplars and DER and MEMO use ResNet50.

APPENDIX F FORGETTING AND INTRANSIGENCE MEASURE

The appendix defines forgetting and intransigence to separately measure retention of earlier tasks and learning difficulty on new tasks. It reports their relationship with average last performance and regularization.

  • Forgetting: Forgetting measures the gap between a task’s best previous performance and its final performance.The single-task measure is f_j,k = max l∈{1,···,k−1} a_l,j − a_k,j for j < k.
  • Forgetting: Average forgetting aggregates forgetting across prior tasks at the final learning stage.The last task is excluded because it does not yet suffer forgetting, leaving B − 1 terms.
  • Intransigence: Intransigence measures the gap between joint-training performance and incrementally learned performance on new tasks.Joint training uses all seen task data, while incremental training proceeds task by task.
  • Empirical findings: Higher average last performance generally corresponds to lower forgetting.The relationship arises because last performance contributes to the forgetting calculation.
  • Empirical findings: Methods with less regularization tend to achieve better intransigence, with finetune performing best on this measure.Regularization terms can reduce plasticity and widen the gap from joint-training performance on new tasks.

APPENDIX G EXPERIMENT ON REPLAY STRATEGIES

The appendix compares image, feature, and generative replay under aligned memory budgets. Image replay performs best when memory is limited, while feature-only and generative replay have important weaknesses.

  • Replay strategies: Replay strategies differ in what they store: raw images, encoded features, or generated exemplars.All strategies update the model using the current task data concatenated with stored exemplars.
  • Feature replay: Feature replay reduces storage and can alleviate privacy concerns, but changing embedding functions make stored features incompatible across stages.Feature replay therefore requires alignment or mapping into a common embedding space.
  • Generative replay: Generative replay fails in the evaluated setting, indicating that its generative model also suffers catastrophic forgetting.The comparison uses aligned memory costs across replay methods.
  • Replay strategies: Feature replay alone cannot achieve competitive performance, whereas hybrid replay can be competitive with image replay.The hybrid strategy stores both images and features within divided memory budgets.
  • Replay strategies: When memory is limited, image replay shows the best performance without requiring other tuning techniques.The appendix characterizes image replay as a simple effective solution when privacy restrictions are absent.

APPENDIX H INTRODUCTION ABOUT COMPARED METHODS

The appendix introduces the compared CIL methods and relates them to the survey’s seven-part taxonomy. The set spans replay, regularization, dynamic networks, distillation, rectification, CNNs, ViTs, and pretrained ViTs.

  • Compared methods: The comparison includes 17 methods selected to cover all seven taxonomy aspects and the development timeline of CIL.It includes early methods, recent state-of-the-art methods, CNN-based methods, ViT-based methods, and pretrained ViT-based methods.
  • Taxonomy: Replay methods revisit exemplars, while data and parameter regularization use exemplars or parameter importance to constrain updates.Replay concatenates exemplars with current data; GEM uses exemplars as indicators, and EWC uses Fisher information.
  • Dynamic networks: Dynamic-network methods expand or adapt network structure to improve representation while balancing stability and plasticity.Examples include DER’s expanding backbones, AANets’ dual branches, and task-specific tokens or prompts in DyTox and L2P.
  • Knowledge distillation: Knowledge-distillation methods preserve prior knowledge through distilled outputs or features, with iCaRL additionally using exemplar replay and template-based classification.PODNET distills diverse pooled features, while Coil addresses crossmodal distillation.
  • Model rectification: Model-rectification methods correct classifier bias through an added correction layer or normalization of fully connected layers.BiC learns a bias-correction layer, whereas WA normalizes fully connected layers.

APPENDIX J FURTHER ANALYSIS

The appendix analyzes how exemplar count affects accuracy and compares computational cost under fixed and memory-aligned replay budgets. More exemplars improve performance but also affect running time.

  • Exemplar-count analysis: Exemplar replay performance is bounded by exemplar-set size, and saving more historical instances commonly improves performance.The analysis varies exemplar counts for Replay, BiC, WA, PODNet, and iCaRL on CIFAR100 B0 Inc10.
  • Exemplar-count analysis: Figure 29 compares average and last accuracy as exemplar number changes on CIFAR100 Base0 Inc10.The figure focuses on how accuracy varies with the number of stored exemplars.
  • Computational efficiency: Running time is evaluated on a single NVIDIA 3090 GPU for methods using 2000 exemplars and exemplars aligned to DER’s memory cost.Figure 30 distinguishes the fixed-exemplar time from the additional time under aligned exemplar budgets.
  • Computational efficiency: Equipping methods with more exemplars leads to better performance, while the appendix also evaluates the resulting computational-efficiency trade-off.The running-time study addresses high-throughput streams where revisiting many past samples can become a bottleneck.
Loading 2302.03648v2…