PaperlayerView exact revision on arXiv

Source-linked AI summary

Class-incremental learning: survey and performance evaluation on image classification

Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, Joost van de Weijer

arXiv:2010.15277v3cs.LGcs.CV

TL;DR

Incremental learning seeks to acquire sequentially arriving knowledge without catastrophic forgetting, especially when class-incremental systems must recognize all classes without a task-ID. The paper surveys class-IL methods and evaluates thirteen of them across datasets, domain shifts, and network architectures, finding that performance depends on bias correction and architecture, while large domain shifts remain challenging.

  • Problem

    Class-incremental learners must preserve earlier knowledge while learning new classes and distinguish all seen classes without a task-ID, but catastrophic forgetting remains a central challenge.

  • Method

    The paper surveys class-IL approaches organized around regularization, rehearsal, and bias correction, and experimentally compares thirteen methods across incremental-learning scenarios and architectures.

  • Results

    Methods addressing task-recency bias generally perform better; BiC reaches state-of-the-art in several experiments, while rankings change substantially with network architecture.

  • Takeaways & Limitations

    Class-IL performance depends on both the method and the network architecture, and approaches effective in smaller-shift settings do not generally dominate under large domain shifts.

  • Takeaways & Limitations

    Most existing results use small domain shifts, and under large domain shifts no method significantly outperforms the FT-E baseline.

Abstract

from arXiv · show

For future learning systems, incremental learning is desirable because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored -- also important when privacy limitations are imposed; and learning that more closely resembles human learning. The main challenge for incremental learning is catastrophic forgetting, which refers to the precipitous drop in performance on previously learned tasks after learning a new one. Incremental learning of deep neural networks has seen explosive growth in recent years. Initial work focused on task-incremental learning, where a task-ID is provided at inference time. Recently, we have seen a shift towards class-incremental learning where the learner must discriminate at inference time between all classes seen in previous tasks without recourse to a task-ID. In this paper, we provide a complete survey of existing class-incremental learning methods for image classification, and in particular, we perform an extensive experimental evaluation on thirteen class-incremental methods. We consider several new experimental scenarios, including a comparison of class-incremental methods on multiple large-scale image classification datasets, an investigation into small and large domain shifts, and a comparison of various network architectures.

1 INTRODUCTION

Incremental learning trains on sequentially arriving tasks while preserving prior knowledge, with class-incremental learning requiring discrimination among all seen classes without a task-ID. This survey organizes class-IL methods and evaluates thirteen approaches across datasets, domain shifts, and architectures.

  • 1 INTRODUCTION: Incremental learners receive one task’s data at a time but must perform all previously learned tasks after each training session.This setting contrasts with jointly supervised training, where labeled data for all tasks is available together.
  • 1 INTRODUCTION: Catastrophic forgetting is the central challenge: finetuning on new-task data causes a drastic performance drop on previous tasks.Incremental learning must also avoid intransigence, the resistance to learning new tasks.
  • 1 INTRODUCTION: Class-incremental learning omits the task-ID at inference, requiring discrimination among classes from all previous and current tasks.Task-incremental learning provides the task-ID, making cross-task class discrimination unnecessary.
  • 1 INTRODUCTION: The survey groups class-IL solutions into regularization, exemplar storage, and approaches that directly address task-recency bias.Task-recency bias favors classes from recently learned tasks.
  • 1 INTRODUCTION: Exemplar-free data regularization outperforms weight regularization, while weight regularization can combine better with exemplars than data regularization in some scenarios.These findings are reported in the survey’s performance evaluation.
  • 1 INTRODUCTION: The evaluation compares thirteen methods across multiple scenarios, including large-scale datasets, domain shifts, and network architectures.The authors also provide an extensible evaluation framework and reproducibility code.

2 CLASS-INCREMENTAL LEARNING

Class-incremental learning studies sequential supervised tasks whose classes must be jointly distinguished without a task oracle, under memory, privacy, and scalability constraints. Its core difficulties include catastrophic forgetting, intransigence, inter-task confusion, and recency bias.

  • 2 CLASS-INCREMENTAL LEARNING: Incremental learning is motivated by limited storage, privacy restrictions, computational cost, and the goal of learning throughout an extended lifecycle.The paper connects these constraints to robotics, health applications, and sustainable AI.
  • 2 CLASS-INCREMENTAL LEARNING: Class-IL evaluates a network on all classes seen so far without access to the task-ID, unlike task-IL, which restricts evaluation to the known task.The survey considers disjoint class sets and sequential training sessions.
  • 2.2 General class-incremental learning setup: The survey’s offline setup presents i.i.d. task data in sessions, permits multiple passes over current data, optionally stores exemplars, and excludes online one-pass learning.Tasks are class-disjoint in the experiments.
  • 2 CLASS-INCREMENTAL LEARNING: Class-IL classifiers commonly use cross-entropy, with variants normalizing over all seen classes or only current-task classes.Without exemplars, current-task-only normalization leads to less forgetting in the reported analysis.
  • 2 CLASS-INCREMENTAL LEARNING: The survey narrows comparable methods to task-agnostic, offline learners with fixed architectures, while separately identifying exemplar-free methods as privacy-relevant.Exemplar-free methods do not require storing image data from previous tasks.
  • 2.4 Challenges of class-incremental learning: The stability-plasticity dilemma requires retaining previous knowledge while acquiring current-task knowledge; focusing only on the new task causes catastrophic forgetting and can produce intransigence.The paper identifies sequential training and all-class evaluation as sources of this tension.

3 APPROACHES

The survey organizes class-incremental approaches into regularization, rehearsal, and bias-correction methods, each targeting different sources of catastrophic forgetting and task-recency bias.

  • 3 APPROACHES: Class-incremental methods are grouped into regularization-based, rehearsal-based, and bias-correction approaches.Regularization limits changes to important weights or representations, rehearsal stores exemplars, and bias correction addresses preference for recently learned classes.
  • 3 APPROACHES: Weight regularization penalizes changes to parameters deemed important after earlier tasks.EWC uses a diagonal Fisher Information approximation, while PathInt and MAS estimate importance online through learning trajectories or function sensitivity.
  • 3 APPROACHES: Data regularization uses knowledge distillation to limit drift in representations learned from previous tasks.LwF applies temperature-scaled predictions from the previous network, but its efficacy drops significantly under large domain shifts.
  • 3 APPROACHES: Exemplar rehearsal stores selected samples from previous classes, with fixed memory trading per-class representation against bounded capacity and growing memory increasing storage linearly.Equal exemplars per class are enforced in both memory schemes.
  • 3 APPROACHES: Distillation can hurt performance when combined with exemplars, whereas weight regularization may combine beneficially with exemplar rehearsal.The survey reports that exemplar-based distillation can reduce performance, while some weight-regularization combinations improve results.
  • 3 APPROACHES: Bias-correction methods target task-recency bias caused by seeing many examples from the newest task and few or none from earlier tasks.This produces larger classifier norms and more favorable biases for new classes, making previous classes less likely to be selected.

4 RELATED WORK

Related work spans surveys and methods for task-incremental, online, variational, generative, architectural, and non-classification settings, while this survey emphasizes offline class-incremental evaluation.

  • 4 RELATED WORK: Earlier surveys primarily addressed catastrophic forgetting, lifelong learning, robotics, or task-incremental learning rather than class-incremental evaluation.Some task-incremental evaluations were extensive but limited in scope, including a two-task paradigm.
  • 4 RELATED WORK: Concurrent class-incremental surveys differ in setting and coverage: one studies online learning, while this survey focuses on offline class-incremental learning.This survey also includes several methods excluded from another class-incremental comparison.
  • 4 RELATED WORK: Mask-based parameter isolation can prevent forgetting but requires different forward passes, making it impractical for task-agnostic evaluation.The limitation arises because simultaneous evaluation of all learned tasks is not possible with task-specific masks.
  • 4 RELATED WORK: Architecture-growing methods add capacity to preserve prior knowledge, but can become impractical without task IDs or when parameter growth is excessive.RPS avoids requiring a task ID at inference but uses a much larger customized architecture, limiting capacity comparability.
  • 4 RELATED WORK: Online incremental learning exposes the learner to each example once, with GEM constraining gradients using per-task exemplar memory.A-GEM is described as an improvement over GEM.
  • 4 RELATED WORK: Pseudo-rehearsal generates synthetic examples to avoid storing exemplars and their associated privacy issues.DGR uses an unconditional GAN, while MeRGAN uses a label-conditional GAN and replay alignment.

5 EXPERIMENTAL SETUP

The evaluation uses a controlled framework, multiple datasets and architectures, explicit incremental scenarios, baseline variants, and continual hyperparameter selection to compare class-incremental methods fairly.

  • 5 EXPERIMENTAL SETUP: The framework synchronizes dataset partitions, data order, randomness, and seeds while withholding prior-task data except exemplar memory.These controls establish identical initial conditions and splits across methods.
  • 5 EXPERIMENTAL SETUP: The implementation evaluates thirteen named class-incremental methods, extends most exemplar-free methods with exemplars, and supports varied network architectures.The framework also supports the experimental scenarios used in the paper.
  • 5 EXPERIMENTAL SETUP: Experiments cover nine image-classification datasets, including CIFAR-100 and several fine-grained datasets with higher-resolution images.The datasets include Flowers, MIT Indoor Scenes, CUB Birds, Stanford Cars, Aircraft, and Stanford Actions.
  • 5 EXPERIMENTAL SETUP: Average accuracy summarizes performance across tasks, while forgetting and intransigence provide complementary measures of stability and plasticity.The survey notes that forgetting and intransigence are less useful in class-incremental learning than in task-incremental learning.
  • 5 EXPERIMENTAL SETUP: Finetuning baselines include FT, FT+, and FT-E, with exemplars improving performance and helping prevent catastrophic forgetting.Joint training over all seen data serves as an upper-bound reference.
  • 5 EXPERIMENTAL SETUP: The continual hyperparameter framework selects parameters using only information available at each task rather than future-task data.It combines maximal-plasticity search with stability-decay search to identify a stability-plasticity trade-off.
  • 5 EXPERIMENTAL SETUP: Most experiments use ResNet-32 for CIFAR-100 and ResNet-18 for larger-resolution datasets, with additional architecture analysis.Both architectures are trained from scratch for most experiments.
  • 5 EXPERIMENTAL SETUP: The main scenarios split datasets into multiple tasks with equal or unequal class counts, including (10/10) and (11/50-5).Single-class tasks are excluded because several methods cannot be straightforwardly applied to them.

6 EXPERIMENTAL RESULTS

The evaluation compares class-incremental methods across memory sizes, domain shifts, datasets, and architectures. Performance rankings depend strongly on the setting, while bias correction and exemplar strategies involve trade-offs.

  • Regularization methods: Regularization methods without exemplars suffer a significant performance drop in class-incremental learning, especially after five and ten tasks.
  • Exemplar comparisons: FT-E obtains the best results among most exemplar-augmented regularization comparisons, while combining exemplars with regularization can hurt performance.LwM-E is an exception in some short-sequence cases.
  • Bias correction: EEIL and LUCIR achieve better overall performance, whereas iCaRL, BiC, and IL2M address task-recency bias more effectively but retain performance headroom.
  • Memory size: 6.2 points are gained by expanding BiC from 10 to 20 exemplars per class, compared with 4.8 points from expanding from 20 to 40.The second expansion doubles memory size, making additional exemplars less cost-effective after 20 per class.
  • Exemplar sampling: Herding is slightly preferred overall, but random sampling is closely tied and both outperform the other sampling strategies.Differences between random and herding are generally not statistically significant.
  • Network architectures: Method rankings can change completely across architectures: iCaRL performs best without skip connections, whereas BiC performs best with ResNet-18.MobileNet also achieves competitive results despite its lower parameter and operation count.
  • Large-scale datasets: 32.4% and 30.2% average accuracy are achieved by BiC and iCaRL after 25 ImageNet tasks, respectively, while EWC-E and FT-E outperform several alternatives.

7 EMERGING TRENDS IN CLASS-IL

Emerging class-incremental learning research explores more storage-efficient rehearsal, unsupervised representation learning, alternative losses, meta-learning, and task-free settings. These directions address limitations of stored images, supervised data, and clearly separated tasks.

  • Exemplar learning: Parametrized exemplars are optimized to prevent forgetting, enabling more efficient use of available storage.
  • Feature rehearsal: Feature rehearsal generates or stores hidden-layer features instead of images, making rehearsal more applicable to complex datasets and reducing storage requirements.Image-generation approaches struggle to produce realistic complex images.
  • Self- and unsupervised incremental learning: Unsupervised incremental learning remains relatively underexplored, including methods for explicit task classification, mixture modeling, and smooth task transitions.
  • Beyond cross-entropy loss: Replacing cross-entropy with metric-learning or energy-based losses has been associated with less forgetting.The paper suggests combining these losses with rehearsal and bias correction.
  • Meta-learning: Meta-learning methods learn parameters or representations intended to reduce interference and promote future transfer, with initial evidence on small datasets.
  • Task-free settings: Task-free settings remove clear task boundaries and instead involve gradually changing class distributions, making the transition from task-aware learning non-straightforward.

8 CONCLUSIONS

The survey organizes class-incremental approaches around regularization, rehearsal, and bias correction, then evaluates thirteen methods across varied scenarios and architectures. Results favor exemplar rehearsal and explicit task-recency correction, while performance depends on architecture and domain shift.

  • Exemplar-free methods: LwF obtains the best results among exemplar-free methods, while exemplar-free methods do not currently match exemplar-rehearsal methods.The authors recommend comparing these settings separately because exemplar-free methods operate under more restrictive storage conditions.
  • Combining methods: Adding LwF regularization to exemplars does not improve performance; finetuning with exemplars performs better.This confirms that the added distillation term can hurt when exemplars are already available.
  • Distillation: GD obtains excellent class-incremental results by specializing on the current task and combining its knowledge with previous-task knowledge through distillation.
  • Regularization: Weight regularization can outperform data regularization in several scenarios, and EWC-E significantly outperforms LwF-E in the reported experiments.
  • Exemplar selection: Herding is marginally better than random exemplar sampling for longer task sequences, but the difference is statistically significant for only some methods.
  • Task-recency bias: Bias-correction methods generally perform better for class-IL; BiC reaches state-of-the-art in several experiments, while LUCIR depends on a good starting representation.
  • Domain shifts: Large domain shifts remain a boundary for current methods, and most results have focused on small shifts; new techniques are needed for satisfactory class-IL performance in large-shift settings.
  • Architectures: Class-IL methods work across varied network architectures, but most are architecture-sensitive and method rankings change with skip connections.

APPENDIX A IMPLEMENTATION AND HYPERPARAMETERS

The appendix documents the datasets, hyperparameter-search procedure, shared experimental settings, and method-specific implementations used to make the comparisons reproducible.

  • Datasets: The study evaluates continual-learning methods on nine image-classification datasets, with CIFAR-100 using 32 × 32 color images across 100 classes.
  • Hyperparameter search: The Continual Hyperparameter Framework searches stability-plasticity trade-off parameters after a maximal-plasticity learning-rate search.
  • Framework: The implementation section lists the evaluated methods and supports exemplar extensions, multiple network architectures, and the paper’s experimental scenarios.
  • Method implementations: LwF combines distillation with cross-entropy and uses temperature scaling T = 2; with exemplars, distillation is also applied to previous-class exemplars.
  • Method implementations: EWC, MAS, and RWalk use fixed importance-weight and Fisher-information settings, with method-specific damping or trade-off values.
  • Method implementations: DMC uses resized ImageNet as an auxiliary dataset with batch size 128, while GDumb uses random exemplar sampling, model reinitialization, and cutmix.
  • Method implementations: iCaRL combines distillation and cross-entropy during training but uses nearest-mean-of-exemplars evaluation instead of softmax outputs.
  • Method implementations: EEIL uses balanced and unbalanced training phases, whereas BiC retains the original class-count-scaled distillation trade-off and a 10% validation split.

B.1 More on CIFAR-100

Additional CIFAR-100 analyses examine random-seed variability, growing-memory behavior, and Random Path Selection under a comparable fixed-memory setup.

  • Variability: CIFAR-100 experiments use 10 fixed random seeds shared across approaches to support fair comparisons.For most approaches and scenarios, standard deviation is below 2.5, although several regularization methods and iCaRL vary more in the initial larger-task setting.
  • Growing memory: With 20 exemplars per class and growing memory, GD, BiC, EEIL, and IL2M achieve the best results after 10 tasks.Most methods show less forgetting with fixed memory because it stores more exemplars during early tasks.
  • Variability: Table S2 reports mean and standard deviation of average accuracy over 10 runs for different CIFAR-100 scenarios.

B.3 On semantic tasks

The appendix tests whether class ordering affects incremental-learning results by constructing semantically grouped and difficulty-ordered CIFAR-100 tasks. It also notes that randomized task orderings may suffice when exemplars are used.

  • Semantic ordering: Random class ordering is common in CIFAR-100 experiments, but semantic class orderings can reveal different method results.
  • Table guide: Table S3 compares RPS with different numbers of paths and baselines on CIFAR-100 (10/10) with fixed 2,000-exemplar memory.
  • Why ordering matters: Joint training can learn features that distinguish otherwise-confusable classes, whereas incremental learning makes such features harder to learn or easier to modify across tasks.
  • Experimental design: The study groups CIFAR-100 classes using 20 coarse labels, orders groups by classification difficulty, and trains a dedicated joint model to estimate that difficulty.
  • Figure guide: Figure S1 reports CIFAR-100 results for 20 exemplars per class under growing memory in both (10/10) and (11/50-5) settings.
  • Figure guide: Figure S2 reports class-ordering results on CIFAR-100 with ResNet-32 trained from scratch, using herding for 20 exemplars per class and six-run standard-deviation bars.
  • Implication: For methods using exemplars, evaluating randomized task orderings often suffices.

B.4 More on sampling strategies

Sampling strategies perform similarly in shorter class-incremental sequences, while herding and inverse boundary-avoidance strategies become more favorable as task sequences grow.

  • Performance is very similar across sampling methods in the CIFAR-100 (11/50-5) scenario.
  • Herding has a slight benefit over other sampling strategies for longer task sequences.
  • For shorter sequences, no sampling strategy consistently outperforms the others.
  • Inverse entropy and inverse distance perform better than their standard counterparts for larger task sequences.These variants select samples farther from task boundaries and closer to class centers.
  • Mann–Whitney U-tests find no significant difference between random and herding exemplar selection in most cases.

B.5 More on external data

The evaluation compares exemplar-based and external-data approaches across datasets and domain-shift settings, finding limited benefit from additional-data distillation but a large advantage for EWC-E under small domain shifts.

  • DMC achieves 25.9% average accuracy on CIFAR-100 (10/10) with external data, versus 20.6% when using exemplars instead.The external-data version provides privacy-preserving properties at some performance cost when replaced by exemplar memory.
  • The supplementary evaluation covers sampling strategies, external-data variants, small domain shifts, and large-scale scenarios.The listed tables include CIFAR-100 sampling experiments, VGGFace2 small-domain-shift results, and ImageNet-Subset-100 architecture results.
  • GD with external data reaches 44.6%, compared with 43.7% for GD without exemplars.The reported gain from distillation using an additional dataset is relatively small.
  • In small-domain-shift experiments, EWC-E achieves 70.0% accuracy, while LwF-E achieves 43.5%.

B.7 More on network architectures

The study evaluates class-incremental methods across multiple network architectures and large-scale image-classification scenarios, with BiC showing the least forgetting but slightly lower immediate task performance.

  • The architecture comparison includes AlexNet, VGG-11, ResNet-18, and WideResNet-50.AlexNet and VGG-11 represent networks with initial convolutional layers followed by fully connected layers; ResNet-18 represents residual architectures.
  • BiC exhibits the least forgetting across the evaluated network architectures.It sometimes has positive forgetting, indicating improved performance on some tasks after subsequent learning.
  • BiC’s lower forgetting comes with slightly lower performance for each task immediately after that task is learned.
  • The evaluation also reports ImageNet (10/100) and ImageNet (11/500-50) scenarios on ResNet-18 with growing exemplar memory.
  • Architecture results report task accuracy when learned, forgetting after all classes, and average accuracy after 10 tasks.
Loading 2010.15277v3…