Source-linked AI summary
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Eden Belouadah, Adrian Popescu, Ioannis Kanellos
TL;DR
Class-incremental learning must incorporate new classes without catastrophic forgetting, but fixed-size-model methods had not been compared as thoroughly as model-growth approaches. The paper unifies their analysis and evaluation across memory and incremental-state settings, finding that no algorithm is best universally and that method choice depends on memory, computation, and data-stream characteristics.
Problem
Class-incremental learning must integrate new data while avoiding catastrophic forgetting, and fixed-size-model methods lacked a thorough common comparison.
Method
The paper formalizes fixed-complexity class-incremental learning and evaluates fine-tuning and fixed-representation methods across datasets, memory conditions, and incremental states.
Results
No algorithm is best in all configurations; with memory, imbalanced-learning treatment with vanilla fine-tuning and bias rectification performs best globally, while distillation becomes competitive with ample memory and few states.
Takeaways & Limitations
Method selection should depend on computation and storage capacities together with the expected characteristics of the data stream.
Takeaways & Limitations
The evaluation assumes streamed new data are readily annotated and uses controlled datasets, fixed class counts per state, and non-imbalanced data.
Abstract
from arXiv · showhide
The ability of artificial agents to increment their capabilities when confronted with new data is an open challenge in artificial intelligence. The main challenge faced in such cases is catastrophic forgetting, i.e., the tendency of neural networks to underfit past data when new ones are ingested. A first group of approaches tackles forgetting by increasing deep model capacity to accommodate new knowledge. A second type of approaches fix the deep model size and introduce a mechanism whose objective is to ensure a good compromise between stability and plasticity of the model. While the first type of algorithms were compared thoroughly, this is not the case for methods which exploit a fixed size model. Here, we focus on the latter, place them in a common conceptual and experimental framework and propose the following contributions: (1) define six desirable properties of incremental learning algorithms and analyze them according to these properties, (2) introduce a unified formalization of the class-incremental learning problem, (3) propose a common evaluation framework which is more thorough than existing ones in terms of number of datasets, size of datasets, size of bounded memory and number of incremental states, (4) investigate the usefulness of herding for past exemplars selection, (5) provide experimental evidence that it is possible to obtain competitive performance without the use of knowledge distillation to tackle catastrophic forgetting and (6) facilitate reproducibility by integrating all tested methods in a common open-source repository. The main experimental finding is that none of the existing algorithms achieves the best results in all evaluated settings. Important differences arise notably if a bounded memory of past classes is allowed or not.
1. Introduction
The paper studies fixed-size-model class-incremental learning, where catastrophic forgetting must be managed while integrating new classes. It establishes a shared framework, evaluates methods across varied settings, and finds that algorithm performance depends strongly on experimental configuration.
- Catastrophic forgetting is the tendency of neural networks to underfit past data when new data are ingested.
- The study defines six desirable properties for incremental learning algorithms: complexity, memory, accuracy, timeliness, plasticity, and scalability.Scalability concerns learning up to tens of thousands of classes for complex real-world applications.
- The authors propose a unified formalization of class-incremental learning and use it to analyze algorithms, results, and combinations of algorithmic components.
- The evaluation spans four public visual-task datasets, three incremental-state splits, three past-memory sizes, and settings with no past memory.
- Herding improves exemplar selection for all tested methods, while competitive performance is also possible without knowledge distillation.Without distillation, the paper uses vanilla fine-tuning with imbalanced-learning modeling and thresholding to reduce bias between past and new classes.
- None of the existing class-incremental algorithms performs best in every configuration, with especially significant changes when memory is allowed versus disallowed.The authors integrate tested methods into a common open-source repository to facilitate reproducibility and extension to other datasets.
2. Related work
Related work is organized around model-growth, fixed-representation, fine-tuning, and online prototype or topology-based approaches. The paper focuses on methods compatible with constant model complexity and bounded or absent memory, while comparing their trade-offs across incremental-learning properties.
- Incremental-learning approaches are categorized into three main groups and mapped to six desirable properties to support a global assessment and recommended use cases.
- Model-Growth methods: Model-growth methods increase deep-model size or use multiple models to preserve past knowledge, but some approaches become difficult to scale across many tasks.Examples include Growing a Brain, Progressive Neural Networks, network-of-experts architectures, PackNet, and Piggyback.
- Online and prototype-based methods: Online unsupervised methods such as SOMs and Neural Gas require supervised adaptations for comparability, while topology-preserving methods represent and stabilize feature-space structure.ILVQ instead inserts prototypes according to training-data distributions but continuously increases its architecture to store learned patterns.
- Fixed-Representation methods: Fixed-representation methods freeze the deep representation after initial training and can reuse initial classifier weights, although earlier implementations used that representation suboptimally.Deep-SLDA updates class means and a shared covariance matrix while predicting by nearest Gaussian in feature space.
- Fine-Tuning methods: Fine-tuning methods commonly use distillation, while iCaRL combines Learning without Forgetting with exemplar memory and an external nearest-mean classifier to reduce new-class bias.Learning without Memorizing avoids past-class memory through an attention-distillation penalty, whereas other systems use separate past and new-class networks with double distillation.
3. Problem formalization
The paper formalizes class-incremental learning as sequential recognition of expanding class sets using complete data for new classes and bounded exemplars for past classes. It describes models, losses, score correction, memory management, and exemplar selection within this setting.
- 3. Problem formalization: Class-incremental learning starts from an initial model and incrementally adds new classes while retaining recognition of all previously learned classes.At state S_t, the model recognizes N_t = P_0 + P_1 + ... + P_t classes and is trained from the previous model M_{t−1}.
- 3. Problem formalization: Each incremental state provides all data for new classes but only a bounded exemplar subset of past-class data.The bounded memory is redistributed across an increasing number of past classes, creating an imbalance favoring new classes.
- 3. Problem formalization: Models may separate feature extraction from classification, with F_t producing compact image representations and C_t converting them into class predictions.The classifier can be an end-to-end neural layer, a nearest-class-mean classifier, or a set of linear SVMs.
- 3. Problem formalization: The classifier’s final fully connected layer uses a weight matrix and bias vector whose class dimension grows with the number of recognized classes.The feature dimension d depends on the CNN architecture, while the classifier parameters have sizes (d, N_t) and N_t.
- 3.2. Score bias correction: Score-bias correction treats limited past data as an imbalance problem and rescales new-class scores so past classes remain competitive during inference.Methods differ in whether they learn scaling factors from validation data, derive them from class statistics, or normalize classifier weights.
- 3.3. Past memory management: The bounded memory keeps |K|/N_t images per class, so each past class progressively loses exemplars as new classes arrive.This degradation biases the model toward new classes and can cause underfitting of past classes.
- 3.3. Past memory management: Dynamic herding selects exemplars using an updated mean of already selected examples to better approximate each class center than static selection.The study compares herding with random and other selection techniques and concludes that herding is useful for all tested methods.
4. Fine-Tuning based IL algorithms
Fine-tuning-based class-incremental methods combine model updates with distillation, bias correction, classifier normalization, or memory-dependent replay. The section also considers simpler variants that remove distillation or calibrate predictions to address the bias toward new classes.
- Most tested approaches update the model at every incremental state, often combining distillation with a separate component for balancing past and new classes.Memory usage may be compulsory or non-compulsory depending on the method.
- LwF uses distillation to preserve past-class scores without storing past data, whereas iCaRL adds bounded-memory replay and a Nearest-Class-Mean classifier.The bounded memory enables efficient replay and helps counter imbalance between past and new classes.
- LUCIR combines cosine normalization, a less-forget constraint, and inter-class separation to balance classifiers and preserve past-class geometry.Its NCM version requires memory, while its network-output version can operate with or without memory.
- Vanilla fine-tuning is the simplest update rule but is strongly affected by catastrophic forgetting without memory and becomes an interesting baseline when memory is available.FT_NEM removes distillation and uses a NEM classifier, but it requires bounded memory.
- Bias-correction variants include balanced fine-tuning, a separately trained BiC layer, classifier normalization and calibration, and thresholding at inference.These approaches use different mechanisms to reduce prediction bias toward new classes, with several requiring past exemplars or validation data.
5. Fixed-Representation based IL algorithms
Fixed-representation methods preserve a learned feature extractor while adapting only a classifier or classifier statistics across incremental states. They avoid or reduce repeated end-to-end representation learning, with REMIND additionally compressing intermediate representations for replay.
- Fixed-Representation freezes the initial model’s layers except the classification layer during later incremental states.This avoids requiring a bounded memory for updates, although the frozen representation can limit performance.
- DeeSIL replaces the DNN classification layer with linear SVMs, applying transfer learning incrementally with faster classifier optimization than end-to-end fixed-representation training.
- Deep-SLDA freezes the feature extractor, updates class means and shared covariance online, and classifies images using the closest Gaussian in feature space.It does not need to store past class data and therefore functions without memory.
- REMIND uses a fixed feature decomposition and Product Quantization to store compressed intermediate representations for fast learning and memory consolidation.The compact vectors can preserve substantially more past data than raw images; all ILSVRC can fit when |K| = 20000.
6. Experimental setup
The evaluation compares class-incremental methods across diverse visual datasets, memory budgets, and numbers of incremental states. Performance is measured primarily with incremental top-5 accuracy and summarized globally using the incremental learning gap.
- Experiments compare all presented incremental approaches with Full, a classical non-incremental training procedure that serves as the upper bound.
- Four datasets cover object, face, and landmark recognition to assess the adaptability and robustness of the tested methods.The datasets are ILSVRC, VGGFACE2, Google Landmarks, and CIFAR100.
- The protocol varies three memory sizes and three incremental-state settings, while also evaluating the case where no past memory is allowed.The tested parameters are identified as especially important for incremental-learning performance.
- Dataset details and implementations of the tested approaches and adaptations are publicly available to facilitate reproducibility.
- Top-5 accuracy is averaged over incremental states only, excluding the initial non-incremental state.Memory size and the number of states are varied to evaluate robustness.
- GIL measures the average performance gap between classical learning and each incremental configuration across tested configurations.Its denominator prevents an individual configuration from exerting an exaggerated influence on the global score.
7. Results and discussion
Results vary substantially with memory size and the number of incremental states, and no algorithm is best across all configurations. Fixed-representation methods are especially competitive when little or no memory is available, while fine-tuning methods benefit from bias-reduction components.
- Larger memories and fewer incremental states produce easier configurations, whereas 0.5% memory with 50 states is most difficult.With T = 10, reducing memory from 2% to 0.5% lowers BiC accuracy by 5.8 points on ILSVRC, versus 1.6 on VGGFACE2 and 1.3 on LANDMARKS.
- None of the evaluated methods is best in every configuration; FTth has the strongest aggregate result at GIL = −3.62 points versus Full.ScaIL, BiC, and LUCIRCNN follow with GIL values of −3.7, −4.03, and −4.13, respectively.
- Fine-tuning-based methods benefit from bias-reduction mechanisms: FTth performs best, followed closely by ScaIL, while FTBAL and IL2M are more sensitive to memory reduction and state growth.The lower performance of FTBAL and IL2M is associated with the smallest memory and largest number of states.
- Fixed-representation methods generally underperform fine-tuning approaches with larger memories but become competitive at 0.5% memory because their performance depends less on past exemplars.DeeSIL is globally better than iCaRL and performs best when no memory is allowed.
- Herding improves performance for most algorithms compared with random exemplar selection, although the effect varies across methods.LUCIRCNN, ScaIL, and BiC obtain the best results with random selection at very similar GIL performance; vanilla FT is more affected by random selection.
- Without memory, FTinit is the best fine-tuning approach, while LUCIRCNN is the only method reported to outperform it through a more sophisticated distillation and separation scheme.With memory, vanilla FT is globally better than iCaRL, challenging the necessity of distillation in that setting; without memory, distillation is useful.
8. Conclusion and perspectives
The comparison shows that class-incremental learning methods have configuration-dependent strengths rather than a universal winner. Memory availability, dataset scale, and task assumptions shape which approaches are competitive, while several evaluation and methodological questions remain open.
- Conclusion: No algorithm is best across all configurations, and the strongest choice depends on computation, storage, and data-stream characteristics.With memory, imbalanced-learning approaches perform best globally, while distillation becomes competitive with sufficient memory and few incremental states.
- Conclusion: When memory is unavailable, fixed-representation methods are more competitive than fine-tuning methods and are simpler and faster to deploy.They are especially advantageous on large datasets, where distillation-based methods fail to scale.
- Perspectives: The conclusions are bounded by evaluated scenarios in which the task does not change over time and by assumptions about streamed data and incremental states.Future evaluations should relax readily annotated data, fixed class counts per state, and controlled balanced datasets.
- Conclusion: The study evaluates four visual-task datasets and releases detailed implementations to support reproducibility.The evaluation setting is intended to be reusable and extensible for robust testing of class-incremental algorithms.
- Perspectives: Class-incremental learning remains open, including the need for more sophisticated bias reduction and a deeper explanation of distillation failures on large-scale datasets.The authors also point to inter-class separation and relational knowledge distillation as promising directions.
- Perspectives: Herding-based exemplar selection outperforms random selection for all tested methods.The authors identify optimal exemplar distributions for replay as a direction for further study.