Source-linked AI summary
iCaRL: Incremental Classifier and Representation Learning
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, Christoph H. Lampert
TL;DR
Class-incremental learning systems must acquire new visual classes as data arrives while preserving existing knowledge. iCaRL jointly learns classifiers and representations with exemplar storage, and experiments show it learns incrementally over long periods where competing methods fail quickly.
Problem
Most artificial object recognition systems require batch training with all classes and their data available at once, motivating class-incremental learning.
Method
iCaRL jointly learns classifiers and feature representations using an exemplar-based classifier, prioritized exemplar selection, and distillation-based representation learning.
Results
iCaRL clearly outperforms competing incremental methods, especially as fewer classes can be processed simultaneously, and learns incrementally over long periods where other methods fail quickly.
Takeaways & Limitations
Exemplar images are an important contributor to iCaRL’s strong classification results in the class-incremental setting.
Takeaways & Limitations
iCaRL’s performance remains lower than systems trained in batch settings with all classes and training examples available simultaneously.
Abstract
from arXiv · showhide
A major open problem on the road to artificial intelligence is the development of incrementally learning systems that learn about more and more concepts over time from a stream of data. In this work, we introduce a new training strategy, iCaRL, that allows learning in such a class-incremental way: only the training data for a small number of classes has to be present at the same time and new classes can be added progressively. iCaRL learns strong classifiers and a data representation simultaneously. This distinguishes it from earlier works that were fundamentally limited to fixed data representations and therefore incompatible with deep learning architectures. We show by experiments on CIFAR-100 and ImageNet ILSVRC 2012 data that iCaRL can learn many classes incrementally over a long period of time where other strategies quickly fail.
1. Introduction
The introduction defines class-incremental learning as continuously adding classes while retaining competitive performance on previously observed classes under bounded resource growth. It presents iCaRL as a strategy combining exemplar-based classification, herding-based exemplar selection, and representation learning through distillation and rehearsal.
- Motivation: Most artificial object recognition systems rely on batch training with all classes and training data available simultaneously, conflicting with incremental learning requirements.The introduction motivates more flexible strategies for large-scale, dynamic real-world categorization.
- Problem definition: Class-incremental learning requires learning from streams where different classes appear at different times while maintaining a competitive classifier for all classes seen so far.Its resource requirements and memory footprint should remain bounded or grow very slowly with the number of classes.
- Proposed strategy: iCaRL simultaneously learns classifiers and a feature representation for the class-incremental setting.The strategy is designed to fulfill the introduction’s three criteria in combination.
- Method components: iCaRL combines nearest-mean-of-exemplars classification, herding-based prioritized exemplar selection, and representation learning with knowledge distillation and prototype rehearsal.The introduction identifies these as the method’s three main components.
- Empirical scope: Experiments on CIFAR and ImageNet show that iCaRL can learn class-incrementally over long periods where other methods quickly fail.The introduction states that experimental details are reported in Section 4.
2. Method
iCaRL combines incremental representation learning with exemplar-based classification, updating its network and bounded exemplar memory as new classes arrive. Its normalized feature extractor, dynamically balanced exemplars, and nearest-mean classifier support class-incremental learning without retaining all prior training data.
- Training: When new classes arrive, iCaRL updates network parameters and exemplars using batches of classes, with training loss combining classification and distillation terms.The update routine incorporates current training data and existing exemplar sets, enabling learning about newly observed classes.
- Architecture: iCaRL uses a CNN as a trainable feature extractor followed by sigmoid outputs for observed classes, with L2-normalized features and re-normalized feature operations.The network parameters comprise feature-extraction parameters and one weight vector per observed class.
- Classification: The network learns the representation, while classification uses nearest means of dynamically selected exemplar features rather than the network outputs.Each class prototype is the average feature vector of its exemplars, and prediction selects the most similar prototype.
- Exemplar management: K exemplars are distributed equally across t observed classes as m = K/t exemplars per class, up to rounding, so the memory budget is fully used without being exceeded.Existing exemplar sets are reduced as new classes appear, while new sets are constructed from newly available class data.
- Exemplar management: Exemplar selection occurs once when a class is first observed, whereas later reductions require no access to earlier training data.The reduction procedure uses a data-independent fixed-order removal strategy, while construction preserves the desired approximation properties.
3. Related work
Related work spans fixed-representation classifiers and methods that jointly learn representations, with iCaRL drawing prototype-based classification from NCM and rehearsal-based strategies for mitigating forgetting. Unlike approaches that grow networks, iCaRL combines rehearsal with distillation while maintaining a small memory footprint.
- Scope: Related work is organized around fixed data representations versus methods that incrementally learn representations.The section covers both classical connectionist and recent deep-learning approaches.
- Fixed data representation: NCM incrementally represents each class by an average feature-vector prototype, avoiding storage of all previously observed examples.Classification assigns each example the label of the most similar class prototype.
- Fixed data representation: NCM cannot easily support jointly learning nonlinear representations because class means then cease to be incrementally computable.iCaRL retains prototype-based classification but averages only a selected subset of examples to limit memory.
- Representation learning: Early connectionist work identified freezing and growing network weights or rehearsal as two strategies for addressing catastrophic forgetting.Recent freeze/grow methods require increasing network resources over time, violating class-incremental resource constraints.
- Representation learning: iCaRL adopts rehearsal by training on current-class data and earlier exemplars, and adds distillation to limit deterioration of network information.Earlier exemplars are retained for prototype-based classification and therefore support representation updates as well.
4. Experiments
The experiments evaluate class-incremental learning under a proposed benchmark protocol on CIFAR-100 and ILSVRC, showing that iCaRL outperforms alternative methods and that its components jointly support performance and reduced bias. Ablations further indicate benefits from exemplar-based classification, representation learning, distillation, and larger memory budgets.
- Benchmark protocol: The proposed protocol evaluates classifiers after each class batch on only the classes observed so far, using fixed random class orders.The iCIFAR-100 benchmark trains all 100 classes in batches of 2, 5, 10, 20, or 50 and averages results over ten class orders.
- Classification accuracy: iCaRL clearly outperforms LwF.MC, fixed-representation training, and finetuning, especially when fewer classes are processed per batch.LwF.MC is generally second best; fixed representation is better only on iILSVRC-full, while finetuning performs worst.
- Classification accuracy: iCaRL’s confusion matrix is homogeneous across classes, indicating no intrinsic bias toward early or late classes and no catastrophic forgetting.LwF.MC favors recent classes, finetuning predicts only the last batch, and fixed representation favors the first batch.
- Component analysis: All iCaRL components contribute substantially to performance, while the mean-of-exemplars classifier is particularly advantageous for smaller batch sizes.Hybrid setups combining or removing iCaRL components mostly fall between iCaRL and LwF.MC in average incremental-training accuracy.
- Component analysis: iCaRL and nearest-class-mean classification show only minor differences, confirming that iCaRL reliably identifies representative exemplars.NCM recomputes class means after representation updates but requires storing all training data, so it is not class-incremental.
- Component analysis: All methods benefit from larger memory budgets, and with at least 1000 prototypes iCaRL performs similarly to the NCM classifier.The comparison uses the same representation for iCaRL, hybrid1, and NCM while varying memory budgets.
5. Conclusion
The conclusion presents iCaRL as a class-incremental strategy that jointly learns classifiers and representations through exemplar-based classification, exemplar selection, and distillation. It emphasizes the strong contribution of exemplars while noting that class-incremental learning remains below batch-training performance and requires further study.
- Main components: iCaRL jointly learns classifiers and feature representations using nearest-mean-of-exemplars classification, herding-based exemplar selection, and exemplar-based distillation.The classifier stores only a small number of exemplars per class and is designed to remain robust to representation changes.
- Role of exemplars: Exemplar images are identified as the main reason for iCaRL’s strong classification results in the class-incremental setting.The conclusion highlights the pronounced benefit of combining stored exemplars with network parameters.
- Component analysis: Switching off individual iCaRL components produces results mostly between iCaRL and LwF.MC, indicating that all new components contribute to performance.The modifications evaluated are hybrid1, hybrid2, and hybrid3.
- Limitations and future work: iCaRL’s performance remains lower than batch-trained systems, and future work targets the remaining gap and settings where raw training data cannot be stored.The conclusion describes class-incremental classification as far from solved and mentions privacy-motivated constraints as a related scenario.