Source-linked AI summary
Memory-Efficient Incremental Learning Through Feature Adaptation
Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, Cordelia Schmid
TL;DR
Incremental learning typically assumes all classes and labeled data are available together, while preserving prior images imposes substantial memory costs. This paper preserves low-dimensional feature descriptors and adapts them to updated feature spaces, achieving state-of-the-art accuracy with significantly lower memory requirements.
Problem
Common models require large labeled datasets and assume all possible classes are available simultaneously, while incremental learning must limit memory growth as classes accumulate.
Method
The method preserves previous feature descriptors, learns a mapping between old and new feature spaces, and uses adapted descriptors to train a classifier across seen classes.
Results
The method achieves state-of-the-art classification accuracy with a significantly lower memory footprint than image-preserving strategies.
Takeaways & Limitations
Preserving and adapting feature descriptors makes incremental learning possible without retaining previous class images and can combine with existing methods for higher accuracy.
Takeaways & Limitations
Future-task original images are assumed unavailable; their use appears only in an ablation study.
Abstract
from arXiv · showhide
We introduce an approach for incremental learning that preserves feature descriptors of training images from previously learned classes, instead of the images themselves, unlike most existing work. Keeping the much lower-dimensional feature embeddings of images reduces the memory footprint significantly. We assume that the model is updated incrementally for new classes as new data becomes available sequentially.This requires adapting the previously stored feature vectors to the updated feature space without having access to the corresponding original training images. Feature adaptation is learned with a multi-layer perceptron, which is trained on feature pairs corresponding to the outputs of the original and updated network on a training image. We validate experimentally that such a transformation generalizes well to the features of the previous set of classes, and maps features to a discriminative subspace in the feature space. As a result, the classifier is optimized jointly over new and old classes without requiring old class images. Experimental results show that our method achieves state-of-the-art classification accuracy in incremental learning benchmarks, while having at least an order of magnitude lower memory footprint compared to image-preserving strategies.
1 Introduction
Class-incremental learning receives classes sequentially, requiring models to retain prior knowledge despite catastrophic forgetting. The paper preserves compact feature descriptors, adapts them to updated feature spaces, and reports high accuracy with substantially lower memory.
- Class-incremental learning receives new-class data sequentially while previous-class data is discarded, making retention of earlier knowledge difficult because of catastrophic forgetting.
- A 512-dimensional ResNet-18 feature vector requires approximately 1% of the storage of a 256 × 256 × 3 ImageNet image.
- The method preserves previous feature descriptors rather than images, compressing information from earlier classes into low-dimensional embeddings.
- Feature adaptation maps preserved descriptors from obsolete previous feature spaces into the current feature space as the model learns new classes.
- The adapted features allow a classifier to use representations from all seen classes, while experiments report top ImageNet accuracy and at least an order-of-magnitude lower memory footprint.
2 Related work
Prior incremental-learning methods use non-rehearsal constraints, image exemplars, generated images, or feature statistics to retain earlier knowledge. This paper instead rehearses preserved feature descriptors, adapts them across network updates, and avoids dependence on a pre-trained model.
- Non-rehearsal methods preserve no previous data, instead modifying network capacity or constraining important parameter updates.
- Exemplar rehearsal preserves selected previous images and reuses them when updating the network for new classes.
- Generated-image rehearsal uses generative models to mimic past data, but often targets low-resolution datasets or depends on combining generated and real images.
- Earlier feature-based methods store feature statistics or generate vectors, but depend on a pre-trained network.
- This paper rehearses preserved feature descriptors, adapts them to new feature spaces, supports training from scratch, and uses less memory than image-preserving rehearsal.
3 Background on incremental learning
Class-incremental learning introduces classes sequentially while discarding previous-class data, requiring models to learn new classes without forgetting earlier knowledge. Standard strategies use distillation and preserved exemplars to mitigate catastrophic forgetting.
- Class-incremental learning receives training data sequentially, with previous-class data discarded as new classes become available.
- The model must classify new classes while retaining accurate predictions for classes introduced in earlier tasks.
- Catastrophic forgetting causes neural networks to quickly lose knowledge acquired from previous tasks during updates on current-task data.
- Distillation: Knowledge distillation adds a loss that encourages the updated network to make predictions similar to a frozen previous model.
- Preserving data of the old classes: Exemplar rehearsal stores selected images from earlier classes and combines them with current-task images during later training.
- Preserving data of the old classes: Herding selects a fixed-size subset of exemplars that approximates the class mean, thereby bounding stored-example memory requirements.
4 Memory-efficient incremental learning
The method preserves compact feature descriptors instead of old images, adapts them as the feature extractor changes, and trains a classifier jointly over current and adapted previous-task features. This reduces memory use while maintaining compatible representations for incremental classification.
- The framework preserves compact feature descriptors rather than old images, enabling greater memory efficiency or more stored examples under the same budget.
- Changing feature extractors make preserved descriptors incompatible with current-task features, preventing direct joint classifier training without adaptation.
- Feature adaptation: Feature adaptation learns a mapping between previous and current feature spaces, using paired outputs for the same current-task images.
- Network training: Cosine normalization and feature distillation improve the accuracy of the proposed method and its baselines.
- Feature adaptation: After adaptation, stored features from previous tasks and features from the current task are combined into memory in the current feature space.
- Feature adaptation: The adaptation loss combines cosine similarity with classification loss, encouraging transformed descriptors to match updated features and retain their correct labels.
- Reducing the size of M_t: Herding reduces memory by retaining a fixed number L of feature vectors per class while approximating each class mean.
- Training the feature classifier: The resulting feature classifier uses adapted previous-task descriptors together with current-task descriptors to improve classification across all seen classes.
5 Experiments
Experiments evaluate incremental classification on CIFAR-100, ImageNet-100, and ImageNet-1000 using accuracy, memory footprint, and feature-adaptation quality. Across ImageNet datasets, preserving feature descriptors achieves comparable or better accuracy than image-preserving baselines with substantially lower memory, while parameter studies examine adaptation behavior.
- Experimental setup: Experiments use CIFAR-100, ImageNet-100, and ImageNet-1000 in a class-incremental setting, reporting classification curves and average incremental accuracy.Top-1 accuracy is used for CIFAR-100, while top-5 accuracy is used for ImageNet.
- Memory footprint: Memory requirements are varied through L, the number of preserved feature descriptors per class, and P, the number of preserved images per class.Figure 2 compares memory in MB with average incremental accuracy across datasets and numbers of classes per task.
- Memory footprint: ImageNet feature descriptors require about 2 KB each versus 192 KB for an uncompressed image, making one descriptor about 1% of image storage.The comparison uses d = 512 floating-point features and 256 × 256 × 3 uint8 ImageNet images.
- Comparison to state of the art: Our method achieves the same accuracy with significantly less memory than γ-iCARL on ImageNet, with at least an order of magnitude lower memory requirements in most cases.For ImageNet-1000, accuracy is superior to γ-iCARL when M ≥100.
- Comparison to state of the art: On ImageNet-1000, the method outperforms γ-iCARL in most cases across different numbers of preserved data points, even ignoring feature-versus-image storage savings.The compared data points are features for the proposed method and images for γ-iCARL.
- Comparison to state of the art: The method uses 250 preserved features per class, while Ours-hybrid combines 250 features with 10 images per class; baselines preserve 20 images per class.Table 1 reports total preserved-data memory and average incremental accuracy for ImageNet-100 and ImageNet-1000.
- Parameter analysis: Higher γ increases adaptation quality ω but lowers classification accuracy because it limits feature-extractor changes and produces less discriminative representations.Most other parameter trends correlate better adaptation with better accuracy.
6 Conclusions
The method preserves feature descriptors rather than images for incremental learning and adapts them as the network changes. It achieves state-of-the-art accuracy with significantly lower memory requirements and can complement existing methods.
- Feature descriptors replace preserved images in the proposed incremental-learning method.
- A feature adaptation function updates preserved descriptors as the network learns new classes.
- The method achieves state-of-the-art classification accuracy with a significantly lower memory footprint.
- The approach is orthogonal to existing methods and can be combined with them for higher accuracy under low memory requirements.
A Algorithm
The framework trains an initial model, stores and reduces feature descriptors, then incrementally adapts stored features and retrains a classifier as new tasks arrive.
- Algorithm: The framework begins by training the first task with a cross-entropy objective and storing feature descriptors for its images.
- Algorithm: Herding reduces the number of stored feature descriptors after each task.
- Algorithm: For each incremental task, the model trains on current-task data and stores the resulting new feature descriptors.
- Algorithm: Previously stored features are adapted with φψ before classifier training combines them with features from all seen classes.
- FeatureAdaptation: Feature adaptation learns a transformation by optimizing a feature-adaptation loss over old- and new-extractor descriptors.
B Feature Adaptation Quality
Feature adaptation quality measures similarity between adapted features and their ground-truth representations. On CIFAR-100, adaptation remains strong across tasks even without preserving images, and improves when images are preserved.
- Adaptation quality measures average similarity between adapted features and their ground-truth values.
- The ωt−1 measure evaluates adaptation between consecutive task feature spaces, whereas ω1 tracks accumulated divergence from first-task representations.
- P = 0 achieves ωt−1 greater than 0.9 in all CIFAR-100 tasks with M = 10.
- After 10 tasks, ω1 remains close to 0.8 despite nine subsequent tasks without preserved images.
- Adaptation quality improves as P increases, indicating that preserving images also helps learn feature adaptation.
C Balanced Feature Classifier Training
The method is evaluated with balanced and imbalanced feature-classifier training because fewer stored examples for old classes can bias classifiers toward new classes. Balanced feature storage provides more old-class examples and lower memory than the compared image-preserving method.
- The evaluation compares class-balanced and class-imbalanced feature-classifier training because fewer old-class examples can bias predictions toward new classes.
- In the imbalanced ImageNet-100 setting, old classes have 250 features per class while new classes have approximately 1300 each; balanced training uses 250 per class.
- The method achieves 0.893 accuracy on ImageNet-100 with class-imbalanced training.
- Feature adaptation reduces memory relative to while storing 250 old-class features per class versus 20 images per class for.
- The larger number of stored old-class examples may explain some improvement over previous methods and permits removing examples flexibly.