Source-linked AI summary
Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning
Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, De-Chuan Zhan
TL;DR
EASE addresses catastrophic forgetting in PTM-based class-incremental learning, where adapting to new classes can conflict with previously learned tasks. It uses task-specific adapters and semantic prototype synthesis, achieving the best performance across seven benchmarks without old-class exemplars.
Problem
Class-incremental learning must absorb new classes without overwriting previously acquired knowledge, while prompt-based PTM adaptation can create conflicts between old and new tasks.
Method
EASE adds a lightweight task-specific adapter for each incoming task, ensembles features across expandable subspaces, and synthesizes former-class prototypes using semantic similarities without exemplars.
Results
EASE achieves the best performance across seven benchmark datasets, outperforming current state-of-the-art methods and exceeding the runner-up by 4%∼7.5% on several benchmarks.
Takeaways & Limitations
EASE provides exemplar-free incremental learning by aggregating historical features and synthesizing former-class prototypes in later subspaces.
Takeaways & Limitations
The setting assumes no access to exemplars from former classes during incremental training.
Abstract
from arXiv · showhide
Class-Incremental Learning (CIL) requires a learning system to continually learn new classes without forgetting. Despite the strong performance of Pre-Trained Models (PTMs) in CIL, a critical issue persists: learning new classes often results in the overwriting of old ones. Excessive modification of the network causes forgetting, while minimal adjustments lead to an inadequate fit for new classes. As a result, it is desired to figure out a way of efficient model updating without harming former knowledge. In this paper, we propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. To enable model updating without conflict, we train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. These adapters span a high-dimensional feature space, enabling joint decision-making across multiple subspaces. As data evolves, the expanding subspaces render the old class classifiers incompatible with new-stage spaces. Correspondingly, we design a semantic-guided prototype complement strategy that synthesizes old classes' new features without using any old class instance. Extensive experiments on seven benchmark datasets verify EASE's state-of-the-art performance. Code is available at: https://github.com/sun-hailong/CVPR24-Ease
1. Introduction
PTM-based CIL preserves generalizable features but still faces conflicts between adapting to new tasks and retaining prior knowledge. EASE addresses this with lightweight task-specific subspaces and semantic prototype synthesis without exemplars.
- CIL must absorb new classes while preventing catastrophic forgetting of previously acquired knowledge.
- Frozen PTMs preserve generalizability, but optimizing prompts for new tasks can rewrite former prompts and create cross-task conflict.
- Expandable networks avoid feature conflict through task-specific backbones, but require substantial storage and exemplars for unified classifier learning.
- EASE synthesizes former-class classifiers from class-wise similarities and reweights predictions across subspaces without using exemplars.
- EASE uses lightweight adapters to create task-specific subspaces that can be concatenated for joint decision-making across tasks.The PTM remains frozen while adapters encode task-specific information.
2. Related Work
Related CIL methods address forgetting through rehearsal, distillation, regularization, rectification, or model expansion. Model expansion reduces cross-task overwriting but is costly for PTMs and typically depends on exemplars.
- Rehearsal methods replay former-class exemplars, while distillation methods preserve former characteristics through mapped logits or features.
- Parameter regularization constrains important-parameter drift, and model rectification corrects incremental models’ inductive bias for more balanced prediction.
- Expandable networks retain previous backbones, add new ones for incoming tasks, concatenate their features, and train a classifier using extra exemplars.
- Large PTM backbones make expansion memory-intensive, while exemplar dependence limits deployment in exemplar-free PTM-based CIL.
3. Preliminaries
The paper formulates exemplar-free CIL as continual unified classification over disjoint task classes using a pretrained feature encoder and linear classifier. It contrasts prompt learning and backbone expansion as competing adaptation strategies.
- CIL continually adds disjoint class sets and seeks one classifier over all classes observed so far.
- In the exemplar-free setting, each stage accesses only current-task data and stores no former-class instances.
- The PTM is decomposed into a feature embedding and linear classifier, with predictions formed by applying classifier weights to embedded inputs.
- Prompt learning freezes pretrained weights and trains prompts, but new-task prompt optimization can conflict with old prompts and cause forgetting.
- Expandable backbones preserve old features by adding new task-specific backbones, but their larger scale and exemplar-based classifier calibration are unsuitable for the current setting.
4. EASE: Expandable Subspace Ensemble
EASE expands PTM-based CIL with lightweight, task-specific adapters and completes incompatible old-class prototypes through semantic mapping across subspaces. The resulting ensemble concatenates subspace features and reweights matching-task contributions for prediction.
- Subspace Expansion with Adapters: EASE creates lightweight task-specific subspaces by training a new adapter for each incremental task while retaining the shared PTM.Each adapter encodes task information without fully fine-tuning or storing a separate backbone.
- Subspace Expansion with Adapters: Adapters produce concatenated features across all learned subspaces, enabling holistic decisions while requiring far fewer parameters than fully fine-tuned backbones.The final representation combines the embeddings generated by all task adapters.
- Prototype Classifier: Prototype classifiers represent each class by concatenating its prototypes across adapter subspaces and comparing them with the concatenated embedding.Prediction uses similarity between the input representation and the corresponding class prototype.
- Semantic Guided Prototype Complement: Expanding subspaces make old classifiers dimensionally incompatible because old-class prototypes cannot be recomputed without former exemplars.After a new adapter is added, old-class prototypes are missing in the latest subspace.
- Semantic Guided Prototype Complement: EASE reconstructs old-class prototypes in each new subspace by transferring class-wise similarities measured in a co-occurrence subspace.The method normalizes old-to-new class similarities and uses them to synthesize prototypes from new-class prototypes.
- Subspace Ensemble via Subspace Reweight: The complete pipeline trains one adapter per task, extracts current-task prototypes, synthesizes former-class prototypes, and reweights logits for inference.The matching subspace receives greater weight during prediction through the trade-off parameter α.
5. Experiments
Experiments across seven benchmark datasets evaluate EASE against state-of-the-art methods, alternative data splits, exemplar-based CIL, component variants, and robustness settings. EASE achieves strong benchmark performance without exemplars, while ablations and visualizations examine its components and task-specific subspaces.
- Benchmark comparison: EASE achieves the best performance across all seven benchmarks with ViT-B/16-IN21K and outperforms CODA-Prompt and ADAM.Table 1 compares average and last performance without using exemplars.
- Benchmark comparison: EASE remains competitive with large base classes and exemplar-based methods while using no exemplars.Traditional CIL comparisons use 20 exemplars per class for competing methods.
- Efficiency: EASE matches prompt-based methods in parameter scale while achieving the best performance among competitors on ImageNet-R B100 Inc50.The parameter-performance comparison includes L2P and DualPrompt.
- Ablation study: Prototype complement further improves performance by using cross-task semantic information, beyond task-specific adapters alone.The ablation is conducted on ImageNet-R B0 Inc20 and compares Vanilla PTM, task-specific adapters, and prototype complement.
- Analysis: t-SNE visualizations examine whether adapters learn task-specific subspaces for corresponding incremental tasks.Two adapters are visualized for two five-class incremental stages, with dots and triangles denoting the two tasks.
- Analysis: Parameter robustness is evaluated by varying projection dimension r across {8, 16, 32, 64, 128} and trade-off parameter α across {0.01, 0.05, 0.1, 0.3, 0.5}.Average performance is reported for these settings on ImageNet-R B0 Inc20.
6. Conclusion
EASE equips a pre-trained model with lightweight adapters to create diverse subspaces, aggregate historical features, and synthesize former-class prototypes without exemplars. The paper reports extensive experimental validation, while noting adapter storage as a limitation.
- EASE equips a pre-trained model with diverse subspaces through lightweight adapters and aggregates historical features for holistic embeddings without forgetting.
- EASE uses semantic information to synthesize former-class prototypes in later subspaces without exemplars.
- Extensive experiments verify EASE’s effectiveness for pre-trained model-based class-incremental learning.
- Adapters consume 0.3% of the total backbone, but saving them still increases model size.
I. Further Ablations
Further analysis examines EASE’s components and reports additional evaluations of robustness, efficiency, and performance limits.
- Further analysis investigates semantic-guided mapping and adapter-spanned subspaces as components of EASE.
- The analysis also compares random seeds, running time, and upper-bound results.
I.1. Prototype-Prototype Similarity VS. Prototype-Instance Similarity
The ablation compares prototype-prototype and prototype-instance similarity for complementing old-class prototypes in new subspaces. Prototype-prototype similarity performs better, even when prototype-instance similarity uses more instances and resources.
- Prototype-prototype similarity reconstructs old-class prototypes in new subspaces using class-wise relationships between old and new subspaces.The complement task estimates old-class prototypes in the new subspace from prototypes in the old and new spaces.
- Prototype-instance similarity instead measures old-class prototype similarity against selected new-class instances, with k chosen from {1, 5, 20, 50, 100, 200}.The selected instances are based on relative similarity.
- Using more instances improves prototype-instance performance, but prototype-instance similarity remains less effective than prototype-prototype similarity despite higher resource use.Figure 1 compares the two similarity calculation methods.
I.2. Adapter VS. VPT
This ablation compares adapter tuning with visual prompt tuning for constructing EASE’s task-specific subspaces. Adapter-based subspaces perform better and are therefore selected for EASE.
- The experiment combines EASE with adapter tuning and visual prompt tuning while changing only the subspace-building technique.Experiments use CIFAR100 and ImageNet-R with other settings held constant.
- Adapters outperform VPT by 2−3% on the evaluated datasets.
- EASE chooses adapter tuning because it provides stronger pre-trained-model tuning than VPT in these experiments.
I.3. Comparison to Upper bound
EASE is evaluated against an upper bound and across multiple experimental settings. It remains competitive without exemplars while maintaining strong performance and efficiency across comparisons.
- Upper-bound comparison: EASE achieves almost the same results as the exemplar-based upper bound without using any exemplars.The upper bound computes accurate class-center estimates from former-class exemplars, whereas EASE uses semantic information for prototype complement.
- Multiple runs: EASE consistently outperforms other methods across multiple random seeds.Experiments use seeds 1993 through 1997 and report mean and standard variance.
- Efficiency: EASE requires less running time than CODA-Prompt, L2P, and DualPrompt while achieving the best performance.The runtime comparison uses a single NVIDIA 4090 GPU.
- Full results: EASE consistently outperforms other methods by a substantial margin across different datasets with ViT-B/16-IN21K.The comparison includes incremental performance curves under the same pretrained backbone initialization.