PaperlayerView exact revision on arXiv

Source-linked AI summary

Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning

Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, De-Chuan Zhan

arXiv:2403.12030v1cs.CVcs.LG

TL;DR

EASE addresses catastrophic forgetting in PTM-based class-incremental learning, where adapting to new classes can conflict with previously learned tasks. It uses task-specific adapters and semantic prototype synthesis, achieving the best performance across seven benchmarks without old-class exemplars.

  • Problem

    Class-incremental learning must absorb new classes without overwriting previously acquired knowledge, while prompt-based PTM adaptation can create conflicts between old and new tasks.

    Source: §1 · p. 1

  • Method

    EASE adds a lightweight task-specific adapter for each incoming task, ensembles features across expandable subspaces, and synthesizes former-class prototypes using semantic similarities without exemplars.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    EASE achieves the best performance across seven benchmark datasets, outperforming current state-of-the-art methods and exceeding the runner-up by 4%∼7.5% on several benchmarks.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    EASE provides exemplar-free incremental learning by aggregating historical features and synthesizing former-class prototypes in later subspaces.

    Source: §5.2 · p. 7, §6 · p. 8

    The setting assumes no access to exemplars from former classes during incremental training.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Class-Incremental Learning (CIL) requires a learning system to continually learn new classes without forgetting. Despite the strong performance of Pre-Trained Models (PTMs) in CIL, a critical issue persists: learning new classes often results in the overwriting of old ones. Excessive modification of the network causes forgetting, while minimal adjustments lead to an inadequate fit for new classes. As a result, it is desired to figure out a way of efficient model updating without harming former knowledge. In this paper, we propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. To enable model updating without conflict, we train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. These adapters span a high-dimensional feature space, enabling joint decision-making across multiple subspaces. As data evolves, the expanding subspaces render the old class classifiers incompatible with new-stage spaces. Correspondingly, we design a semantic-guided prototype complement strategy that synthesizes old classes' new features without using any old class instance. Extensive experiments on seven benchmark datasets verify EASE's state-of-the-art performance. Code is available at: https://github.com/sun-hailong/CVPR24-Ease

1. Introduction

PTM-based CIL preserves generalizable features but still faces conflicts between adapting to new tasks and retaining prior knowledge. EASE addresses this with lightweight task-specific subspaces and semantic prototype synthesis without exemplars.

  • CIL must absorb new classes while preventing catastrophic forgetting of previously acquired knowledge.Source: §1 · p. 1
  • Frozen PTMs preserve generalizability, but optimizing prompts for new tasks can rewrite former prompts and create cross-task conflict.Source: §1 · p. 1
  • Expandable networks avoid feature conflict through task-specific backbones, but require substantial storage and exemplars for unified classifier learning.Source: §1 · pp. 1–2
  • EASE synthesizes former-class classifiers from class-wise similarities and reweights predictions across subspaces without using exemplars.Source: §1 · p. 2
  • EASE uses lightweight adapters to create task-specific subspaces that can be concatenated for joint decision-making across tasks.The PTM remains frozen while adapters encode task-specific information.Source: §1 · p. 2

2. Related Work

Related CIL methods address forgetting through rehearsal, distillation, regularization, rectification, or model expansion. Model expansion reduces cross-task overwriting but is costly for PTMs and typically depends on exemplars.

  • Rehearsal methods replay former-class exemplars, while distillation methods preserve former characteristics through mapped logits or features.Source: §2 · p. 2
  • Parameter regularization constrains important-parameter drift, and model rectification corrects incremental models’ inductive bias for more balanced prediction.Source: §2 · p. 2
  • Expandable networks retain previous backbones, add new ones for incoming tasks, concatenate their features, and train a classifier using extra exemplars.Source: §2 · p. 2
  • Large PTM backbones make expansion memory-intensive, while exemplar dependence limits deployment in exemplar-free PTM-based CIL.Source: §2 · p. 2

3. Preliminaries

The paper formulates exemplar-free CIL as continual unified classification over disjoint task classes using a pretrained feature encoder and linear classifier. It contrasts prompt learning and backbone expansion as competing adaptation strategies.

  • CIL continually adds disjoint class sets and seeks one classifier over all classes observed so far.Source: §3.1 · pp. 2–3
  • In the exemplar-free setting, each stage accesses only current-task data and stores no former-class instances.Source: §3.1 · pp. 2–3
  • The PTM is decomposed into a feature embedding and linear classifier, with predictions formed by applying classifier weights to embedded inputs.Source: §3.1 · p. 3
  • Prompt learning freezes pretrained weights and trains prompts, but new-task prompt optimization can conflict with old prompts and cause forgetting.Source: §3.2 · p. 3
  • Expandable backbones preserve old features by adding new task-specific backbones, but their larger scale and exemplar-based classifier calibration are unsuitable for the current setting.Source: §3.2 · p. 3

4. EASE: Expandable Subspace Ensemble

EASE expands PTM-based CIL with lightweight, task-specific adapters and completes incompatible old-class prototypes through semantic mapping across subspaces. The resulting ensemble concatenates subspace features and reweights matching-task contributions for prediction.

Subspace Expansion with Adapters

  • EASE creates lightweight task-specific subspaces by training a new adapter for each incremental task while retaining the shared PTM.Each adapter encodes task information without fully fine-tuning or storing a separate backbone.Source: §4.1 · p. 4
  • Adapters produce concatenated features across all learned subspaces, enabling holistic decisions while requiring far fewer parameters than fully fine-tuned backbones.The final representation combines the embeddings generated by all task adapters.Source: §4.1 · p. 4

Prototype Classifier

  • Prototype classifiers represent each class by concatenating its prototypes across adapter subspaces and comparing them with the concatenated embedding.Prediction uses similarity between the input representation and the corresponding class prototype.Source: §4.1 · p. 4

Semantic Guided Prototype Complement

  • Expanding subspaces make old classifiers dimensionally incompatible because old-class prototypes cannot be recomputed without former exemplars.After a new adapter is added, old-class prototypes are missing in the latest subspace.Source: §4.2 · pp. 4–5
  • EASE reconstructs old-class prototypes in each new subspace by transferring class-wise similarities measured in a co-occurrence subspace.The method normalizes old-to-new class similarities and uses them to synthesize prototypes from new-class prototypes.Source: §4.2 · p. 5

Subspace Ensemble via Subspace Reweight

  • The complete pipeline trains one adapter per task, extracts current-task prototypes, synthesizes former-class prototypes, and reweights logits for inference.The matching subspace receives greater weight during prediction through the trade-off parameter α.Source: §4.2 · p. 5, §4.3 · p. 5

5. Experiments

Experiments across seven benchmark datasets evaluate EASE against state-of-the-art methods, alternative data splits, exemplar-based CIL, component variants, and robustness settings. EASE achieves strong benchmark performance without exemplars, while ablations and visualizations examine its components and task-specific subspaces.

Benchmark comparison

  • EASE achieves the best performance across all seven benchmarks with ViT-B/16-IN21K and outperforms CODA-Prompt and ADAM.Table 1 compares average and last performance without using exemplars.Source: Table 1 · p. 6, §5.2 · p. 7
  • EASE remains competitive with large base classes and exemplar-based methods while using no exemplars.Traditional CIL comparisons use 20 exemplars per class for competing methods.Source: §5.2 · p. 7, Figure 4 · p. 7, Table 2 · p. 7

Efficiency

  • EASE matches prompt-based methods in parameter scale while achieving the best performance among competitors on ImageNet-R B100 Inc50.The parameter-performance comparison includes L2P and DualPrompt.Source: §5.2 · p. 7

Ablation study

  • Prototype complement further improves performance by using cross-task semantic information, beyond task-specific adapters alone.The ablation is conducted on ImageNet-R B0 Inc20 and compares Vanilla PTM, task-specific adapters, and prototype complement.Source: Figure 5 · p. 7, §5.3 · p. 8

Analysis

  • t-SNE visualizations examine whether adapters learn task-specific subspaces for corresponding incremental tasks.Two adapters are visualized for two five-class incremental stages, with dots and triangles denoting the two tasks.Source: §5.4 · p. 8, Figure 6 · p. 8
  • Parameter robustness is evaluated by varying projection dimension r across {8, 16, 32, 64, 128} and trade-off parameter α across {0.01, 0.05, 0.1, 0.3, 0.5}.Average performance is reported for these settings on ImageNet-R B0 Inc20.Source: §5.4 · p. 8, Figure 7 · p. 8

6. Conclusion

EASE equips a pre-trained model with lightweight adapters to create diverse subspaces, aggregate historical features, and synthesize former-class prototypes without exemplars. The paper reports extensive experimental validation, while noting adapter storage as a limitation.

  • EASE equips a pre-trained model with diverse subspaces through lightweight adapters and aggregates historical features for holistic embeddings without forgetting.Source: §6 · p. 8
  • EASE uses semantic information to synthesize former-class prototypes in later subspaces without exemplars.Source: §6 · p. 8
  • Extensive experiments verify EASE’s effectiveness for pre-trained model-based class-incremental learning.Source: §6 · p. 8
  • Adapters consume 0.3% of the total backbone, but saving them still increases model size.Source: §6 · p. 8

I. Further Ablations

Further analysis examines EASE’s components and reports additional evaluations of robustness, efficiency, and performance limits.

  • Further analysis investigates semantic-guided mapping and adapter-spanned subspaces as components of EASE.Source: §I · p. 11
  • The analysis also compares random seeds, running time, and upper-bound results.Source: §I · p. 11

I.1. Prototype-Prototype Similarity VS. Prototype-Instance Similarity

The ablation compares prototype-prototype and prototype-instance similarity for complementing old-class prototypes in new subspaces. Prototype-prototype similarity performs better, even when prototype-instance similarity uses more instances and resources.

  • Prototype-prototype similarity reconstructs old-class prototypes in new subspaces using class-wise relationships between old and new subspaces.The complement task estimates old-class prototypes in the new subspace from prototypes in the old and new spaces.Source: §I.1 · p. 11
  • Prototype-instance similarity instead measures old-class prototype similarity against selected new-class instances, with k chosen from {1, 5, 20, 50, 100, 200}.The selected instances are based on relative similarity.Source: §I.1 · p. 12
  • Using more instances improves prototype-instance performance, but prototype-instance similarity remains less effective than prototype-prototype similarity despite higher resource use.Figure 1 compares the two similarity calculation methods.Source: §I.1 · p. 12, Figure 1 · p. 12

I.2. Adapter VS. VPT

This ablation compares adapter tuning with visual prompt tuning for constructing EASE’s task-specific subspaces. Adapter-based subspaces perform better and are therefore selected for EASE.

  • The experiment combines EASE with adapter tuning and visual prompt tuning while changing only the subspace-building technique.Experiments use CIFAR100 and ImageNet-R with other settings held constant.Source: §I.2 · pp. 12–13
  • Adapters outperform VPT by 2−3% on the evaluated datasets.Source: §I.2 · p. 13
  • EASE chooses adapter tuning because it provides stronger pre-trained-model tuning than VPT in these experiments.Source: §I.2 · p. 13

I.3. Comparison to Upper bound

EASE is evaluated against an upper bound and across multiple experimental settings. It remains competitive without exemplars while maintaining strong performance and efficiency across comparisons.

Upper-bound comparison

  • EASE achieves almost the same results as the exemplar-based upper bound without using any exemplars.The upper bound computes accurate class-center estimates from former-class exemplars, whereas EASE uses semantic information for prototype complement.Source: §I.3 · p. 13

Multiple runs

  • EASE consistently outperforms other methods across multiple random seeds.Experiments use seeds 1993 through 1997 and report mean and standard variance.Source: §I.4 · p. 13

Efficiency

  • EASE requires less running time than CODA-Prompt, L2P, and DualPrompt while achieving the best performance.The runtime comparison uses a single NVIDIA 4090 GPU.Source: §I.5 · p. 13

Full results

  • EASE consistently outperforms other methods by a substantial margin across different datasets with ViT-B/16-IN21K.The comparison includes incremental performance curves under the same pretrained backbone initialization.Source: Figure 5 · p. 14, §III · p. 15

Related papers

Loading 2403.12030v1…