Source-linked AI summary
Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need
Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, Ziwei Liu
TL;DR
CIL with PTMs must balance adaptation to emerging classes against retention of transferable pretrained knowledge, while existing benchmarks can overlap with pretraining data. The paper introduces APER, which combines adapted and frozen PTM embeddings for prototype-based classification. SimpleCIL already outperforms current state-of-the-art by 5% without downstream tuning, and APER is validated across new benchmarks; its adaptation scope remains challenging for extensive domain-incremental gaps.
Problem
CIL must incorporate new classes without forgetting old ones, but PTM generalizability and adaptation to downstream distribution gaps need to be unified.
Method
APER adapts a PTM with parameter-efficient tuning, concatenates adapted and pretrained embeddings, and constructs classifiers from their average embeddings.
Results
5%: SimpleCIL outperforms current state-of-the-art without downstream tuning, while extensive experiments demonstrate APER’s effectiveness across new PTM-based CIL benchmarks.
Takeaways & Limitations
PTM-based CIL benefits from jointly cultivating generalizability for knowledge transfer and adaptivity for downstream task features.
Takeaways & Limitations
APER may face challenges under extensive domain gaps in domain-incremental learning because it adapts only during the first incremental stage.
Abstract
from arXiv · showhide
Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Traditional CIL models are trained from scratch to continually acquire knowledge as data evolves. Recently, pre-training has achieved substantial progress, making vast pre-trained models (PTMs) accessible for CIL. Contrary to traditional methods, PTMs possess generalizable embeddings, which can be easily transferred for CIL. In this work, we revisit CIL with PTMs and argue that the core factors in CIL are adaptivity for model updating and generalizability for knowledge transferring. 1) We first reveal that frozen PTM can already provide generalizable embeddings for CIL. Surprisingly, a simple baseline (SimpleCIL) which continually sets the classifiers of PTM to prototype features can beat state-of-the-art even without training on the downstream task. 2) Due to the distribution gap between pre-trained and downstream datasets, PTM can be further cultivated with adaptivity via model adaptation. We propose AdaPt and mERge (APER), which aggregates the embeddings of PTM and adapted models for classifier construction. APER is a general framework that can be orthogonally combined with any parameter-efficient tuning method, which holds the advantages of PTM's generalizability and adapted model's adaptivity. 3) Additionally, considering previous ImageNet-based benchmarks are unsuitable in the era of PTM due to data overlapping, we propose four new benchmarks for assessment, namely ImageNet-A, ObjectNet, OmniBenchmark, and VTAB. Extensive experiments validate the effectiveness of APER with a unified and concise framework. Code is available at https://github.com/zhoudw-zdw/RevisitingCIL
1 Introduction
CIL must learn emerging classes while preserving prior knowledge, and PTMs offer strong transferable representations for this setting. The paper shows that frozen PTM features already perform strongly, while APER combines PTM generalizability with adaptation to downstream data.
- CIL addresses streaming data with new classes while seeking to prevent catastrophic forgetting of previously learned knowledge.
- PTMs provide strong generalizability through pretraining on massive datasets, contrasting with CIL models trained from random initialization.
- SimpleCIL freezes the pretrained embedding function and uses each class’s average training embedding as its classifier weight.
- 5%: SimpleCIL outperforms current state-of-the-art methods without tuning on downstream tasks, demonstrating transferable pretrained features.
- Sequential finetuning can address the pretraining–downstream distribution gap, but freezing is not universally sufficient for out-of-distribution or specialized tasks.
- APER adapts the PTM with parameter-efficient tuning, concatenates adapted and pretrained embeddings, and uses their average embeddings for classifier construction.
- The paper introduces new PTM-based CIL benchmarks with larger domain gaps from ImageNet because traditional ImageNet-based benchmarks overlap with pretraining data.
2 Related Work
Related work covers conventional CIL strategies, PTM-based continual learning, and parameter-efficient tuning. These approaches motivate combining pretrained representations with task-specific adaptation.
- Conventional CIL methods include exemplar replay, prototype calibration, normalization or feature rectification, and network expansion.
- PTM-based CIL sequentially adjusts pretrained models for new classes, with L2P using instance-specific visual prompt retrieval and DualPrompt adding general and expert prompts.
- Other PTM-based approaches extend prompt learning across image and text modalities, while frozen PTMs have also been studied for novel class discovery.
- Parameter-efficient tuning adapts pretrained models by updating only a small number of additional or modified parameters, reducing tuning cost relative to full finetuning.
3 From Old Classes to New Classes
The paper frames PTM-based CIL as a balance between adaptivity for new classes and generalizability for retaining transferable knowledge. SimpleCIL exposes the strength of frozen features, while adaptation improves task-specific representation.
- 3.1 Class-Incremental Learning: CIL receives a sequence of disjoint class tasks and evaluates a unified model over all classes seen so far.
- 3.1 Class-Incremental Learning: In the exemplar-free setting, historical data are unavailable for rehearsal, so the model must learn new classes while preserving former knowledge.
- 3.1 Class-Incremental Learning: A CIL model must balance discriminability across all seen classes with learning new classes and remembering old ones.
- 3.2 Adaptivity and Generalizability in Class-Incremental Learning: Adaptivity is obtained by updating the incremental model with cross-entropy loss and forgetting-resistance regularization.
- 3.2 Adaptivity and Generalizability in Class-Incremental Learning: SimpleCIL freezes the PTM and sets each classifier weight to its class prototype, transferring pretrained generalizability without model training.
- 3.2 Adaptivity and Generalizability in Class-Incremental Learning: Finetuning performs better on new classes, whereas frozen PTM features perform better on old classes, exposing a trade-off between adaptivity and generalizability.
- 3.2 Adaptivity and Generalizability in Class-Incremental Learning: Adaptivity bridges the pretraining–incremental-learning domain gap, while generalizability transfers knowledge from pretraining to incremental learning.
4 Aper: AdaPt and mERge PTMs for CIL
APER unifies adaptivity and generalizability by adapting a PTM on the first incremental stage, merging adapted and pre-trained embeddings, then freezing the merged representation for later prototype-based classification.
- Training Procedure: The adapted and pre-trained embedding functions are concatenated, preserving generalizable features alongside task-specific representations.The merged embedding function is frozen throughout subsequent incremental stages.
- Training Procedure: APER adapts the PTM with incremental data to bridge the distribution gap and capture task-specific features.The adapting algorithm F optimizes selected parameters using a dataset and produces an adapted model with domain-specific knowledge.
- Training Procedure: Model adaptation is restricted to the first incremental stage because continual tuning can cause catastrophic forgetting.After first-stage adaptation, the embedding functions are frozen and classifiers are replaced with prototypes as new classes arrive.
- Training Procedure: APER uses class prototypes from the merged embeddings as classifier weights with a cosine classifier for incremental classification.Prototype features represent common patterns of each class, while the cosine classifier compares normalized instance embeddings with normalized classifier weights.
- Adapting the PTM: APER can use fully finetuned or parameter-efficient adaptation methods, including VPT, Scale & Shift, Adapter, and batch-normalization tuning.Parameter-efficient methods reduce tuning costs for large PTMs and support both ViT and CNN backbones.
- Related Concepts: By aggregating adapted and pre-trained models, APER combines task-specific adaptivity with pre-trained generalizability in one frozen representation.The paper distinguishes these characteristics from the traditional stability-plasticity dilemma associated with training from scratch.
5 Experiments
The experiments compare APER with state-of-the-art methods and introduce four benchmarks with large domain gaps from ImageNet to evaluate PTM-based CIL more appropriately.
- Experiments: The experiments compare APER with state-of-the-art methods on benchmark datasets and include ablations and visualizations.The study also examines different pre-trained models in class-incremental learning.
- Experiments: Four new benchmarks are advocated because overlap between pre-trained datasets and traditional CIL benchmarks can make PTM-based evaluation unsuitable.The proposed benchmarks are designed to provide larger domain gaps from ImageNet.
5.1 Implementation Details
Implementation evaluates PTM-based CIL across established and newly proposed datasets, ViT and CNN backbones, and average and last-stage performance metrics.
- Datasets: Evaluation uses CIFAR100, CUB200, ImageNet-R, ImageNet-A, ObjectNet, OmniBenchmark, and VTAB.The four newly advocated datasets have large domain gaps with ImageNet; ImageNet-A and ObjectNet contain challenging samples for ImageNet-pre-trained models.
- Backbones: Figure 4 reports incremental performance for ViT-B/16-IN1K and ResNet18, with APER consistently improving both ViT and CNN backbones.L2P and DualPrompt are omitted from ResNet results because they cannot be deployed with that backbone.
- Evaluation Protocol: Accuracy is measured after each incremental stage using last-stage performance A_B and average performance across stages.The average metric summarizes performance along the incremental stages.
- Training Details: The implementation trains adaptation modules for 20 epochs with batch size 48, SGD with momentum, cosine-annealed learning rate, prompt length 5, and Adapter projection dimension 16.These settings are reported for the adaptation experiments.
5.2 Benchmark Comparison
Across benchmark settings, SimpleCIL already outperforms a state-of-the-art method, while APER further improves performance across datasets and pretrained architectures. New benchmarks retain improvement space because they have larger domain gaps from ImageNet.
- The evaluation compares methods using pretrained ViT-B/16-IN21K and includes settings with large and small base classes.Table 2 compares classical CIL methods with ViT-B/16-IN1K without exemplars.
- SimpleCIL outperforms DualPrompt by 20% on CUB and 8% on ImageNet-A in terms of AB.
- APER consistently outperforms SimpleCIL on seven benchmark datasets.
- APER consistently boosts incremental performance for pretrained ViTs and CNNs, while simple BN tuning outperforms full or partial ResNet fine-tuning.The CNN evaluation uses an ImageNet1K-pretrained ResNet18.
- Typical benchmarks approach saturation because of their small domain gap with ImageNet, whereas the new benchmarks retain room for improvement under larger domain gaps.
5.3 Ablation Study
The ablations examine feature dimensionality, component contributions, parameter scale, pretrained-model choice, and adaptation stages. Results support combining pretrained generalizability with adaptation, while limiting tuning to the first stage.
- Downscale features: APER retains competitive performance after PCA projection to 30 dimensions and after random sampling of 200 concatenated features.The aggregated representation is 1536-dimensional versus 768 dimensions for a vanilla PTM.
- Sub-modules: SimpleCIL-Adapted outperforms SimpleCIL-PTM, but the adapted model suffers greater degradation than vanilla SimpleCIL because adaptation overwrites high-level features.This contrasts the adapted model’s task-specific representation with the PTM’s generalizability.
- Sub-modules: APER with finetuning outperforms each individual sub-module by combining adaptivity and generalizability.
- Parameter scale: APER matches or exceeds compared performance with the same or fewer total parameters, while many adaptation parameters are shared with pretrained weights.L2P and DualPrompt use an additional pretrained ViT as a prompt retriever, giving them a similar total scale.
- Influence of adapting stages: Tuning only the first incremental stage achieves the best performance; multi-stage tuning harms generalizability and creates incompatible features across classes.T=0 corresponds to SimpleCIL, while later stages use prototypes after the encoding functions are frozen.
- Different PTMs: The best APER variation consistently improves SimpleCIL across the evaluated pretrained models, with ViTs generally outperforming ResNets under SimpleCIL.The evaluated models include pretrained ResNets, ViTs, DINO-v2, SAM, MAE, and CLIP.
5.4 Visualization of Incremental Sessions
Visualizations examine decision boundaries and image regions for pretrained models and APER. They indicate that pretrained features separate classes competitively, while APER emphasizes task-specific regions.
- Decision boundaries: The decision-boundary visualization compares old-class dots, new-class triangles, class-prototype squares, and shaded boundaries across two incremental tasks.
- Grad-CAM: Grad-CAM visualizations compare the image regions emphasized by the PTM and APER on OmniBenchmark using pretrained ResNet18.Warm colors indicate important regions for prediction.
- Cross-domain context: Table 3 reports domain-incremental and cross-domain CIL results using pretrained ViT-B/16-IN1K.
- Grad-CAM: APER concentrates more on task-specific features than vanilla PTMs, supporting the role of adaptivity in pretrained-model CIL.
5.5 Experiments with Multiple Domains
The paper extends evaluation beyond same-domain CIL to domain-incremental and cross-domain settings. APER remains competitive with prompt-based methods despite not being specifically designed for multiple stages.
- Experimental settings: The multiple-domain evaluation includes Office-Home domain-incremental learning and DomainNet cross-domain class-incremental learning.Office-Home organizes each task around a new domain, while DomainNet is split into five tasks.
- Experimental settings: All compared methods use the same pretrained ViT-B/16-IN1K in the multiple-domain experiments.
- Results: APER remains competitive against L2P and DualPrompt on cross-domain tasks despite not being specially designed for multiple stages.
6 Conclusion
The paper revisits PTM-based CIL around generalizability and adaptivity, proposing APER and four new benchmarks. It reports state-of-the-art performance while identifying limitations in exemplar use and domain-incremental settings.
- A frozen PTM can provide generalizable embeddings, enabling prototype-based classifiers to outperform current state-of-the-art methods.
- APER combines PTM generalizability with adapted-model adaptivity and can be combined orthogonally with any parameter-efficient tuning method.
- Four new benchmarks address the unsuitability of traditional ImageNet-based evaluation caused by data overlapping.
- Extensive experiments verify APER’s state-of-the-art performance in PTM-based CIL.
- APER cannot fully use exemplars because the adapted model should fully reflect downstream features.
- Because APER is adapted only during the first incremental stage, extensive domain gaps during continual learning may challenge it.