SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model

Gengwei Zhang; Liyuan Wang; Guoliang Kang; Ling Chen; Yunchao Wei

Source-linked AI summary

SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model

Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, Yunchao Wei

arXiv:2303.05118v4Mar 9, 2023cs.CVcs.AIcs.LG

TL;DR

Continual learning on pre-trained models must adapt to incremental tasks while preserving the generalizability of pre-trained representations, but sequential updates suffer progressive overfitting. The paper proposes SLCA, combining a slow representation learner with post-hoc classifier alignment, and reports substantial improvements that bring sequential fine-tuning close to joint-training performance on several benchmarks.

Problem
Pre-trained knowledge must be adaptively exploited for each incremental task while maintaining generalizability, because sequential fine-tuning can progressively overfit pre-trained representations.
Method
SLCA selectively reduces the representation learning rate and aligns classification layers post hoc by modeling class-wise feature distributions.
Results
49.76%, 50.05%, 44.69% and 40.16% improvements are reported on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively.
Takeaways & Limitations
SLCA provides a strong baseline for re-evaluating continual learning on pre-trained computer-vision models and exploring pre-training paradigm and downstream granularity.
Takeaways & Limitations
The study uses a pre-trained ViT backbone and does not examine joint upstream and downstream continual learning, other architectures, or applications such as detection and segmentation.

Abstract

from arXiv · show

The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research. Code has been made available at: https://github.com/GengDavid/SLCA.

1. Introduction

Continual learning on pre-trained models must adapt pre-trained representations to new tasks without sacrificing future generalizability. The paper identifies progressive overfitting as the key challenge and proposes SLCA to address it.

Pre-trained continual learning must exploit prior knowledge adaptively while preserving generalizability across future tasks.
A uniform learning rate makes sequential fine-tuning overfit representations to incremental tasks, progressively reducing their generalizability.
SLCA selectively reduces the representation learning rate and aligns classification layers using class-wise distributions in post-hoc processing.
49.76%, 50.05%, 44.69% and 40.16% improvements are reported on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively.

2. Related Work

Related work has traditionally trained continual-learning models from scratch, while newer approaches exploit supervised or self-supervised pre-training. Prompt-based methods and self-supervised pre-training have shown advantages, motivating further study of pre-trained continual learning.

Traditional continual learning methods sequentially train models from scratch while mitigating forgetting through regularization, replay, or related strategies.
Supervised pre-training supports knowledge transfer and robustness to catastrophic forgetting in downstream continual learning.
Prompt-based approaches were reported to substantially outperform traditional continual-learning baselines, challenging the prevailing computer-vision paradigm.
Self-supervised pre-training is preferable when pre-training data are unlabeled or arrive incrementally, and has been associated with less catastrophic forgetting upstream.

3. Continual Learning on a Pre-trained Model

Continual learning on a pre-trained model must adapt pre-trained representations to new tasks without losing future generalizability. The analysis identifies progressive overfitting and develops Slow Learner with Classifier Alignment to address representation and classification-layer problems.

Problem Formulation: CLPM adapts a pre-trained representation layer and classification layer across incremental tasks without task labels during evaluation.The representation layer transfers pre-trained knowledge, while the classification layer produces predictions for all classes seen so far.
Problem Formulation: Sequentially fine-tuning the entire model with uniform learning rates causes progressive overfitting of pre-trained representations and catastrophic forgetting of earlier tasks.Updating representations can erode pre-training generalizability, while updating the classifier and representations can interfere with previously learned task knowledge.
Problem Formulation: Traditional baselines focus mainly on forgetting, whereas prompt-based approaches fix representations and add learnable parameters to improve CLPM performance.The cited prompt-based methods include L2P and DualPrompt; traditional baselines include regularization- and replay-based approaches.
Slow Learner: Using learning rates 0.0001 for representations and 0.01 for classifiers substantially improves sequential fine-tuning over the uniform 0.005 setting.This selective reduction slows representation updates while allowing somewhat faster classifier learning; the appendix provides a broader learning-rate analysis.
Slow Learner: Slow Learner reduces the continual-learning gap to joint training to 4.36% on Split CIFAR-100 and 7.80% on Split ImageNet-R for Seq FT.The upper-bound joint-training performance changes only marginally, while the continual-learning gap is greatly reduced.
Classifier Alignment: SLCA further aligns classification layers by modeling each class with its feature mean and covariance, sampling features post hoc, and normalizing logit magnitude.Linear probing shows that Slow Learner largely addresses representation quality, while the classification layer remains sub-optimal, especially for fine-grained benchmarks.

4. Experiments

The experiments evaluate CLPM across benchmark datasets, pre-training paradigms, baseline families, and ablations. SLCA substantially improves continual learning, with classifier alignment especially valuable for fine-grained tasks, while remaining computationally lightweight.

Experimental Setups: The study evaluates four downstream datasets split into 10 disjoint tasks, spanning relatively coarse-grained and fine-grained classification.It considers CIFAR-100, ImageNet-R, CUB-200, and Cars-196.
Experimental Setups: The benchmark compares joint training with replay-based, regularization-based, prompt-based, and fine-tuning continual learning baselines.Replay methods use a 1000-image memory buffer.
Experimental Results: SLCA nearly reaches the joint-training upper bound on coarse-grained benchmarks, with gaps below 2% for supervised and 4% for self-supervised pre-training.It clearly outperforms L2P and DualPrompt in these settings.
Experimental Results: Different replay methods respond differently to Slow Learner: BiC improves most substantially, whereas GDumb adapts poorly to the pre-trained model with limited samples.The methods differ in how they update or retrain the model using old samples.
Ablation Study: Seq FT with fixed representations outperforms uniform-rate Seq FT but remains significantly below Seq FT with Slow Learner, supporting reduced-rate representation updates.The ablation identifies both representation updating and learning-rate reduction as important.
Ablation Study: For ImageNet-21K supervised pre-training, CA+LN improvements are 2.67%, 5.20%, 19.64% and 17.99% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively.The gains are larger on the fine-grained CUB-200 and Cars-196 benchmarks.
Pre-training Paradigm and Downstream Granularity: Fine-grained tasks increase the performance gap because of a sub-optimal classification layer, which classifier alignment can greatly improve.The paper reports that CA+LN is particularly beneficial in these settings.
Scalability: Generated features support test-time output-layer alignment using only 0.67% to 5% of total running time across benchmarks.The covariance can also be simplified to a variance.

5. Conclusion

The conclusion identifies slow representation learning and classifier alignment as an effective route for CLPM. SLCA brings sequential fine-tuning close to joint-training performance, while the paper leaves broader settings for future work.

Conclusion: CLPM must transfer pre-trained knowledge to each incremental task while preserving generalizability for future tasks.Uniformly updating the entire model fails to accommodate both objectives.
Conclusion: SLCA combines a slow learner for the representation layer with post-hoc classifier alignment for the classification layer.The approach targets progressive representation overfitting and classification-layer deficiencies.
Conclusion: Across upstream and downstream scenarios, SLCA enables sequential fine-tuning to almost reach the joint-training upper bound and exceed current state-of-the-art performance.The paper also identifies pre-training paradigm and downstream granularity as critical factors.
Limitations: The study does not address joint upstream and downstream continual learning, other backbones beyond ViT, or downstream applications such as detection and segmentation.The experiments are based on a pre-trained ViT backbone for comparison with previous works.

A. More Details and Results.

Additional analyses examine implementation choices, representation similarity, another dataset, method combinations, and representation learning rates. They extend evaluation of SLCA and its computational and storage requirements.

Implementation Details: The implementation uses a pre-trained ViT-B/16 backbone, Adam for prompting methods, SGD for other baselines, and batch size 128.The slow learner uses learning rate 0.0001, compared with 0.005 for the original baseline implementation.
Extended Analysis: Extended analysis compares CKA similarity of pre-trained representations before and after downstream-task learning and after joint versus continual learning.These comparisons are presented in Figures 7 and 8.
Results on Additional Dataset: SLCA delivers consistently strong performance on Sketch-345, a 345-class sketch-image subset of DomainNet.The results are reported in Table 4.
Combine with other methods: The efficacy of Classifier Alignment is additionally validated when combined with representative non-replay and replay methods.These combinations are evaluated on ImageNet-21K supervised pre-training.
Learning-Rate Analysis: The study reports continual-learning performance under different representation-layer learning rates using Last-Acc after all classes are learned.The comparison includes ImageNet-21K supervised and ImageNet-1K self-supervised pre-training.