PaperlayerView exact revision on arXiv

Source-linked AI summary

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning

Yabin Wang, Zhiwu Huang, Xiaopeng Hong

arXiv:2207.12819v2cs.CVcs.LG

TL;DR

Continual learning remains vulnerable to catastrophic forgetting, and exemplar-free DIL seeks to avoid storing prior examples. The paper introduces S-Prompts, which independently learns prompts for each domain using fixed pre-trained transformers. Across three standard DIL tasks, its best approach reports about 30% average relative improvement over the best exemplar-free methods, with 0.03% parameter growth per domain.

  • Problem

    Continual learning struggles with catastrophic forgetting, while exemplar-free DIL avoids storing old-session examples for security, privacy, and memory reasons.

  • Method

    S-Prompts independently learns domain prompts with fixed pre-trained transformers, using image or synchronously learned language-image prompts.

  • Results

    30% average relative improvement is reported over the best exemplar-free methods across three standard DIL benchmark datasets, with 0.03% parameter growth per domain.

  • Takeaways & Limitations

    Independent prompting provides a simple exemplar-free DIL paradigm that reduces forgetting while scaling to many domains with small parameter increases.

  • Takeaways & Limitations

    The paradigm does not obtain appealing results when applied directly to class-incremental learning because CIL is more sensitive to incorrect domain or task identification.

Abstract

from arXiv · show

State-of-the-art deep neural networks are still struggling to address the catastrophic forgetting problem in continual learning. In this paper, we propose one simple paradigm (named as S-Prompting) and two concrete approaches to highly reduce the forgetting degree in one of the most typical continual learning scenarios, i.e., domain increment learning (DIL). The key idea of the paradigm is to learn prompts independently across domains with pre-trained transformers, avoiding the use of exemplars that commonly appear in conventional methods. This results in a win-win game where the prompting can achieve the best for each domain. The independent prompting across domains only requests one single cross-entropy loss for training and one simple K-NN operation as a domain identifier for inference. The learning paradigm derives an image prompt learning approach and a novel language-image prompt learning approach. Owning an excellent scalability (0.03% parameter increase per domain), the best of our approaches achieves a remarkable relative improvement (an average of about 30%) over the best of the state-of-the-art exemplar-free methods for three standard DIL tasks, and even surpasses the best of them relatively by about 6% in average when they use exemplars. Source code is available at \url{https://github.com/iamwangyabin/S-Prompts}.

1 Introduction

Continual learning suffers from catastrophic forgetting, while exemplar-free DIL must avoid storing prior examples. S-Prompts addresses this by independently prompting fixed pre-trained transformers for each domain.

  • Catastrophic forgetting occurs when networks trained on new tasks forget knowledge from previously learned tasks.
  • S-Prompts produces separate subspaces for domains, making classes more separable than dependent sequential learning.
  • Exemplar-free DIL avoids storing old-task examples to improve data security, privacy, and memory consumption.
  • S-Prompts learns domain prompts independently with pre-trained transformers, avoiding exemplars while reducing catastrophic forgetting.Fixed transformers extract general features, while tuned prompts encode domain knowledge.
  • The paradigm includes image-only and language-image approaches, with the latter synchronously prompting CLIP’s language and image transformers.
  • 30% relative improvement is reported over the best exemplar-free methods across three standard DIL benchmark datasets, with 0.03% parameter growth per domain.

2 Related Work

Related work addresses continual learning through memory, distillation, and prompting, while S-Prompts differs by independently learning joint language-image prompts. Its inference pipeline uses K-NN domain identification after K-Means centroid construction.

  • Continual Learning: Memory-based methods replay stored or synthesized examples, whereas distillation-based methods use knowledge distillation to mitigate catastrophic forgetting.
  • Continual Learning: DIL adaptations often reuse CIL methods by treating shared classes across domains as different incremental classes.
  • Prompt Learning: Most prompting methods learn image-end or language-end prompts, while S-Prompts explicitly learns joint language-image prompting.
  • Prompt Learning: S-liPrompts identifies a test domain with K-NN over centroids obtained by K-Means in the fixed image-transformer feature space.

3 Proposed Approach

S-Prompts addresses exemplar-free domain-incremental learning by independently learning domain-specific prompts with fixed pre-trained transformers. It instantiates this paradigm with ViT-based image prompts and CLIP-based synchronized language-image prompts, using domain-specific classifier pools for inference.

  • S-Prompts Learning Paradigm: S-Prompts learns domain-specific prompts independently across sequential domains while keeping pre-trained transformer features fixed.Each domain's knowledge is encoded by its own prompt or prompt set, avoiding exemplars during incremental learning.
  • Concrete Approaches: The paper develops Image S-Prompts using ViT and Language-Image S-Prompts using CLIP.The two approaches mainly differ in prompt and classifier design.
  • Image S-Prompts: Image S-Prompts append an independent continuous image prompt to each domain's image and maintain a separate fully connected classifier for every domain.The classifier pool stores domain-specific linear projections, and inference selects the classifier associated with the selected domain center.
  • Language-Image S-Prompts: Language-Image S-Prompts synchronously prompt CLIP's image and text transformers with domain-specific learnable image and language contexts.The language-end prompt replaces manually designed CLIP prompts with learnable context vectors, while text prompts form a domain-wise pool.
  • Language-Image S-Prompts: Learnable context prompts improve adaptation of pre-trained CLIP to varied domains, while CLIP classifiers provide more domain information and reduce stored context per session.Class names are shared across domains, so only one domain context needs to be stored for each session.
  • Language-Image S-Prompts: CLIP-based classifiers compute image-class probabilities from cosine similarities between image features and text features, followed by softmax activation.The text encoder supplies class-specific weight vectors, while domain-specific context prompts provide domain information.

4 Experiments

Experiments on CDDB-Hard, CORe50, and DomainNet evaluate S-Prompts against exemplar-free and exemplar-based continual-learning methods, along with ablations, feature-space visualizations, efficiency, and out-of-domain generalization. S-liPrompts and S-iPrompts achieve strong accuracy with low forgetting, scalable overhead, and improved generalization under domain-incremental learning.

  • Efficiency: S-liPrompts adds only 0.03% memory per domain, while domain identification adds about 2.5 ms over pretrained CLIP inference.Classification averages 11.84 ms for S-liPrompts versus 9.37 ms for pretrained CLIP on one RTX-3090.
  • Main results: S-iPrompts and S-liPrompts significantly outperform exemplar-free competitors, with S-liPrompts averaging roughly 30% relative improvement in forward classification accuracy.The methods also show much lower forgetting than competing approaches across the reported experiments.
  • Main results: S-liPrompts achieves about 6% average relative improvement over the best methods using exemplars.The comparison includes methods that retain old-task exemplars, whereas S-Prompts is exemplar-free.
  • Ablations: 1-NN gives the best domain-incremental accuracy among tested K-NN settings because other K values reduce domain-identification accuracy.Language prompts also benefit from sufficient length, with 16 being the longest tested length.
  • Scenario comparison: S-liPrompts nearly closes the task-agnostic accuracy gap between domain-incremental and task-incremental learning.Task-incremental learning supplies domain indexes at inference, whereas domain-incremental learning does not.
  • Generalization: S-liPrompts generalizes better than DyTox to three unseen CDDB-Hard domains, with OOD accuracy increasing as more seen domains are learned.The OOD evaluation uses FaceForensic++, Glow, and StarGAN in both task-wise and task-agnostic settings.

5 Discussion and Conclusion

The paper proposes independent, prompt-based learning for domain-incremental learning, avoiding prior-domain expert knowledge while reducing forgetting. Its empirical study reports about 30% relative improvement over state-of-the-art competitors, alongside scalability to many domains, but direct application to class-incremental learning remains limited.

  • Proposed paradigm: S-Prompts learns expert knowledge independently for each domain using prompts and pre-trained transformers, then gathers the learned prompts in a pool for inference.The paradigm avoids using expert knowledge from previously learned domains except general knowledge from pre-trained transformers.
  • Proposed paradigm: Independent knowledge learning avoids the traditional tug-of-war among learning sessions and instead targets the best fit for each session.
  • Empirical findings: 30% relative improvement over state-of-the-art competitors is reported, together with scalability to a large number of incrementally appearing domains.
  • Limitations and outlook: The paradigm does not obtain appealing results when applied directly to class-incremental learning because those tasks are more sensitive to incorrect domain or task identification.The paper also notes potential relevance to transfer learning and zero-shot or few-shot learning, including unseen-domain generalization.

6 Appendix

The appendix documents implementation, inference, memory, visualization, ablation, and failure-case analyses for S-Prompts. It describes K-NN-based prompt selection, reports low per-domain parameter growth, and connects performance gains to independent prompting and domain separation.

  • Inference pipeline: S-liPrompts inference extracts an image feature, selects a nearest K-Means centroid with K-NN, applies associated image and language prompts, and computes language-image probabilities.The image and text transformers receive the corresponding tokens, prompts, and class information before final classification.
  • Training procedure: S-liPrompts trains independently initialized image and language prompts for each domain, using transformer encoders and cross-entropy supervision before clustering training features.The training algorithm incrementally adds domain-specific prompts and later runs K-Means on stored features to obtain domain centroids.
  • Memory consumption: 80.89K parameters per session, or a 0.03% relative increase, is added by S-liPrompts, while S-iPrompts adds 52.22K parameters or 0.05%.The experiments use five cluster centers for each benchmark dataset, contributing additional per-session parameters.
  • Failure cases: Failure cases concentrate on WildDeepfake and SAN, including blurry or occluded real faces and irregular-looking buildings that are confused across real and fake classes.The appendix identifies these domains as challenging cases in the reported examples.
  • Feature-space analysis: t-SNE primarily demonstrates domain separation rather than class separation because it projects high-dimensional features into two dimensions.The appendix therefore supplements visualization with quantitative separation metrics.
Loading 2207.12819v2…