Source-linked AI summary
CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, Zsolt Kira
TL;DR
Rehearsal-free continual learning avoids storing private past data but existing prompting methods limit learning capacity and are not optimized end-to-end. CODA-Prompt decomposes prompts into learnable components with attention-based weighting, achieving state-of-the-art results across established and dual-shift benchmarks.
Problem
Rehearsal can require storing private past data, while DualPrompt provides fixed per-task capacity and lacks end-to-end optimization for new tasks.
Method
CODA-Prompt assembles learnable prompt components with attention-based weights and inserts the resulting prompts into a pre-trained ViT encoder, optimizing the method end-to-end.
Results
4.5%: CODA-Prompt beats DualPrompt by as much as 4.5% in average final accuracy on ImageNet-R and CIFAR-100, while also outperforming the state of the art on a dual-shift benchmark.
Takeaways & Limitations
CODA-Prompt scales prompting capacity and sets a new state of the art on established rehearsal-free and dual-distribution-shift benchmarks.
Takeaways & Limitations
Hyperparameters for DualPrompt and CODA-Prompt were tuned on ImageNet-R to evaluate robustness of all methods.
Abstract
from arXiv · showhide
Computer vision models suffer from a phenomenon known as catastrophic forgetting when learning novel concepts from continuously shifting training data. Typical solutions for this continual learning problem require extensive rehearsal of previously seen data, which increases memory costs and may violate data privacy. Recently, the emergence of large-scale pre-trained vision transformer models has enabled prompting approaches as an alternative to data-rehearsal. These approaches rely on a key-query mechanism to generate prompts and have been found to be highly resistant to catastrophic forgetting in the well-established rehearsal-free continual learning setting. However, the key mechanism of these methods is not trained end-to-end with the task sequence. Our experiments show that this leads to a reduction in their plasticity, hence sacrificing new task accuracy, and inability to benefit from expanded parameter capacity. We instead propose to learn a set of prompt components which are assembled with input-conditioned weights to produce input-conditioned prompts, resulting in a novel attention-based end-to-end key-query scheme. Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4.5% in average final accuracy. We also outperform the state of art by as much as 4.4% accuracy on a continual learning benchmark which contains both class-incremental and domain-incremental task shifts, corresponding to many practical settings. Our code is available at https://github.com/GT-RIPL/CODA-Prompt
1. Introduction
Continual learning must adapt vision models to emerging concepts without overwriting prior knowledge or retaining private training data. CODA-Prompt addresses this rehearsal-free setting with end-to-end decomposed prompting and reports state-of-the-art results across standard and dual-shift benchmarks.
- Dynamic deployment requires continual updates because real-world data contains continuously emerging objects and categories.
- Rehearsal can improve continual learning but may be impractical because storing private user data creates privacy and memory concerns.
- CODA-Prompt uses learnable prompt components assembled with input-conditioned weights, expanding prompting capacity and enabling end-to-end optimization.
- CODA-Prompt also outperforms the state of the art on a benchmark combining semantic and covariate distribution shifts.
2. Background and Related Work
Continual learning methods trade off retention, scalability, computation, and data availability. Rehearsal is effective but can conflict with privacy and memory constraints, motivating rehearsal-free approaches using pretrained vision transformers.
- Continual learning methods include architecture expansion, parameter regularization, and rehearsal using stored or generated data.
- Rehearsal is highly effective when feasible, but long-term data storage can violate privacy and impose substantial memory costs.
- Generative-model rehearsal adds computational and memory demands, while model inversion is slow and underperforms rehearsal-based methods.
- Vision transformers have been studied for robustness to forgetting, including frozen pretrained models and parameter-isolation approaches.
3. Preliminaries
The paper studies class-incremental learning with pretrained vision transformers, where models learn non-overlapping classes sequentially without task labels at inference. Its prompting setup uses prefix-tuning, while prior L2P and DualPrompt select prompts through separately trained key-query matching.
- 3.1. Continual Learning: The setting presents non-overlapping class subsets sequentially and requires retaining classification across all previously seen classes.
- 3.1. Continual Learning: Class-incremental learning differs from task-incremental learning because task labels are unavailable during inference.
- 3.1. Continual Learning: Experiments also combine class-incremental and covariate domain shifts to model simultaneous semantic and domain changes.
- 3.2. Prompting with Prefix-Tuning: The method changes prompt selection and formation while keeping the vision-transformer prompting foundations aligned with DualPrompt.
- 3.2. Prompting with Prefix-Tuning: Prefix-tuning prepends prompts to the keys and values of selected multi-head self-attention layers rather than to input tokens.
- 3.3. L2P and DualPrompt: L2P and DualPrompt select instance-specific prompts by matching image-conditioned queries to learned keys using cosine similarity.
- 3.3. L2P and DualPrompt: Their keys are optimized separately from task classification loss, so task-loss gradients do not directly update the keys.
4. Method
CODA-Prompt forms input-conditioned prompts by weighting learnable prompt components with attention-based key-query matching, expanding capacity while reusing prior components. It freezes existing components as tasks arrive and optimizes the prompting and task parameters end-to-end with orthogonality regularization.
- Prompt Formation: CODA-Prompt replaces task-level prompt selection with weighted sums of learnable prompt components, allowing prompting capacity to expand while keeping prompt length fixed.The weighted sum uses component weights α and prompt components P_m.
- Full Optimization: All learnable parameters are optimized through the classification loss rather than separate key-query and task objectives, enabling end-to-end learning.The paper identifies this distinction from L2P and DualPrompt as important for benchmark performance.
- Prompt-Component Weighting: Each component receives a key and learnable attention vector, with attention-weighted query-key similarity determining its contribution to the assembled prompt.The query is element-wise multiplied by the attention vector before cosine similarity with the corresponding key.
- Architecture: The method is parameter-efficient and rehearsal-free: it learns prompting and task parameters without storing training data for replay.The approach passes inputs and produced prompts through a pretrained transformer and sends the outputs to a task head.
- Continual Learning: When a new task arrives, existing components are frozen and only newly added components are optimized to avoid overwriting prior knowledge.The component set expands with tasks, while previously learned components remain fixed.
- Full Optimization: An orthogonality constraint is applied to prompt components, keys, and attention vectors to reduce interference between existing and new knowledge.The orthogonality loss is balanced by hyperparameter λ in the full objective.
5. Experiments
CODA-P is evaluated against rehearsal-free and rehearsal-based baselines across class-incremental benchmarks, including a dual-shift ImageNet-R setting. It improves average accuracy over prior prompting methods and benefits from expanded prompt-component capacity, while ablations identify attention, freezing, and orthogonality as important components.
- Experimental setup: CODA-P is benchmarked with LwF, L2P, DualPrompt, FT, FT++, offline training, and Experience Replay using a pretrained ViT-B/16 backbone.Experiments use PyTorch implementations across several image datasets in the class-incremental setting.
- Forgetting and capacity: As task sequences lengthen, CODA-P’s forgetting converges toward DualPrompt’s while CODA-P maintains higher learning capacity.The authors report that CODA-P can accept marginally higher forgetting in exchange for learning new tasks more effectively.
- Existing benchmarks: +3.2% and +2.5% improvements are reported on ten-task CIFAR-100 and five-task DomainNet, respectively.The authors note that LwF slightly outperforms prompting on DomainNet.
- Existing benchmarks: 4.5% improvement in average accuracy over DualPrompt is reported on the 5-task ImageNet-R benchmark.ImageNet-R contains 200 object classes with diverse image styles and a distribution distant from ImageNet pretraining data.
- Dual-shift benchmark: 4.4% improvement in average accuracy is reported on ImageNet-R with dual semantic and covariate distribution shifts.Training data randomly removes image domains from each task while evaluation data remains unmodified.
- Ablations and analysis: Removing attention slightly worsens average accuracy and forgetting, while removing freezing or orthogonality causes larger performance drops.The ablation is conducted on 10-task ImageNet-R and identifies freezing and orthogonality as crucial aspects of CODA-P.
- Ablations and analysis: Increasing CODA-P’s component count improves average accuracy toward the offline upper bound, whereas L2P peaks at twice the task-sequence length and then declines.The analysis uses 5-task ImageNet-R and reports means over three trials; accuracy saturates with prompt length.
6. Conclusion
CODA-Prompt assembles learnable prompt components for insertion into a pre-trained ViT encoder and supports end-to-end optimization. It sets new state-of-the-art results on established benchmarks and a dual-distribution-shift benchmark.
- CODA-Prompt assembles learnable prompt components and inserts them into a pre-trained ViT encoder for image classification.
- End-to-end optimization distinguishes CODA-Prompt from prior state-of-the-art methods using separate optimizations.
- CODA-Prompt scales prompting capacity to arbitrary sizes.
- CODA-Prompt establishes new state-of-the-art results on established benchmarks and a dual-distribution-shift benchmark containing semantic and covariate shifts.
A. Additional Implementation Details
The experiments specify shared optimization, preprocessing, training, and validation procedures, alongside method-specific prompt settings. Classification training masks past-task logits to prevent new-class bias in the linear head.
- CIFAR-100 and DomainNet are trained for 20 epochs, while ImageNet-R is trained for 50 epochs to ensure convergence for each task.
- Validation tuning selects a 1e−3 learning rate for prompting methods and 1e−4 for methods that fully fine-tune the model.
- CODA-Prompt uses prompt length 8, 100 prompt components, and λ = 0.1 for orthogonality regularization after validation sweeps.
- Replacing past-task logits with negative infinity yields zero softmax probability for those classes and prevents gradients from reaching their linear-head outputs.
- This logit treatment prevents the linear head from learning a bias toward new classes over past classes in class-incremental learning.
B. Additional Results
Additional results report benchmark coverage, evaluation metrics, randomized-trial aggregation, and trainable or final parameter counts. Final average accuracy is emphasized because it reflects both plasticity and forgetting.
- Extended results cover 5-, 10-, and 20-task ImageNet-R, 10-task CIFAR-100, 5-task DomainNet, and dual-shift ImageNet-R.
- Final average accuracy A_N averages the final model’s test accuracy over N tasks, while average forgetting F_N averages performance drops over N tasks.
- A_N is the more important metric because it encompasses method plasticity and forgetting, whereas F_N provides additional context subject to plasticity.
- Means and standard deviations are computed across runs with different class-order shuffles and matched random seeds across methods within each trial.
- The evaluation reports trainable and final parameter counts as percentages of the pre-trained ViT backbone, with CODA-P-S designed below DualPrompt’s parameter count in the main setting.
C. ImageNet-R Dual-Shift Benchmark
The dual-shift ImageNet-R benchmark tests continual learning under semantic and covariate distribution shifts. It uses class tasks with domain types removed from training, while the reported tables define accuracy, forgetting, parameter counts, and trial aggregation.
- The dual-shift benchmark targets robustness to semantic and covariate continual distribution shifts.
- The 5-task ImageNet-R, DomainNet, and covariate-shift results report means and standard deviations over 5, 3, and 5 trials, respectively.
- Removing domains such as clipart, paintings, and cartoon makes the continual-learning task more challenging through injected image-type domain shifts.
- Tables report accuracy averaged over tasks, average forgetting, and trainable and final parameters relative to the pre-trained ViT backbone.
- ImageNet-R is divided into 5 tasks of 40 classes, with 8 of 15 image-type domains randomly removed from each task’s training data.