PaperlayerView exact revision on arXiv

Source-linked AI summary

GAP-Prompt: Gated Adaptive Prompting for Efficient Continual Learning

Trung-Anh Dang, Duy-Cuong Bui, Ngoc-Son Vu, Christel Vrain, Vincent Nguyen

arXiv:2608.23782v1cs.LG

TL;DR

Continual learning must preserve prior knowledge while acquiring sequential tasks, yet static prompting overlooks instance-specific visual variation. GAP-Prompt uses instance-conditioned gating, dynamic knowledge fusion, and shared prompt distillation; it reports state-of-the-art results across benchmarks, including 89.24% average accuracy on CIFAR-100. Its evaluated setting includes sequential task learning, and its prompt topology remains a limitation when shared across all inputs within a task.

  • Problem

    Continual learning struggles with catastrophic forgetting, while static prompt insertion overlooks instance-specific visual variations and requires nontrivial placement choices.

  • Method

    GAP-Prompt combines instance-conditioned layer-wise gating, dynamic fusion of current and historical prompts, and shared prompts learned in early layers.

  • Results

    GAP-Prompt achieves state-of-the-art performance across CIFAR-100, ImageNet-R, and CUB-200, with 89.24% average accuracy on CIFAR-100.

  • Takeaways & Limitations

    The framework provides instance-level prompt adaptability, cross-task knowledge reuse, and improved knowledge retention within prompt-based continual learning.

  • Takeaways & Limitations

    The evaluated continual-learning setting consists of a sequence of T tasks, and task-conditioned gating remains shared across all inputs within each task.

Abstract

from arXiv · show

Continual learning faces the persistent challenge of catastrophic forgetting, where sequential task updates degrade previously acquired knowledge. While prompt-based methods integrated with pre-trained models offer a compelling solution by freezing the backbone, they often rely on static, task-level prompting strategies that overlook fine-grained intra-task diversity. In this paper, we propose Gated Adaptive Prompting (GAP-Prompt), a novel method that introduces instance-level adaptability to the prompting process. GAP-Prompt consists of three synergistic modules: (1) instance-conditioned gating, which dynamically determines optimal prompt injection layers for each individual image; (2) dynamic knowledge fusion, which performs instance-aware aggregation of current and historical prompts, enabling knowledge integration across tasks; and (3) shared prompt distillation, which anchors foundational knowledge in early shared layers to mitigate forgetting. Extensive evaluations on CIFAR-100, ImageNet-R, and CUB-200 benchmarks demonstrate that GAP-Prompt consistently achieves state-of-the-art performance. Notably, on the fine-grained CUB-200 dataset, GAP-Prompt reaches 87.29% accuracy, approaching the joint training upper bound (88.00%) and outperforming existing methods by a significant margin.

1 Introduction

Prompt-based continual learning freezes pre-trained backbones to reduce forgetting, but static task-level prompt placement overlooks instance-level visual diversity. GAP-Prompt addresses this with instance-conditioned gating, dynamic knowledge fusion, and shared prompt distillation, achieving strong benchmark performance.

  • Continual learning must acquire sequential task knowledge while preventing catastrophic forgetting and balancing memory, privacy, and computational costs.
  • Prompt-based continual learning freezes the backbone and optimizes compact task-specific prompts, but shared pools can interfere while independent prompts limit knowledge reuse.
  • Static or task-conditioned prompt placement remains fixed across inputs, ignoring instance-specific visual complexity and potentially misallocating layer-wise prompts.
  • GAP-Prompt combines instance-level adaptability with prompt efficiency by using compact task-specific prompts and lightweight gating modules instead of dense composition machinery.
  • GAP-Prompt dynamically activates prompts per image and progressively integrates historical prompts with the current task prompt for adaptive cross-task knowledge reuse.
  • GAP-Prompt achieves state-of-the-art results across multiple continual-learning benchmarks while improving accuracy, mitigating forgetting, and remaining stable across task variations.

2 Related works

Continual learning methods seek to preserve earlier knowledge while learning sequential tasks, but existing prompting approaches face integration and triggering limitations. RainbowPrompt unifies task prompts, whereas GAP-Prompt uses input context to improve prompt activation.

  • Continual learning methods address catastrophic forgetting through regularization, replay, and related strategies, each involving practical trade-offs as task counts grow.
  • Prompt-based vision continual learning uses ViT prompting to retain task-specific knowledge without requiring a memory-intensive replay buffer.
  • RainbowPrompt aggregates independent task prompts into a unified structure, but its reliance on static statistics can produce suboptimal triggering.

3 Methodology

GAP-Prompt operates in rehearsal-free class-incremental learning with a frozen ViT backbone, augmenting attention layers with shared and task-specific prompts. It replaces fixed prompt-injection choices with instance-conditioned gating and combines prompts across tasks.

  • Class-incremental learning uses non-overlapping classes, unknown task identity at inference, and no stored past samples.
  • The frozen ViT backbone preserves base knowledge while learnable prompt tokens adapt representations for continual tasks.
  • Prefix tuning prepends learnable key and value prompts to the key and value representations in ViT self-attention layers.
  • Fixed prompt injection across all layers or selected early layers can be suboptimal because ViT layers capture different levels of visual abstraction.
  • RainbowPrompt learns task-conditioned prompt-insertion gates, but fixes the resulting topology across all inputs within each task.
  • GAP-Prompt partitions prompts into shared and task-specific components and uses shared prompts in early layers alongside dynamically selected expert prompts.
  • Instance-conditioned gates adapt prompt activation to each image, while dynamic fusion aggregates current and historical task prompts according to those gates.

3.3 Gated adaptive prompting mechanism

GAP-Prompt combines instance-conditioned gating, dynamic knowledge fusion, and shared prompt distillation to adapt prompt use per image while reusing prior task knowledge and limiting forgetting. Its inference procedure queries task keys, retrieves gated prompts, and fuses them with shared early-layer prompts.

  • 3.3.1 Instance-conditioned gating (ICG): ICG replaces RainbowPrompt’s task-level sampled mask with generative, instance-specific gates that address intra-task visual diversity.
  • 3.3.1 Instance-conditioned gating (ICG): Gumbel-noise sigmoid relaxation and temperature annealing move gates from exploratory soft decisions toward sharper continuous activations without forcing binary values.
  • 3.3.1 Instance-conditioned gating (ICG): The gating network derives layer-wise activation parameters from each image’s query representation, which is extracted from the frozen ViT class token.
  • 3.3.2 Dynamic knowledge fusion (DKF): DKF uses ICG weights to aggregate the current prompt with frozen expert prompts from previously learned tasks for each input.
  • 3.3.2 Dynamic knowledge fusion (DKF): Normalized gated fusion forms a soft ensemble and stabilizes the magnitude of the injected prompt relative to the pretrained backbone.
  • 3.3.2 Dynamic knowledge fusion (DKF): Unlike dense prompt-component composition or static unified representations, GAP-Prompt controls where and how strongly prompts enter attention layers.
  • 3.3.3 Shared prompt distillation (SPD): SPD distills first-two-layer representations from previous tasks so shared prompts preserve earlier features while remaining learned across tasks.
  • 3.3.5 Inference process: During inference, the query selects the closest task key, gates and retrieves prompts without Gumbel noise, and combines the result with shared early-layer prompts.

4 Experiments

GAP-Prompt is evaluated for rehearsal-free 10-task class-incremental learning on CIFAR-100, ImageNet-R, and CUB-200 using Average Accuracy and Forgetting. Across benchmarks and ablations, instance-conditioned gating, dynamic knowledge fusion, and shared prompt distillation improve accuracy, reduce forgetting, and maintain robustness across task variations and pre-training paradigms.

  • Experimental setup: Evaluations cover CIFAR-100, ImageNet-R, and fine-grained CUB-200 under 10-task class-incremental learning, using Average Accuracy and Forgetting.Average Accuracy measures performance across tasks after the sequence; Forgetting measures performance drops on previously learned tasks.
  • Main results: 89.24% average accuracy on CIFAR-100 surpasses EvoPrompt by +1.27%, while GAP-Prompt records 3.03% forgetting.The result is attributed to the combined effect of shared prompt distillation and dynamic knowledge fusion anchoring historical knowledge.
  • Main results: 78.72% average accuracy on ImageNet-R exceeds CPrompt’s 77.14% and EvoPrompt’s 76.83%, with the benchmark’s lowest forgetting rate at 3.12%.The reported results cover a larger and more diverse dataset than CIFAR-100.
  • Main results: 87.29% average accuracy on CUB-200 approaches the 88.00% joint-training upper bound and exceeds OVOR-Deep by +9.18% and RainbowPrompt by +16.86%.Forgetting is reported as 3.68% on this fine-grained benchmark.
  • Robustness: Across all three datasets, GAP-Prompt achieves the highest accuracy and lowest forgetting, while self-supervised-backbone results remain consistently improved across iBOT-1K and DINO-1K settings.With iBOT-1K, it reaches 79.90% on CIFAR-100 and 69.29% on ImageNet-R; with DINO-1K, it reaches 66.46% on ImageNet-R versus CODA-Prompt’s 63.15%.
  • Task-split analysis: On CUB-200 task-split comparisons, GAP-Prompt consistently outperforms RainbowPrompt after the first, fifth, and tenth tasks while showing less forgetting and lower run-to-run variation.The first-task comparison isolates instance-conditioned gating; later comparisons additionally involve dynamic knowledge fusion and shared prompt distillation.
  • Ablation: ICG raises CIFAR-100 average accuracy from 84.78% to 87.82% (+3.04%), DKF raises it to 88.94%, and SPD reaches 89.24% while reducing forgetting to 3.03%.The full combination outperforms the baseline by 4.46% in accuracy and reduces forgetting by 1.53%.
  • Prompt position granularity: ICG outperforms task-based prompting by 1.25% in average accuracy and reduces forgetting by 1.63% on CIFAR-100.The comparison uses a fixed-position baseline without ICG, DKF, or SPD before integrating either ICG or task-based adaptive prompting.

5 Conclusion

GAP-Prompt is presented as an efficient prompt-based continual-learning framework that combines instance-conditioned adaptation, historical knowledge integration, and shared prompt distillation. Experiments across three benchmarks support its performance in accuracy and forgetting mitigation, with particularly strong results on fine-grained bird classification.

  • GAP-Prompt combines adaptive plasticity through ICG, historical knowledge integration through dynamic knowledge fusion, and foundational stability through shared prompt distillation.These components target new-knowledge acquisition, reuse of prior prompts, and preservation of previously learned knowledge.
  • Experiments on CIFAR-100, ImageNet-R, and CUB-200 demonstrate superiority over existing state-of-the-art methods in accuracy and forgetting mitigation.
  • Strong performance on fine-grained bird species classification highlights GAP-Prompt’s ability to adapt to complex representation spaces.
  • The paper positions GAP-Prompt as a scalable paradigm for lifelong learning and identifies online continual learning as a future direction.The stated future work concerns more challenging continual-learning settings.

A Hyperparameter selection

GAP-Prompt uses grid search for hyperparameter selection and places shared and task-specific prompts in different ViT layers. The final design leaves the highest layers unprompted because they support final classification readout.

  • Hyperparameter selection: Grid search is used to select GAP-Prompt hyperparameters, with final chosen values reported in Table 6.Table 5 describes the hyperparameters, while Table 6 lists the selected settings.
  • Prompt placement: Shared prompts are injected into the first two layers, while task-specific expert prompts occupy the next eight layers with instance-conditioned gates.
  • Prompt placement: The last two layers receive no prompts because they focus on [CLS]-token updating for final classification readout, while spatial tokens largely bypass them through skip connections.The passage states that task-specific prompting at this stage can disrupt feature aggregation and produce negligible or negative effects.

B Algorithm overview

GAP-Prompt training initializes shared and task-specific prompts, extracts an image query, generates instance-conditioned gates, performs dynamic knowledge fusion, and optimizes the resulting objectives. During inference, it queries the task, retrieves and fuses prompts, forwards the image through the frozen ViT, and predicts across learned classes.

  • Algorithm 1: GAP-Prompt training: Training initializes shared prompts and, for each task, task-specific expert prompts, task keys, and linear gating modules.
  • Algorithm 1: GAP-Prompt training: The training procedure extracts a query from the frozen ViT and uses a linear module to compute instance-conditioned gates.
  • Algorithm 1: GAP-Prompt training: Dynamic knowledge fusion is performed to obtain fused prompts, followed by computation of the matching and cross-entropy losses.
  • Algorithm 1: GAP-Prompt training: When learning beyond the first task, the training procedure applies shared prompt distillation in addition to the other objectives.
  • Algorithm 2: GAP-Prompt inference: Inference queries the task, computes instance-conditioned gates, retrieves and fuses prompts, forwards the image through the frozen ViT, and predicts across all learned classes.
  • Algorithm 2: GAP-Prompt inference: The inference output is the predicted label produced from logits computed with the classification head.

C Design choices and stability analysis

The supplementary analysis examines GAP-Prompt’s gating design, its relationship to prior prompting strategies, the suitability of Gumbel-Sigmoid, and normalized fusion for historical prompts.

  • Gating network: The analysis compares the lightweight linear gate with a bottleneck MLP for the gating network.
  • Comparison with prior methods: It contrasts ICG with dense attention-based component weighting in CODA-Prompt and task-level gating in RainbowPrompt.
  • Gating formulation: The section discusses why Gumbel-Sigmoid is suitable for instance-conditioned gating.
  • Dynamic knowledge fusion: The analysis explains why DKF uses normalized fusion for instance-conditioned integration of historical prompts.

C.1 Gating module architecture design

GAP-Prompt evaluates whether a bottleneck MLP or linear layer is better for learning its gating network. Across three datasets, replacing the linear gate with an MLP consistently reduces average accuracy.

  • Interpretation: The bottleneck MLP was motivated as a more expressive and noise-resilient gating mechanism, but its empirical performance was worse than the linear alternative.The tested MLP follows a compression–expansion design with nonlinear activation.
  • Architecture comparison: The gating-network comparison evaluates Linear and MLP architectures on CIFAR-100, ImageNet-R, and CUB-200 under 10-task class-IL.The MLP uses downprojection, normalization, GELU activation, and up-projection layers.
  • Results: 0.96%, 2.45%, and 0.97% average-accuracy decreases occur on CIFAR-100, ImageNet-R, and CUB-200, respectively, when using the MLP.These decreases are measured relative to the linear gating layer.

C.2 Why GAP-Prompt differs from prior prompt integration methods

GAP-Prompt differs from prior prompt integration methods by making prompt activation and historical knowledge fusion instance-conditioned. ICG produces layer-wise activation strengths per input, while DKF recomputes normalized fusion of current and historical task prompts.

  • C.2 Why GAP-Prompt differs from prior prompt integration methods: GAP-Prompt contrasts with CODA-Prompt and RainbowPrompt through different prompt activation mechanisms and historical-knowledge integration strategies.The comparison covers ICG, DKF, dense component composition, and task-level prompt evolution.
  • C.2.1 How ICG differs from prior weighting/gating mechanisms: Unlike CODA-Prompt’s component composition, ICG controls the activation strength of a task-specific prompt at a particular layer.CODA-Prompt uses input-conditioned weights to assemble decomposed prompt components.
  • C.2.1 How ICG differs from prior weighting/gating mechanisms: ICG predicts a prompt activation strength for each input and layer using Gumbel-Sigmoid relaxation.The gate is generated from a query feature extracted from the frozen ViT and constrained to [0, 1].
  • C.2.1 How ICG differs from prior weighting/gating mechanisms: Unlike RainbowPrompt’s fixed task-level insertion pattern, ICG allows samples from the same task to receive different continuous gate values.RainbowPrompt samples and fixes discrete gates after early training, sharing the resulting topology across task inputs.
  • C.2.1 How ICG differs from prior weighting/gating mechanisms: Gumbel-Sigmoid supports discrete-like activation decisions while retaining differentiability, with Gumbel noise encouraging exploration of layer-wise patterns.The sigmoid relaxation keeps gate values in [0, 1] for gradient-based optimization.
  • C.2.1 How ICG differs from prior weighting/gating mechanisms: Dense normalized attention lets every candidate prompt contribute to each input, which can reuse irrelevant historical prompts and introduce interference in class-incremental learning.This differs from independent activation decisions for task-specific prompts.
  • C.2.2 Why normalized dynamic knowledge fusion (DKF)?: DKF fuses current and historical task prompts on the fly using instance-conditioned gates rather than component coefficients or a stored task-level evolved representation.Its fused prompt is recomputed for every input.
  • C.2.2 Why normalized dynamic knowledge fusion (DKF)?: Figure 4 shows smoothly decreasing loss curves without severe oscillations, suggesting that input-specific DKF does not destabilize optimization.The stability claim is supported across task 2, task 5, and task 10 training curves on different datasets.

D Task querying accuracy and sensitivity analysis

GAP-Prompt’s inference first retrieves a task index using query–key matching, then uses the resulting candidate prompt range for DKF. Task-querying errors reduce accuracy, but input-conditioned gates preserve useful performance under some incorrect queries.

  • Task querying: Task querying predicts an index by matching the input query q with learned task keys, determining the candidate prompt range for DKF.The task matching metric measures how often the querying stage retrieves the correct task key before prompt fusion.
  • Task querying: 44.30% to 55.75% task-querying accuracy is reported across datasets, while final average accuracy remains high, especially on CIFAR-100 and CUB-200.The authors attribute this robustness to input-conditioned gates and reuse of useful prompts within the candidate range.
  • Querying correctness: 81.91% on CIFAR-100 and 82.57% on CUB-200 remain attainable under wrong task querying, although correct querying performs better.The candidate range may still contain useful historical prompts despite an incorrect queried task.
  • Error-direction sensitivity: Under-estimation is more harmful than over-estimation on ImageNet-R and CUB-200 because the correct task prompt may be excluded from the DKF candidate range.Accuracy drops from 90.65% to 66.38% on ImageNet-R and from 92.75% to 81.20% on CUB-200.

E Inference computational overhead and scalability

GAP-Prompt controls inference cost by restricting candidate prompts through task querying and thresholding gates before DKF. Its latency increases with task count but remains controlled, while measured computation is comparable to other prompt-based methods.

  • Inference operations: Inference adds task-key querying, linear ICG gates, and thresholded DKF fusion while keeping the ViT backbone frozen.The pipeline extracts q(x), computes candidate gates, and aggregates only active candidate prompts.
  • Candidate range: Task querying restricts fusion to the candidate range {1, . . . , ˆt(x)} instead of evaluating all learned tasks.The queried task index determines the range used for prompt fusion.
  • Thresholded sparsity: 29.61 of 60.3 candidate gates remain active on CIFAR-100 after 10 tasks; corresponding active/candidate counts are 19.18/49.9 on ImageNet-R and 7.06/52.7 on CUB-200.Thresholding reduces effective fusion workload by selecting only a subset of relevant prompts.
  • Latency scalability: 19.87 to 30.77 ms/image is the CIFAR-100 latency increase from 2 to 10 learned tasks, while 10-task latency remains around 29 to 30 ms/image on ImageNet-R and CUB-200.Latency is measured under the same hardware, batch size, and input resolution.
  • Scalability: The practical fusion cost depends on active task-layer prompts after thresholding rather than the total number of stored historical prompts.This design avoids dense aggregation as the number of learned tasks grows.
  • Method comparison: GAP-Prompt achieves comparable GFLOPs to DualPrompt, CODA-Prompt, and RainbowPrompt while using substantially fewer extra stored parameters than CODA-Prompt and RainbowPrompt on CIFAR-100.GFLOPs are averaged over the test set because routing is input-dependent.
Loading 2608.23782v1…