PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual prompting reimagined: The power of the Activation Prompts

Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen, Shuai Zhang, Pin-Yu Chen, Meng Wang, Sijia Liu

arXiv:2604.06440v1cs.CVcs.LG

TL;DR

Visual prompting adapts pretrained vision models through input perturbations, but its performance and efficiency remain limited relative to model-based adaptation. This paper introduces activation prompting, which applies perturbations to intermediate activations and uses the framework to analyze layer preferences and improve prompting. Across 29 datasets, AP improves VP and narrows its gap with other parameter-efficient methods, while its advantage is limited on compact models.

  • Problem

    Input-level visual prompting adapts pretrained vision models without changing parameters, but still lags behind model-based fine-tuning, motivating analysis of when VP is effective.

  • Method

    Activation prompting generalizes VP by inserting learnable perturbations into intermediate activation maps and analyzing their layer and architecture effects.

  • Results

    Across 29 datasets, AP enhances input-level VP and narrows its performance gap with six other state-of-the-art parameter-efficient fine-tuning methods.

  • Takeaways & Limitations

    AP provides an effective prompting framework and analytical tool, connecting prompting with normalization tuning while revealing model-dependent layer preferences.

  • Takeaways & Limitations

    For compact models such as ResNet-18 and ViT-Tiny, AP improves VP but does not outperform Norm-Tune, indicating dependence on pretrained-model size.

Abstract

from arXiv · show

Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP introduces a universal perturbation directly into the input data to facilitate task-specific fine-tuning rather than modifying model parameters. However, there exists a noticeable performance gap between VP and conventional fine-tuning methods, highlighting an unexplored realm in theory and practice to understand and advance the input-level VP to reduce its current performance gap. Towards this end, we introduce a generalized concept, termed activation prompt (AP), which extends the scope of the input-level VP by enabling universal perturbations to be applied to activation maps within the intermediate layers of the model. By using AP to revisit the problem of VP and employing it as an analytical tool, we demonstrate the intrinsic limitations of VP in both performance and efficiency, revealing why input-level prompting may lack effectiveness compared to AP, which exhibits a model-dependent layer preference. We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers, although each model type has distinct layer preferences for prompting. We also theoretically elucidate the rationale behind such a preference by analyzing global features across layers. Through extensive experiments across 29 datasets and various model architectures, we provide a comprehensive performance analysis of AP, comparing it with VP and parameter-efficient fine-tuning baselines. Our results demonstrate AP's superiority in both accuracy and efficiency, considering factors such as time, parameters, memory usage, and throughput.

1 Introduction

The paper examines why input-level visual prompting (VP) remains less effective than model-based fine-tuning and introduces activation prompting (AP) as a generalized alternative. AP applies learnable perturbations to intermediate activations, enabling analysis of layer preferences and improving VP across diverse settings.

  • Input-level VP reprograms pretrained vision models with task-specific input perturbations without changing model parameters, yet still trails model-based fine-tuning.
  • The paper asks when VP benefits vision tasks and when it is effective or ineffective, focusing on convolutional networks and vision Transformers.
  • AP generalizes VP by inserting learnable perturbations into intermediate activation maps rather than only the input layer.The framework treats VP as a special case, allowing AP to analyze and potentially improve input-based prompting.
  • AP concentrates prompts on a selected model layer, unlike VPT methods that distribute prompts across multiple layers; this distinction supports systematic layer-preference analysis.
  • The paper connects AP with normalization tuning and theoretically validates how layer preference relates to architecture and global-feature capacity.
  • Experiments across 29 datasets show that AP enhances input-level VP and narrows its performance gap with six other state-of-the-art parameter-efficient fine-tuning methods.

2 Related Work

Related work spans visual prompting, theoretical analyses of prompt engineering, and parameter-efficient fine-tuning. Prior studies cover prompt expressivity, optimization, generalization, and lightweight adaptation, but theoretical prompt analyses have often used restricted architectures.

  • Visual prompting: Visual prompting repurposes fixed pretrained models for new tasks through input perturbations, with later work improving label mapping, normalization, robustness, distribution-shift handling, and vision-language applications.
  • Theoretical study on prompt engineering: Theoretical prompt-engineering research studies expressive power, optimization, and generalization, with most work focusing on tuning-free in-context learning.
  • Theoretical study on prompt engineering: Existing soft-prompt theory includes subspace-optimization interpretations and generalization analysis for gradient-based tuning.
  • Parameter-efficient fine-tuning: Parameter-efficient fine-tuning either updates a small subset of pretrained parameters, such as normalization parameters, or adds modules to the backbone.

3 Activation Prompt: Design and Rationale

Activation prompting (AP) generalizes visual prompting from input pixels to intermediate activation maps, aiming to improve accuracy and efficiency while preserving a frozen backbone. Its effectiveness depends on the insertion layer and architecture, and it is closely related to normalization tuning.

  • AP: Generalizing VP in feature space: AP applies learnable, input-agnostic perturbations to intermediate feature maps, extending conventional input-level visual prompting into feature space.Unlike VP, AP can target a selected layer rather than modifying only the input.
  • Efficiency advantages: AP parameter count follows the selected feature-map size, while deeper placement can avoid back-propagation through the entire network and reduce training cost.These properties provide potential parameter and computational advantages over input-level VP.
  • Preliminary comparison: At ResNet-101 layer 31, AP surpasses VP by approximately 5% accuracy and uses only 100k parameters at its best-performing layer.The same experiment reports that deeper AP can improve computational efficiency because gradients need not propagate back to the input layer.
  • Connection to normalization tuning: AP and normalization tuning use related linear feature transformations, with the correspondence expressed through activation shifts and normalization scale-bias parameters.For CNNs, unit-scaling BatchNorm tuning closely mirrors AP when perturbations are consistent across feature maps; for ViTs, analogous equivalence arises under uniform token perturbations.
  • Connection to normalization tuning: AP’s flexible perturbations can exceed normalization tuning while leaving the pretrained model backbone parameters unchanged.The paper presents AP as both a prompting method and an analytical tool for studying VP.

4 A Deep Dive into AP: Layer and Architecture Effects

AP exhibits architecture-dependent layer preferences: CNNs favor deeper layers, whereas ViTs favor shallower layers. Feature analyses associate these preferences with how models capture global and local information across layers.

  • Architecture-dependent layer preference: AP’s effectiveness is contingent on the layer where it is installed, motivating separate analyses of ResNet and ViT architectures.The paper treats layer selection as a central design choice rather than applying prompts uniformly across layers.
  • Architecture-dependent layer preference: CNNs prefer AP in deeper layers, while ViTs tend to favor AP in shallower layers on OxfordPets.The comparison covers ResNet-50, ResNet-101, ViT-Base/12, and ViT-L/16; the best layer is marked separately for each architecture.
  • Global and local features: Deep ResNet features align predominantly with middle ViT layers, while CNNs emphasize low-level local concepts such as color and texture.The CKA comparison is used to relate representation depth across architectures.
  • Global and local features: Attention-distance analysis tracks global information across ViT layers, with larger attention distances indicating more global features.The analysis uses average attention distance across attention heads and layers.
  • Global and local features: The observed layer preferences are linked to the distribution of global features across layers, and the paper provides theoretical support for this architecture effect.The reported trend in global-feature ratio aligns with the layer patterns observed for AP.

5 Theoretical Analyses for Layer and Architecture Effects

The theoretical analysis studies whether ViT AP placement changes generalization and sample complexity. It concludes that shallow-layer AP can require fewer samples than deep-layer AP under the stated two-layer ViT assumptions.

  • Theoretical question: The theory asks whether shallow-layer AP for ViTs requires less sample complexity than deep-layer AP to achieve the same generalization performance.The setup focuses on global-feature conditions and compares AP placement across layers.
  • Theoretical setup: The analysis uses binary classification with a single-head, two-layer ViT, stochastic gradient descent, and population risk as the generalization criterion.The data contain one discriminative token among noisy discriminative and non-discriminative patterns.
  • Attention-distance analysis: The average attention distance decreases from (1 + d_A)/P to 1/P after the first layer in the simplified two-layer ViT.The lemma formalizes the layer-wise reduction in attention distance used by the subsequent sample-complexity analysis.
  • Sample-complexity result: N1 = Θ(P) samples suffice for first-layer AP, whereas N2 = Θ(P^2 log P) samples are required for second-layer AP, making N2 order-wise larger than N1.The theorem assumes batch size B ≥ Ω(1) and zero generalization error under the stated model and data conditions.
  • Sample-complexity result: With the same training samples and setup, shallow-layer AP generalizes better because deep-layer AP requires a larger prompt magnitude when global features are reduced.The paper attributes the difference to the difficulty of directing attention toward discriminative tokens in deeper layers.

6 Experiments

Experiments across diverse datasets, architectures, and data regimes evaluate AP against visual prompting, parameter-efficient baselines, and full fine-tuning. AP generally improves accuracy over VP while offering strong efficiency, though its advantage depends on model scale.

  • Datasets and models: Experiments cover more than 20 transfer-learning datasets from FGVC and VTAB, plus CIFAR-10, UCF101, GTSRB, Food101, and Waterbirds.The evaluation uses ResNet101 and ViT-L/16 pretrained on ImageNet-1K, with both full-data and VTAB-1k few-shot regimes.
  • Experimental setup: AP is evaluated against Linear-Probe, input-level VP, Norm-Tune, Full-Tune, and nine additional state-of-the-art PEFT baselines.Efficiency is measured by tunable parameters, memory cost, training time per epoch, and inference throughput.
  • Full-data results: AP outperforms VP on all 19 datasets, improving average accuracy by 4% for ResNet-101 and 1.5% for ViT-L/16.AP performs best on 13/15 datasets for ResNet-101/ViT-L/16, respectively, and can approach or surpass Full-Tune on selected datasets.
  • Efficiency: AP reduces memory and training time relative to VP while maintaining comparable parameter efficiency.For ResNet-101, memory decreases from 12.2G to 6.3G and training time from 72 to 41 s/epoch; the parameter-ratio difference is ≤0.05%.
  • Few-shot results: In the VTAB-1K few-shot setting, AP improves over VP by over 1% with ViT-L/16 and by 4.3% with ResNet-101.The study restricts training to 1000 samples per dataset.
  • PEFT comparison: Across eight PEFT methods, AP ranks roughly second to fourth in accuracy and first in efficiency.AdapterFormer achieves the best accuracy in the comparison but has three times lower throughput efficiency.
  • Architectural generalization: AP also works on CLIP and Swin-Transformer, where it significantly outperforms VP by 1%–6% and retains architecture-dependent layer preferences.The observed preference follows the ViT pattern for these architectures, favoring shallower layers.
  • Limitations: For compact models such as ResNet-18 and ViT-Tiny, AP improves VP but does not outperform Norm-Tune.The authors identify pretrained-model size as a potential limitation for achieving superior accuracy.

7 Conclusion

The paper presents activation prompting as an extension of input-level visual prompting that improves empirical performance and connects to normalization tuning. Experiments and theory examine its efficiency and layer preferences across CNNs and ViTs.

  • Conclusion: AP extends conventional input-level VP to improve empirical performance and establish a connection with normalization tuning.The paper positions AP as both an enhancement to VP and an analytical framework.
  • Conclusion: AP is more efficient and outperforms VP while achieving performance comparable to state-of-the-art PEFT methods.The conclusion highlights efficiency advantages alongside comparative accuracy.
  • Conclusion: The study investigates AP layer preferences on CNNs and ViTs through both empirical experiments and theoretical analysis.These preferences differ by architecture.

Checklist

The checklist reports that the paper documents its mathematical setup, theoretical assumptions and proofs, algorithmic complexity, efficiency analysis, and reproducibility materials.

  • The paper describes the mathematical formulation, settings, and assumptions in Sections 5–6 and Appendix A.
  • The paper analyzes algorithmic properties and complexity, including time, space, and sample size, and reports an efficiency analysis in Section 6.
  • The paper states that anonymized source code and its external-library dependencies are submitted in the supplementary material.
  • The paper includes complete proofs for its theoretical results, with proofs provided in Section C.
  • The paper states that code, data, instructions, training details, evaluation measures, error statistics, and computing infrastructure are included for reproducing experiments.
  • Existing assets are cited and their licenses are reported, while new assets and consent information are marked not applicable.

A Experiment Setting Details

The experiments cover 29 downstream image-classification datasets, with dataset attributes and training configurations documented in Table A1. Default implementations place AP near the end of ResNet and ViT-Large/16 backbones and use specified settings for LoRA, VPT, and optimization.

  • The study evaluates 29 downstream image-classification tasks across diverse domains.
  • Table A1 records dataset attributes and training configurations for the 29 target datasets.
  • AP is installed by default at the input of the third-to-last ResNet block and third Transformer block in ViT-Large/16.
  • LoRA uses rank r = 10, VPT uses prompt length 10, and all methods train for 1000 epochs with Adam.

B Additional Experiment Results

Additional experiments examine AP across architectures, layers, prompt variants, data sizes, and baselines. They consistently support AP’s architecture-dependent layer behavior, competitive efficiency, and advantage over VP across training-data regimes.

  • Additional architectures: AP works on CLIP and Swin-Transformer models, extending evaluation beyond the original vision-source setting.
  • Layer effects: AP’s layer effects generalize across multiple datasets, while CNNs and ViTs exhibit opposite layer preferences.
  • Prompt comparisons: In the VPT setting, AP averages 3% higher performance than VPT-Shallow and 3.5% lower performance than VPT-Deep, while occupying an intermediate efficiency position.
  • Prompt design and placement: The original additive AP outperforms product and concatenation variants, and adding AP to multiple layers does not significantly improve performance.
  • PEFT comparisons: AP remains competitive with PEFT baselines under a re-initialized classification head, while SSF ranks approximately second or third among PEFT methods.
  • Efficiency ablations: VPT-10 matches VPT-50 approximately while improving efficiency, and LoRA gains little from increasing rank beyond 10 despite substantially lower efficiency.
  • Data-size ablation: AP consistently outperforms VP from 10% to 100% training data on Camelyon, FOOD101, and CIFAR10, whereas Full-Finetune benefits most from larger datasets.
  • Architectural definitions: CNN and ViT architectures are defined through their respective convolutional or Transformer-layer compositions, with normalization, pooling, and output layers specified.

C.2 Proposition 1 and its proof

Proposition 1 establishes when activation prompting at a layer is equivalent to normalization tuning at that layer. The equivalence is stated separately for CNNs and ViTs under explicit normalization-parameter conditions.

  • Norm-Tune trains only Batch Normalization layers in CNNs or Layer Normalization layers in ViTs.
  • Proposition 1 states that AP on the l-th layer is the same as Norm-Tune on that layer under specified parameter conditions.
  • CNN condition: For CNNs, equivalence requires γ_d/σ_d = 1 and identical additive perturbations across batch elements, with β_d tied to the layer prompt.
  • ViT condition: For ViTs, equivalence requires γ_d/σ_b,p = 1, shared normalization means across batches, and β_d defined from the layer prompt and token-position mean.

C.3 Proof of Lemma 1

The proof analyzes a two-layer single-head ViT with discriminative and background patterns to track how attention and activation prompts alter feature locality. It shows layer-dependent changes in global and local feature behavior and derives zero-generalization-error conditions for training prompts.

  • Model formulation: The proof uses a two-layer single-head Transformer with self-attention, MLP, and fully connected output components.
  • Data model: The theoretical setup models noisy tokens containing discriminative patterns v1 and v2 and non-discriminative background patterns v3 and v4.
  • Pretrained model: The pretrained model represents all four patterns, while selected feed-forward neurons encode pattern representations across the two layers.
  • First-layer analysis: In the first layer, v1 and v4 patterns have strong self-correlation and are treated as global features, with attention distance determined by their token arrangement.
  • Prompted features: After the first layer, activation prompting makes v1 and v4 tokens local features because v4 contributes strongly to correlations with both patterns.
  • Training result: Training the first- or second-layer prompt with SGD returns zero generalization error under the stated noise, batch-size, and iteration conditions.

C.4.1 Proof of Lemma 2

The proof analyzes iterative updates to the perturbation δ under noisy patch configurations and derives conditions ensuring particular feature contributions emerge. It concludes that reaching the desired behavior requires a specified number of iterations.

  • Proof setup: The proof fixes attention weights and distinguishes δ(h) from its t-th update δ(t) while analyzing the network.The setup uses Θ(1)-scaled query, key, and value weights and tracks perturbation updates across iterations.
  • Gradient behavior: At t ≥ 2, data containing e2 and e3 contributes gradients that produce comparable amounts of e1 and e2 in δ(t).The same relationship is stated for the corresponding v1 and v2 features in the parallel analysis.
  • Activation behavior: With sufficiently large kµ, the analysis shows that no patches are activated by e4 with probability at least 1 − e^(−P^2).The condition is tied to Gaussian-noise bounds and the activation indicators for the relevant output weights.
  • Iteration bounds: For the first feature configuration, the proof requires t ≥ Θ(η^−1P^2(1 − σ)^−1 log P) iterations.This bound follows after establishing the needed condition for the update dynamics.
  • Iteration bounds: For the second configuration, the proof gives the sufficient bound t ≥ Θ(Pη^−1(1 − Pσ)^−1(1 + γ)^−1).The bound is stated after analyzing both yn = +1 and yn = −1 cases.
Loading 2604.06440v1…