PaperlayerView exact revision on arXiv

Source-linked AI summary

ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

Can Jin, Ying Li, Jingchen Sun, Hongwu Peng, Jiahui Zhao, Yang Zhou, Lei Li, Dimitris N. Metaxas

arXiv:2608.21194v1cs.CV

TL;DR

Existing visual prompting methods trade flexibility for efficiency by using either one prompt for all images or auxiliary networks to generate diverse prompts. ES-VP generates image-specific prompts through low-rank initialization and energy-guided, parameter-free adaptation, outperforming prior methods across architectures and datasets; with CLIP, it improves accuracy by 2.6% over DAM-VP while using 590× fewer VP parameters.

  • Problem

    Visual prompting lacks an approach that captures image-specific characteristics without the parameter costs and auxiliary networks used by diverse-prompt methods.

  • Method

    ES-VP combines universal low-rank prompt initialization with parameter-free, energy-guided dynamic adjustment generated directly from the pre-trained model.

  • Results

    Across five architectures and fifteen datasets, ES-VP consistently outperforms state-of-the-art Single-VP and Diverse-VP methods; with CLIP, it improves accuracy by 2.6% over DAM-VP while using 590× fewer VP parameters.

  • Takeaways & Limitations

    ES-VP establishes an efficient and generalizable visual-prompting approach that avoids auxiliary networks and supports simpler deployment.

Abstract

from arXiv · show

Visual prompting (VP) has emerged as a parameter-efficient method for adapting pre-trained models to downstream tasks. However, existing approaches encounter a trade-off between flexibility and efficiency. Some methods apply a fixed prompt to all images, ignoring individual image characteristics, while others introduce auxiliary networks to generate diverse prompts. Although the latter can improve performance, it also significantly increases parameter usage and the potential for overfitting to specific datasets. Furthermore, the auxiliary networks, combined with inherent biases in pre-trained models, limit scalability and generalization. In this paper, we propose Energy-Shaped Visual Prompting (ES-VP), a novel approach that generates image-specific prompts using low-rank initialization and energy-guided dynamic adaptation, achieving superior performance with fewer parameters compared to single-prompt methods. ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization. Extensive experiments conducted on five architectures across fifteen datasets demonstrate that ES-VP consistently outperforms current state-of-the-art (SOTA) single and diverse VP methods. For instance, using the CLIP architecture across four datasets, ES-VP outperforms the SOTA method DAM-VP by an average of 2.6\% in accuracy while utilizing 590$\times$ fewer VP parameters, thereby establishing a new benchmark for efficient and generalizable model adaptation.

1 Introduction

ES-VP generates image-specific visual prompts through low-rank initialization and energy-guided dynamic adaptation without auxiliary networks or additional VP parameters. Across five architectures and fifteen datasets, it consistently outperforms existing Single-VP and Diverse-VP methods, including DAM-VP.

  • Single-VP methods are parameter-efficient but use one universal prompt, limiting specificity and discriminative capability across variable images.Existing VP methods are categorized as Single-VP or Diverse-VP approaches.
  • ES-VP combines a universal low-rank prompt initialization with an image-specific, parameter-free dynamic adjustment derived from a label-independent energy function.The initialization captures global task characteristics and common dataset patterns, while the adjustment captures unique and discriminative image features.
  • 2.6% accuracy improvement over DAM-VP on four CLIP datasets was achieved with × 590 fewer VP parameters.Experiments across five architectures and fifteen datasets also showed consistent superiority over current SOTA Single-VP and Diverse-VP methods.
  • ES-VP demonstrates superior in-distribution and out-of-distribution generalization, while comprehensive analyses and ablations examine its effectiveness and efficiency.The method is designed to generate discriminative image-specific prompts with fewer parameters than existing Single-VP and Diverse-VP methods.

2 Related Works

This section situates visual prompting (VP) within prompting and transfer learning, describing how prompts adapt pre-trained models by modifying inputs or outputs. It also motivates parameter-efficient adaptation as conventional full fine-tuning becomes inefficient for larger models.

  • Visual Prompting: Prompting adapts pre-trained models to specific tasks by modifying their inputs.
  • Visual Prompting: Visual prompting, introduced for computer vision by [4], embeds adjustable parameters into visual inputs and uses input design plus output transformation.
  • Transfer Learning: Transfer learning reuses knowledge from pre-trained tasks for related applications in NLP and computer vision, including sentiment analysis, question answering, object detection, and segmentation [11].
  • Transfer Learning: Traditional transfer learning fully fine-tunes model parameters using pre-trained weights as initialization, but this becomes inefficient as models grow larger.

3 Methodology

ES-VP combines low-rank visual-prompt initialization with energy-guided, image-specific adaptation generated directly through a pre-trained model. Its meta-learning framework jointly refines per-image adaptation and a global low-rank prompt prior without auxiliary networks or additional parameters.

  • ES-VP Overview: ES-VP combines low-rank prompt initialization with energy-guided dynamic adaptation to generate image-specific visual prompts and improve generalization.The method directly uses the pre-trained model, avoiding auxiliary prompt-generation networks.
  • Visual Prompting: The visual prompt resizes and pads each input before adding a transformed tunable prompt, while output transformation maps source-model logits to downstream labels.The pre-trained model operates at L × L resolution, with experiments using 224 × 224 inputs.
  • Low-Rank Prompt Initialization: Low-rank matrices B and A initialize the prompt as δ = B · A, with zero initialization at training start and rank r controlling expressiveness.Setting r = L approximately recovers full-rank expressiveness, while r = 4 achieves performance comparable to full-rank visual prompts.
  • Energy-Shaped Dynamic Adaptation: ES-VP defines energy as the negative LogSumExp of transformed logits and adapts each prompt by following the negative energy gradient with learnable strength α.Energy-based modeling assigns lower scores to observed data and higher scores to unobserved data.
  • Optimization: A meta-learning framework jointly optimizes inner-loop image-specific adaptation and an outer-loop global low-rank prior, creating mutually reinforcing updates that support fast convergence.The adaptation requires neither labels, additional parameters, nor auxiliary networks, improving scalability relative to DAM-VP and SMM.

4 Experiment

The experiments evaluate ES-VP for in-distribution classification and out-of-distribution robustness across five architectures and fifteen datasets using standard visual-prompting protocols. The method combines a global low-rank prompt prior with energy-based, image-specific adaptation and is assessed against six representative state-of-the-art baselines.

  • Experimental Protocol: ES-VP is evaluated on ten small-scale in-distribution datasets and on ImageNet-1K after ImageNet-21K pre-training to assess OOD performance.The evaluation follows protocols used by AutoVP and SMM.
  • Method and Analysis: ES-VP generates prompted images by adding a global low-rank prior to resized inputs and applying energy-based dynamic adaptation with a learnable image-specific coefficient.The algorithm initializes δ = B · A with rank r ≪ L, computes μδ(x) = −α∂δ, and updates δ, α, and the output transformation through a meta-objective.
  • Networks: The study examines five 224 × 224 architectures, including ResNet-18, ResNet-50, ViT-B/32, Swin-B, and CLIP.The models use ImageNet-1K or ImageNet-21K pre-training and corresponding classifier or vision-language configurations.
  • Baselines: ES-VP is compared with six representative SOTA methods spanning Single-VP and Diverse-VP approaches, including ILM-VP, AutoVP, DAM-VP, SMM, LoR-VP, and LP.The baselines include methods using label mapping, dataset clustering, image-specific masks, global low-rank prompts, and output transformations.
  • Training and Evaluation: With LP [24] as the default output transformation and rank 4, ES-VP uses approximately 5K trainable parameters.Baseline results are reproduced from their original experimental configurations, while ES-VP hyperparameters are selected by grid search.

5 Main Results

ES-VP outperforms all baselines across networks and downstream datasets for in-distribution evaluation, while its energy-based dynamic prompting supports OOD robustness. Training curves further show consistently strong final accuracy across model–dataset combinations.

  • Training Performance: 8.0% higher accuracy than DAM-VP is achieved by ES-VP in the reported CLIP training comparison.ES-VP consistently achieves the best final accuracy across all stated model–dataset combinations.
  • In-distribution Performance: ES-VP outperforms all baselines across the evaluated networks and ten in-distribution downstream datasets.Experiments use ImageNet-1K pre-trained ResNet-18, ImageNet-21K pre-trained ViT-B/32, and ImageNet-1K fine-tuned ViT-B/32; results are reported in Table 1.
  • Out-of-distribution Performance: Energy-based dynamic visual prompts are evaluated for OOD robustness on four datasets using Swin-B further trained on ImageNet-1K.The evaluation uses an ImageNet-21K pre-trained Swin-B and compares ES-VP with the three strongest baseline methods.
  • Out-of-distribution Performance: These results highlight ES-VP’s robustness and potential advantages in complex, real-world applications.

6 Additional Investigation

Additional investigations show that ES-VP benefits from combining low-rank initialization with energy-based dynamic adaptation, performs best with LP output transformation, and improves discriminative power and efficiency while simplifying deployment.

  • Impact of Components in ES-VP: Combining low-rank prompt initialization with energy-based dynamic adaptation achieves the highest performance, while dynamic adaptation improves results with or without initialization.The ablation indicates that the two components are orthogonal and that dynamic adaptation contributes the larger improvement.
  • Impact of Output Transformations: ES-VP achieves its highest performance with LP as the output transformation and outperforms baseline methods even when using identical transformations.The comparison uses ImageNet-21K-pretrained ViT-B/32 on CIFAR100, with results summarized in Table 4.
  • Impact of the Energy-based Method on VP’s Discriminative Power: Energy-based dynamic VP adaptation significantly enhances discriminative capability by improving predicted probabilities for correctly classified images and distinguishing them from incorrect predictions.This analysis uses ImageNet-1K-pretrained ResNet-18 on CIFAR100 and compares predictions after each training epoch.
  • Impact of the Strength Coefficient α: An initial strength coefficient α of approximately 0.05 yields the highest accuracy for ES-VP on CIFAR100 with ImageNet-1K-pretrained ResNet-50.The comparison includes ES-VP without dynamic VP adaptation as a reference.
  • Efficiency of ES-VP: ES-VP exceeds DAM-VP by 5.3% in accuracy while using 1030× fewer VP parameters, with comparable inference latency and memory usage.It also avoids SMM’s auxiliary networks and DAM-VP’s extensive hyperparameter tuning and complex prompt designs.

7 Conclusion

ES-VP is an efficient adaptive energy-shaped visual prompting method that combines low-rank prompt initialization with energy-based dynamic adaptation to generate image-specific prompts. Experiments show superior generalization and parameter efficiency while avoiding auxiliary networks, extensive tuning, and multiple prompt designs.

  • 7 Conclusion: ES-VP combines low-rank prompt initialization with energy-based dynamic visual prompting adaptation to generate image-specific prompts.The method is designed to improve efficiency and effectiveness in adaptive visual prompting.
  • 7 Conclusion: ES-VP consistently achieves superior generalization and parameter efficiency versus state-of-the-art Single-VP and Diverse-VP approaches.Additional investigations support the effectiveness and training efficiency of its energy-based dynamic adaptation.
  • 7 Conclusion: By avoiding auxiliary networks, extensive hyperparameter tuning, and multiple visual-prompt designs, ES-VP improves practical scalability and deployment simplicity.
Loading 2608.21194v1…