FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Haoyang Li; Liang Wang; Siyu Zhou; Jiacheng Sun; Jing Jiang; Chao Wang; Guodong Long; Yan Peng

Source-linked AI summary

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Haoyang Li, Liang Wang, Siyu Zhou, Jiacheng Sun, Jing Jiang, Chao Wang, Guodong Long, Yan Peng

arXiv:2603.08708v1Mar 9, 2026cs.CV

TL;DR

Prompt tuning changes VLM internal attention, but existing methods give limited attention to foreground shifts and can face unreliable foreground views and base-new trade-offs. FVG-PT adds adaptive foreground reliability, attention compensation, and prior calibration; across four backbones and eleven datasets, it improves prompt tuning effectiveness and compatibility, while remaining unsuitable for some visual-only or costly distillation settings.

Problem
Prompt tuning can shift visual attention away from foregrounds, while foreground-view quality and the balance between base adaptation and new-class generalization remain insufficiently adaptive.
Method
FVG-PT uses a Foreground Reliability Gate, Foreground Distillation Compensation, and Prior Calibration to assess foreground quality, guide attention, and balance the CLIP prior.
Results
Across four backbone models and eleven datasets, FVG-PT consistently improves base adaptation and new-class generalization, with clear zero-shot gains on most target datasets.
Takeaways & Limitations
FVG-PT provides a plug-and-play enhancement compatible with diverse prompt-tuning backbones and cross-dataset transfer settings.
Takeaways & Limitations
FVG-PT is unsuitable for visual-only prompt tuning without a text branch, and foreground-view costs increase significantly for methods using all unlabeled dataset images.

Abstract

from arXiv · show

CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT

1 Introduction

Prompt tuning can shift visual attention away from image foregrounds because optimized text prompts reshape image-patch response weights. FVG-PT addresses this with adaptive foreground reliability, attention compensation, and prior calibration.

Prompt tuning methods often refine prompts or feature interactions while paying limited attention to changes in VLM internal attention representations.
Optimized text prompts directly reshape image-patch attention distributions, and Grad-CAM case studies examine inconsistent CLIP and CoOp predictions.
Foreground-guided methods face unreliable foreground views and a base-new trade-off caused by weakening background commonsense features.
FVG-PT uses a learnable Foreground Reliability Gate, Foreground Distillation Compensation, and Prior Calibration to adapt foreground trust, guide attention, and balance generalization.
FVG-PT is attached to four backbone models and evaluated on eleven datasets to assess effectiveness and compatibility.

2 Related Work

Related work establishes prompt tuning as a parameter-efficient way to adapt VLMs and highlights attention guidance and base-new generalization as central concerns. FVG-PT is positioned as an adaptive approach addressing both.

Prompt tuning adapts pretrained VLMs without full fine-tuning, reducing the computational cost associated with their large parameter scale and data requirements.
CLIP visual encoders aggregate image patches through self-attention, making visual attention representations relevant to image-text similarity classification.
Existing attention-guidance methods use gradients or visual cues, but generally lack explicit evaluation or control of foreground-view quality.
The base-new trade-off reflects reduced new-class generalization when prompts overfit target base classes.
FVG-PT uses Prior Calibration to decouple foreground-focused adaptation and balance it against the CLIP prior.

3 Proposed Method

FVG-PT adds adaptive foreground guidance to prompt-tuned VLMs through reliability-based foreground selection, foreground distillation, and prior calibration. Its decoupled branches separately support foreground-focused base adaptation and calibrated new-class inference.

Foreground Reliability Gate: FVG-PT extracts a segmentation-based foreground view and uses a learnable Foreground Reliability Gate to adaptively weight the foreground and full image.The gate is trained from whether the foreground view yields lower classification loss than the full image.
Foreground Reliability Gate: The Foreground Reliability Gate combines entropy, distribution similarity, and area-ratio indicators to assess foreground quality and predict a trust score.The entropy difference favors sharper foreground predictions, similarity limits distributional shift, and area ratio prevents insufficiently informative regions.
Foreground Distillation Compensation: Foreground Distillation Compensation inserts residual bottleneck adapters into both visual and textual branches to re-project features toward foreground-oriented attention.It produces compensated logits while preserving cross-modal alignment through separate adapters in the two branches.
Foreground Distillation Compensation: FDC distills toward the foreground distribution in proportion to foreground reliability, otherwise interpolating toward the full-image distribution.The distillation objective uses r to select between p_fg and p_full as adaptive targets for p_FDC.
Prior Calibration: Prior Calibration separates base and new branches at the logit level, preventing the new branch from using FDC and reducing branch interference.The base branch uses FDC logits, while the new branch combines full-image backbone logits with zero-shot CLIP logits through a learned trust weight b.
Prior Calibration: The Backbone Reliability Gate adaptively controls the CLIP-prior contribution in the new branch using three logit-based statistics.Sharper CLIP-prior distributions encourage a larger b, increasing reliance on z_CLIP in the calibrated logits.

4 Experiments

FVG-PT is evaluated across four prompt-tuning backbones and eleven datasets using base-to-new generalization, cross-dataset transfer, ablations, and efficiency analyses. Results show consistent gains across adaptation and generalization settings, while component studies support the roles of foreground guidance, reliability control, and prior calibration.

Base-to-New Generalization: FVG-PT improves both base-class adaptation and new-class generalization across four backbones and eleven datasets.The evaluation uses the same base-class training data as the corresponding backbones and reports base-to-new performance.
Cross-Dataset Transfer: FVG-PT improves performance on the ImageNet source and achieves clear zero-shot gains on most target datasets in cross-dataset transfer.The authors attribute the target-dataset gains to the weight allocation scheme learned by Prior Calibration.
Comparison with Related Methods: FVG-PT achieves better overall HM performance than ProGrad and DAPT-S among visual-attention prompt-tuning methods.The comparison supports explicit foreground supervision and more reliable guidance toward correct foreground regions.
Validity of Proposed Components: FDC improves base performance, whereas adding Prior Calibration produces gains on new classes and mitigates the base-new trade-off.The PC-only configuration also decouples the base and new branches, reducing interference from base-class fine-tuning on generalization.
Impact of FDC Distillation Loss: FVG-PT reaches its best HM performance at λd = 10, while λd = 1 still significantly outperforms the CoOp backbone.This indicates that adaptive behavior does not require tuning the distillation-loss weight precisely, although the reported performance can incur a small cost.
Computational Cost: With ViT-B/16 encoders, FVG-PT activates 0.13M trainable parameters and improves parameter count, memory cost, and processing FPS relative to the backbone.The reported computational comparison is based on fine-tuning the MMRL backbone on Flowers102.

5 Conclusion

FVG-PT uses explicit foreground supervision and adaptive reliability mechanisms to correct visual-attention shifts while preserving compatibility across prompt-tuning backbones. Experiments support its robustness, but its applicability is limited for visual-only methods and some unlabeled-image distillation settings.

FVG-PT guides visual attention with Foreground Distillation Compensation and maintains foreground-view quality through a Foreground Reliability Gate.Prior Calibration decouples the new branch from the base and balances it with the CLIP prior through a Backbone Reliability Gate.
Experiments confirm compatibility with diverse backbones and consistency across hyperparameter settings.These findings are presented as evidence of FVG-PT's adaptive behavior.
FVG-PT is unsuitable for visual-only prompt-tuning methods without a text branch, such as VPT.
Foreground-view extraction becomes significantly more costly for knowledge-distillation methods that use all unlabeled images in the dataset.

Supplementary Material

The supplementary material extends the paper with quantitative analyses, implementation details, additional experiments, and failure cases concerning foreground attention shifts and FVG-PT's limitations.

The supplementary material includes quantitative analysis and additional case studies of foreground attention shifts.
Appendix B provides implementation details for the SEEM segmentation model and backbones to support reproducibility.
Appendix C adds error-bar analysis, data-efficiency analysis, and more detailed ablations.
Appendix D presents bad-case studies that further analyze FVG-PT's limitations and potential directions for improvement.

A.1 Quantitative Analysis

The quantitative analysis measures foreground reliance using a foreground shift index and compares it with base-class performance across CLIP, CoOp, and FVG-PT. The reported trend is that stronger base performance accompanies a lower foreground shift index.

The foreground shift index is evaluated against target-task performance on the entire test set of the datasets.
Fig. 5 reports foreground shift index and base-class accuracy on Caltech101 and Flowers102.
Figures 6 and 7 provide additional attention-map failure cases for CLIP, CoOp, and FVG-PT.The figures use Grad-CAM-generated visual-encoder attention maps.
The index uses D_fg = KL(p_full∥p_fg), where smaller divergence indicates more consistent full-image and foreground-view predictions.
Base performance increases while the foreground shift index decreases across foundation CLIP, fine-tuned CoOp, and FVG-PT.

A.2 More Visualization Results

Additional Grad-CAM visualizations compare cases where CLIP and CoOp succeed or fail, showing that misclassified examples tend to involve attention shifting away from the foreground. FVG-PT is motivated as a correction for these shifts.

Fig. 6 shows examples where CLIP misclassifies while CoOp succeeds, whereas Fig. 7 shows cases where CoOp fails but CLIP predicts correctly.
Across the visualized cases, misclassified examples tend to show visual-encoder attention shifting away from the foreground.
These visualizations motivate FVG-PT's foreground-attention guidance as a way to correct such shifts and improve prediction accuracy.

B.1 Experimental Setup

FVG-PT is evaluated as a plug-and-play module across four prompt-tuning backbones using ViT-B/16 CLIP, shared data, and matched initialization. Experiments include robustness, baseline comparisons, ablations, and foreground-mask generation with SEEM.

Backbone and evaluation setup: FVG-PT is evaluated on four prompt-tuning backbones built on ViT-B/16 CLIP with identical fine-tuning data and initialization parameters.The backbones are CoOp, KgCoOp, PromptSRC, and MMRL.
Foreground-mask generation: SEEM provides optional pixel-level foreground masks, using textual prompts as queries for semantic segmentation.The masks support the foreground-guided components of FVG-PT.
Backbone integration: FVG-PT integrates with CoOp, KgCoOp, PromptSRC, and MMRL by attaching or adapting its foreground-guided modules to each backbone’s feature-processing pipeline.MMRL retains its original mid-layer processing and applies additional foreground-guided fine-tuning after image-text interaction.
Evaluation protocol: The study reports base-to-new performance, error-bar robustness across matched random seeds, comparisons with additional prompt-tuning baselines, and loss-level ablations.Each dataset uses three runs, and the robustness analysis reports mean performance and standard deviation.
Baseline comparison: FVG-PT combined with MMRL achieves higher performance than the listed mainstream prompt-tuning baselines.The comparison includes CoCoOp, ProDA, MaPLe, KAPT, KDPL, CLIP-Adapter, TCP, and MMA.

C.3 Detailed Ablation Study

Detailed ablations examine FVG-PT’s losses, adapter branches, and CLIP encoder scale. The results support contributions from all five losses, dual-branch adapters, and stronger gains with larger encoders, especially for new classes.

Loss ablations: Removing any of FVG-PT’s five losses prevents it from reaching best performance, while the PC KL loss most affects new-class accuracy.The KL loss aligns the weighted distribution p_PC with the CLIP prior p_CLIP.
Adapter branches: Using adapters in both visual and textual branches improves base performance over a visual-only adapter configuration.The reported interpretation is that dual-branch adapters help maintain cross-modal alignment during fine-tuning.
Encoder scaling: FVG-PT improves over CoOp with both ViT-B/16 and ViT-L/14 foundation encoders, with a larger new-class boost under ViT-L/14.The result suggests that a stronger CLIP prior supports better new-class generalization.

C.4 Data Efficiency Analysis

FVG-PT shows improved data efficiency in base-to-new tasks, matching CoOp’s full-data HM performance with half the training samples.

Data efficiency: FVG-PT matches CoOp’s full-data HM performance when trained with 50% of the training samples.The comparison uses matched CoOp and FVG-PT subsets across training proportions of 25%, 50%, 75%, and 100%.

D Failure Case Study

FVG-PT+CoOp has a severe EuroSAT base-performance failure under the default setting because the small dataset underfits the FDC adapters. Increasing training to 20 epochs mitigates this degradation.

Failure case: 88.43 to 72.62 (−15.81 points) is the EuroSAT base-performance drop for FVG-PT+CoOp under the default configuration.The base set has five classes and only 80 image-text pairs in the 16-shot setting, fewer than the FDC adapters’ 0.06M learnable parameters.
Mitigation: With ep = 20, FVG-PT exceeds CoOp on EuroSAT base performance and improves new-class generalization, yielding a 10.27-point HM gain over CoOp.The main experiments therefore use 20 epochs for EuroSAT with CoOp and FVG-PT.