ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

1 Introduction

• Prompt learning adapts a small set of context tokens, avoiding costly full-backbone fine-tuning that can destroy CLIP’s transferable knowledge.Low-shot prompts may overfit incidental cues, while shallow or one-way links can limit visual refinement of text semantics.Source: §1 · pp. 1–2
• Gaussian prompt parameterization with KL/L2 penalties and a symmetric InfoNCE head are used to stabilize low-shot adaptation and align image features with class-level text representations.The InfoNCE head uses cross-attended image features; the KL/L2 penalties apply to prompt parameters.Source: §1 · pp. 2–3
• ProCAP links learned visual and textual prompts with stacked bidirectional multi-head cross-attention across prompt depth while keeping CLIP frozen.The two branches can refine one another throughout training.Source: §1 · p. 2
• In base-to-novel transfer, ProCAP improves the harmonic-mean balance over CoOp, CoCoOp, and MaPLe on several challenging benchmarks.Figure 1b provides a cross-benchmark view using HM, which balances base- and novel-class performance.Source: §1 · p. 2, Figure 1 · p. 3
• Across 11 standard image-classification benchmarks, ProCAP shows strong aggregate performance relative to MaPLe-style multimodal prompting and recent prompt-learning baselines.The reported evaluation also includes few-shot and base-to-novel benchmarks.Source: §1 · p. 3

2 Related Work

• Prompt learning tunes a small set of context embeddings rather than all model weights, reducing computation and helping avoid catastrophic forgetting.The literature describes retaining a CLIP-style contrastive objective while adding task-specific prompt modules as a robust adaptation recipe.Source: §2 · pp. 3–4
• Follow-up approaches explore multimodal cues, attribute structure, language-model guidance, input- or dataset-specialized prompts, consistency, style robustness, and unlabeled-data learning.Structured or adaptive prompting also targets stronger image–text alignment.Source: §2 · p. 3
• VAMP uses sample-specific variational prompting with inference-time sampling, whereas DPC uses dual-prompt collaboration to decouple base- and new-class optimization.ProCAP instead jointly optimizes bidirectional visual–text interaction, Gaussian prompt regularization, and symmetric image–class alignment in the standard frozen-CLIP setting, without external descriptions, test-time sampling, or a teacher/plugin framework.Source: §2 · p. 4
• Recent methods study hierarchical adapters, diversified composite prompts, codebook-selected input-adaptive prompts, and low-rank multimodal prompts; ProCAP is complementary to these approaches.Its central mechanism is stacked bidirectional cross-attention between probabilistic visual and textual prompts, jointly optimized with prompt-space regularization and symmetric image–class contrastive alignment.Source: §2 · p. 4

3 Method

• ProCAP keeps the CLIP encoders frozen and updates prompts, cross-attention modules, and lightweight projection heads.Its training objective combines classification, symmetric InfoNCE, KL, and L2 losses.Source: §3 · p. 4, §3.4 · p. 7

3.1 Preliminaries: CLIP and Multimodal Prompting

• CLIP encodes class-name prompts into text features that act as classifier weights, while prompt learning optimizes context embeddings using image-to-class cross-entropy.MaPLe-style prompting adds visual prompts and couples visual and textual prompts through a shared projection space.Source: §3.1 · p. 4

3.2 ProCAP: Cross-Attentive Multimodal Prompts

• ProCAP learns visual and textual prompts and repeatedly refines both streams through stacked bidirectional cross-attention across prompt depth.Visual prompts are prepended to image patch tokens and textual prompts to the class prompt; residual updates preserve each stream while adding cross-modal information.Source: §3.2 · p. 5

3.3 Gaussian Prompt Regularization

• Gaussian means and variances parameterize prompt tokens, with KL and L2 regularization encouraging them to remain near a simple prior unless supported by data.At inference, ProCAP uses prompt means without sampling, making the model deterministic.Source: §3.3 · pp. 5–6

3.4 Symmetric InfoNCE Head over Image and Class Representations

• A symmetric InfoNCE head aligns cross-attended image features with class-level text representations in a shared low-dimensional space, complementing classification.Separate projection heads map image and class text features into that space before similarity-based contrastive training.Source: §3.4 · pp. 6–7

4.1 Experimental Setup

• Evaluation covers 11 datasets, with 16-shot base-class training for base-to-novel tests, ImageNet-trained cross-dataset transfer, and four unsupervised ImageNet shifts.The base-to-novel protocol evaluates both base and novel test classes; cross-dataset evaluation uses 10 target datasets without target fine-tuning.Source: §4.1 · pp. 7–8

4.2 Main Results

• 81.33 aggregate average HM; ProCAP also records 70.02 average accuracy across 10 targets and 72.14 source / 60.58 target-mean Top-1 on ImageNet shifts.Table 1 reports base-to-novel results, Table 3 cross-dataset transfer, and Table 2 domain generalization.Source: §4.2 · pp. 8–9, Table 1 · p. 9, Table 3 · p. 10

4.3 Few-shot Evaluation

• Across K=1, 2, 4, 8, and 16 shots, accuracy rises with supervision, while ProCAP’s clearest advantages occur at 1–4 shots and narrow as K increases.ProCAP remains among the top performers as the number of labeled examples increases.Source: §4.3 · p. 10, Figure 3 · p. 11

4.4 Ablation Study

• Removing any component reduces average performance, with the largest drop after removing Gaussian prompt modeling, followed by InfoNCE and cross-attention.Ablations cover representative texture, remote-sensing, and fine-grained datasets.Source: §4.4 · p. 11, Table 4 · p. 11
• Base/novel accuracy and HM peak at Nctx=4; longer prompts tend to overfit and reduce HM, so experiments use four context tokens.The prompt-length analysis averages results over EuroSAT, Caltech101, OxfordPets, and UCF101.Source: §4.4 · p. 11, Figure 4 · p. 12
• HM remains stable across broad sweeps of λ, β, and τp; the selected settings are λ=0.01, β=10^-5, and τp=0.07.Figure 5 examines the contrastive-loss weight, uncertainty-regularization weight, and InfoNCE temperature initialization.Source: §4.4 · p. 11, Figure 5 · p. 12

5 Conclusion

• ProCAP couples learned visual and textual prompts through stacked bidirectional multi-head cross-attention across prompt depth while keeping CLIP frozen.Gaussian prompt parameters receive lightweight KL/L2 regularization, and a compact symmetric InfoNCE head aligns cross-attended image features with class-level text representations.Source: §5 · p. 12
• Across 11 datasets, cross-dataset transfer, and ImageNet shift benchmarks, ProCAP shows strong aggregate base-to-novel performance and competitive transfer performance.The method updates prompts and lightweight components rather than the CLIP backbone.Source: §5 · p. 12
• Future work will explore more efficient cross-attention designs and scaling to larger vision–language backbones.Source: §5 · p. 12

• ProCAP is a general-purpose prompt-learning method for image classification and introduces no new data collection or human-subject experiments.Source: Broader Impact · p. 12
• Its predictions may inherit biases, dataset imbalances, and representation limitations from pretrained CLIP and evaluation benchmarks.Source: Broader Impact · p. 12
• High-stakes deployment may amplify errors or unequal performance, so the method should not be the sole basis for safety-critical decisions.Suggested mitigations include application-specific dataset auditing, subgroup evaluation, uncertainty calibration, and human oversight.Source: Broader Impact · p. 12

• The appendix includes qualitative comparisons, three-seed stability analysis, and runtime measurements.Source: A Additional Experimental Results · p. 15
• It also provides reproducibility details and dataset statistics.Source: A Additional Experimental Results · p. 15
• Prompt templates used in the experiments are included.Source: A Additional Experimental Results · p. 15

A.1 Qualitative Comparison

• On selected difficult Oxford Flowers examples, ProCAP more often predicts the correct class and concentrates on discriminative regions than MaPLe.The prediction comparison uses input, ground truth, MaPLe prediction, and ProCAP prediction columns; green and red mark correct and incorrect predictions.Source: §A.1 · p. 15, Figure 6 · p. 16
• EigenCAM comparisons show where each model concentrates evidence, with warmer regions indicating stronger contributions to predictions.The analyses use identical samples and CAM extraction and rendering settings for both models.Source: §A.1 · p. 15, Figure 7 · p. 17
• 98.43% base accuracy for ProCAP versus 96.86% for the trained baseline on EuroSAT base classes.Source: Figure 8 · p. 17
• Figure 8 shows structured learned prompt patterns, with ProCAP learning a distinct pattern from the baseline.The figure includes random initialization, the trained baseline, and ProCAP on EuroSAT base classes.Source: §A.1 · p. 17, Figure 8 · p. 17

A.2 Stability Analysis

• Table 5 reports three-seed variability across all 11 base-to-novel benchmarks.Source: §A.2 · p. 18, Table 5 · p. 18
• Table 5 presents stability results as mean ± standard deviation.Source: Table 5 · p. 18
• The average-row HM is the arithmetic mean of dataset-wise HMs, whereas Table 1 reports aggregate HM from averaged Base and Novel accuracy.Source: Table 5 · p. 18

A.3 Runtime and Training Analysis

• Runtime was benchmarked over 10 epochs on one NVIDIA A100 GPU with AMP.All methods used the same environment, preprocessing, and evaluation conditions.Source: Table 6 · p. 18
• ProCAP is slower in training and inference but provides clear accuracy gains in the Table 6 comparison.The table compares two representative datasets using CLIP ViT-B/16 in the same 16-shot setting.Source: §A.3 · p. 18, Table 6 · p. 18
• At 50 epochs, ProCAP obtains its best aggregate HM while longer training primarily improves base performance and preserves novel-class transfer.Table 7 reports base-to-novel results averaged over 11 datasets.Source: §A.3 · p. 18, Table 7 · p. 18

A.4 Implementation Details and Reproducibility

• The additional experiments use the main paper’s training and evaluation pipeline unless otherwise stated.All experiments use a shared codebase and pipeline.Source: §A.4 · p. 19

A.4.1 Training Setup

• Training freezes CLIP ViT-B/16 and updates only prompt-learning parameters and ProCAP’s additional learnable components.The main setup uses AdamW for 50 epochs, batch size 32 for training and 100 for evaluation, and learning rate 2.5×10−4.Source: §A.4.1 · p. 19

A.4.2 Evaluation Protocol

• Base-to-novel evaluation reports base accuracy, novel accuracy, and HM; domain-generalization prompts transfer from ImageNet without target supervision.Evaluation uses deterministic preprocessing without horizontal-flip test-time augmentation.Source: §A.4.2 · p. 19

A.4.3 Qualitative Visualization Protocol

• Figures 6 and 7 use the same Oxford Flowers samples and identical EigenCAM extraction and rendering settings for MaPLe and ProCAP.These controls isolate model behavior rather than visualization differences.Source: §A.4.3 · p. 19

A.4.4 Implementation Environment

• Experiments use PyTorch on NVIDIA GPUs with the same codebase, backbone initialization, training framework, and evaluation pipeline as the main experiments.Source: §A.4.4 · p. 19

A.5 Dataset Details

• The reported prompt-learning counts follow fixed CoOp/MaPLe benchmark splits and may differ from full raw-collection sizes.Table 8 summarizes the evaluation datasets and benchmark split sizes.Source: §A.5 · p. 19, Table 8 · p. 20

A.6 Prompt Templates

• The appendix lists representative configured templates, but systematic template sensitivity remains future work.The placeholder is replaced by the class name, and the || delimiter is removed before CLIP tokenization.Source: §A.6 · p. 20, Table 9 · p. 20