Understanding the Multi-modal Prompts of the Pre-trained Vision-Language Model

• Prompts work in two ways: they adapt attention weights between the [cls] token and input tokens, and provide fixed features for updating tokens.Attention weights are assigned according to similarity between prompts and corresponding tokens.Source: §1 · p. 2
• Text-prompt contribution to alignment increases while category-token and tSOS contributions decrease; tSOS serves as a bias that prompts redirect toward dataset-specific bias.Source: §1 · p. 2
• Vision prompts do not significantly alter feature extraction, and their similarity to overlooked background features can bridge those features to the [cls] token’s attention.Source: §1 · p. 2
• Multi-modal prompts mainly improve recognition by learning dataset bias; bias tuning, which adds learnable bias to each transformer block, outperforms prompt tuning with the same number of parameters.Source: §1 · p. 2

2.1. Vision Language models

• Pre-trained vision-language models learn multi-modal representations from image-text pairs, but transferring them to downstream tasks without compromising their original generalization remains challenging.Source: §2.1 · p. 2

2.2. Prompt learning

• Prompt learning adapts pre-trained models by adding learnable input embeddings; prior work includes language, visual, conditioned, and hierarchical multi-modal prompts, while this paper builds on independent prompts in both encoders.Source: §2.2 · pp. 2–3

3.1. Vision-Language Pre-trained Model (VLM)

• The visual encoder converts image patches and a class token into a visual feature, while the text encoder processes template and class-label tokens into a textual feature.Source: §3.1 · p. 3
• Zero-shot inference predicts a category by matching image features with text-template features for the candidate class labels using cosine similarity and temperature.Source: §3.1 · p. 3

3.2. Multi-Prompt Learning for VLM

• Multi-prompt learning appends learnable prompts to both encoders, where prompts provide fixed features during token updates; the prompts are optimized for downstream image classification while pre-trained weights remain fixed.Source: §3.2 · p. 3

4.1. Experiment Setting

• Experiments use ViT-B/16 CLIP and a 16-shot setting across 11 image-recognition datasets, with 10 training epochs and no dataset-specific hyperparameter search.Source: §4.1 · pp. 3–4
• Figure 2 compares zero-shot and prompt-tuned token contribution profiles for OxfordPets examples misidentified by zero-shot CLIP but correctly identified by independent prompt tuning.The caption identifies token index as the horizontal coordinate and relative contribution as the vertical coordinate.Source: Figure 2 · p. 4

4.2.1 Attention Formulation

• Prompts affect self-attention by changing the attention weights between [cls] and input tokens.Source: §4.2.1 · p. 4
• Layer-specific prompts are fixed after learning, but their effects on tokens vary with token-prompt similarity and can alter later-layer attention.Source: §4.2.1 · p. 5
• The proposed mechanisms are improved [cls] focus on foreground objects and a learned bias term during token updates; the paper treats both as relevant.Source: §4.2.1 · pp. 5–6

4.2.2 Aligning Contribution Statistics

• The analysis constructs layerwise relevance maps by accumulating gradient-weighted attention contributions across heads and transformer layers.Source: §4.2.2 · p. 6
• Relevance is read from the vision [cls] and text tEOS rows, with normalized contribution statistics examined across eleven fine-grained recognition datasets.Source: §4.2.2 · pp. 6–7
• Text-branch prompt contributions to alignment increase significantly, while category-token and tSOS contributions decrease.Source: §4.2.2 · p. 7
• In the vision branch, prompts contribute prominently to alignment without significantly changing the relative relevance of input patches to [cls].Source: §4.2.2 · p. 7
• The alignment statistics align more closely with the proposed bias-update mechanism than with attention reweighting as the primary explanation.Source: §4.2.1 · pp. 5–6, §4.2.2 · p. 7

4.3. What do the multi-modal prompts learn?

• Because prompts occupy the same high-dimensional feature level as input tokens, the authors examine their attention and similarity to those tokens.The section explores text and vision prompts through attention and similarity.Source: §4.3 · p. 7, §4.3.2 · p. 8

4.3.1 Learned Textual prompts on Language Branch

• Text prompts resemble tSOS as fixed information preceding category tokens, but learn biases for the corresponding dataset rather than the entire language domain.In zero-shot CLIP, tEOS predominantly attends to tSOS, which cannot attend to subsequent category information.Source: §4.3.1 · p. 7
• Text prompts redirect tEOS attention from tSOS to dataset-specific prompts while attention to category tokens stays the same.The authors report that this shift enhances discriminative performance on the corresponding dataset.Source: §4.3.1 · p. 7

4.3.2 Vision prompts on Image Branch

• Vision prompts do not significantly change input-patch attention distance, indicating that they leave the pre-trained model’s feature extraction process largely intact.Figure 5 compares CLIP, W/O Prompt (excluding prompt attention distance), and W Prompt (retaining it).Source: Figure 5 · p. 7, §4.3.2 · p. 8
• Vision prompts do not significantly affect attention scope, as indicated by the attention entropy results.Higher entropy denotes a more uniform, global focus; lower entropy denotes a more extreme, local focus.Source: §4.3.2 · p. 8
• The [cls] token’s attention to input patches is not influenced by vision prompts in the ImageNet visualizations.Figure 6’s first two rows show original-CLIP and prompt-CLIP attention across encoder transformer blocks; its last row shows prompt similarity to input tokens.Source: §4.3.2 · p. 8, Figure 6 · p. 8
• Vision prompts resemble background features that the [cls] token did not attend to, and the authors suggest they can bridge attention to those features.The authors further propose that vision prompts, which are independent of input features, act as dataset bias.Source: §4.3.2 · pp. 8–9

4.4. Validation for the importance of the bias

• Bias tuning adds an independent learnable embedding to all tokens after each image- and language-branch transformer layer, then compares it with prompt tuning and other methods in the same few-shot setting.The comparison also includes zero-shot CLIP.Source: §4.4 · p. 9

5. Conclusion

• Across analyses of alignment relevance, attention, attention distance, attention entropy, and visualizations, the authors find that prompts mainly learn dataset bias.They propose bias tuning, which directly adds learnable bias to each transformer block, to validate this finding.Source: §5 · p. 9