PaperlayerView exact revision on arXiv

Source-linked AI summary

MaPLe: Multi-modal Prompt Learning

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan

arXiv:2210.03117v3cs.CV

TL;DR

Single-branch prompt learning does not flexibly adapt both representation spaces of vision-language models for downstream tasks. MaPLe jointly learns coupled prompts in the vision and language branches, with separate prompts across stages, and outperforms Co-CoOp on novel-class and harmonic-mean performance across 11 datasets.

  • Problem

    Single-branch prompting adapts only the language or vision representation space, limiting flexible downstream adaptation of vision-language models.

  • Method

    MaPLe jointly learns language and vision context prompts, conditions vision prompts on language prompts, and uses deep prompts across transformer blocks.

  • Results

    3.45% absolute average gain on novel classes and 2.72% on harmonic-mean over Co-CoOp across 11 diverse image recognition datasets.

  • Takeaways & Limitations

    MaPLe improves generalization to novel categories, cross-dataset transfer, and datasets with domain shifts.

Abstract

from arXiv · show

Pre-trained vision-language (V-L) models such as CLIP have shown excellent generalization ability to downstream tasks. However, they are sensitive to the choice of input text prompts and require careful selection of prompt templates to perform well. Inspired by the Natural Language Processing (NLP) literature, recent CLIP adaptation approaches learn prompts as the textual inputs to fine-tune CLIP for downstream tasks. We note that using prompting to adapt representations in a single branch of CLIP (language or vision) is sub-optimal since it does not allow the flexibility to dynamically adjust both representation spaces on a downstream task. In this work, we propose Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. Our design promotes strong coupling between the vision-language prompts to ensure mutual synergy and discourages learning independent uni-modal solutions. Further, we learn separate prompts across different early stages to progressively model the stage-wise feature relationships to allow rich context learning. We evaluate the effectiveness of our approach on three representative tasks of generalization to novel classes, new target datasets and unseen domain shifts. Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable performance and achieves an absolute gain of 3.45% on novel classes and 2.72% on overall harmonic-mean, averaged over 11 diverse image recognition datasets. Our code and pre-trained models are available at https://github.com/muzairkhattak/multimodal-prompt-learning.

1. Introduction

CLIP prompt learning traditionally adapts one branch, limiting how downstream tasks can adjust aligned vision-language representations. MaPLe jointly learns coupled, hierarchical prompts in both branches and improves generalization across diverse evaluation settings.

  • Motivation: CLIP prompt learning avoids full fine-tuning by learning prompts while keeping the original model weights frozen.This reduces the risk of forgetting pretrained knowledge and overfitting when downstream data are limited.
  • Motivation: Existing approaches primarily learn prompts in only the language or vision branch, adapting CLIP partially.The paper argues that single-branch prompting cannot fully model adaptations needed by both encoders.
  • MaPLe: MaPLe jointly adapts language and vision representations to improve their downstream alignment.It uses a coupling function to condition vision prompts on language prompts, enabling mutual gradient propagation and synergy.
  • Results: 3.45% absolute average gain on novel classes and 2.72% on harmonic-mean over Co-CoOp across 11 diverse image recognition datasets.The evaluation covers base-to-novel generalization, cross-dataset transfer, and domain generalization.
  • Contribution: MaPLe is introduced as multimodal prompt learning for CLIP and is reported as the first such approach for fine-tuning the model.The method is designed to align the vision-language representations rather than learn independent unimodal solutions.
  • MaPLe: MaPLe learns separate multimodal prompts across multiple transformer blocks to progressively model stage-wise contextual relationships.Deep prompting provides more flexibility for aligning vision-language representations.

2. Related Work

Vision-language models learn joint image-language representations from large-scale web data and support broad visual recognition tasks. Prior prompt-learning methods mainly adapt one modality, whereas MaPLe jointly prompts both branches.

  • Vision Language Models: Vision-language models combine language supervision with natural images to encode rich multimodal representations.Models such as CLIP, ALIGN, LiT, FILIP, and Florence are trained self-supervised on abundant web data.
  • Vision Language Models: CLIP and related vision-language models have demonstrated strong performance in few-shot and zero-shot visual recognition.CLIP and ALIGN use approximately 400M and 1B image-text pairs, respectively, for training.
  • Prompt Learning: Prompt learning adapts models by optimizing prompt tokens instead of manually designing task-specific text templates.The approach originated in NLP and was later applied to vision-language and vision-only models.
  • MaPLe: MaPLe differs by learning context prompts in both branches and explicitly coupling vision prompts to language prompts.This design targets interaction between modalities rather than partial adaptation of CLIP.
  • Prompt Learning: Prior vision-language prompt-learning methods generally learn prompts independently in either the language or vision branch.CoOp and Co-CoOp focus on language prompts, while other work explores vision prompting.

3. Method

MaPLe adapts CLIP by jointly learning coupled prompts in its vision and language branches, with prompts introduced across early transformer stages. This design aims to improve cross-modal synergy and progressively model stage-wise feature relationships.

  • Multimodal prompting: MaPLe fine-tunes CLIP through context optimization with prompts learned in both language and vision branches.Unlike prior approaches that prompt only the language branch, MaPLe jointly adapts both representation spaces.
  • CLIP representation pipeline: The image and text encoders project their final representations into a shared vision-language latent space for alignment.The image representation uses the final class token, while the text representation uses the final token from the last text-transformer block.
  • Hierarchical prompting: MaPLe introduces learnable prompts in the first J transformer layers of both branches to progressively model stage-wise feature representations.After the prompted depth, subsequent layers process the resulting representations through the remaining transformer blocks.
  • Hierarchical prompting: MaPLe's deep prompting supports feature-hierarchy adaptation, while shared prompts across stages exploit correlations created by successive transformer blocks.The method argues that independent later-stage prompts provide less complementary information than shared prompts.
  • Vision-language prompt coupling: Explicit prompt coupling addresses the lack of interaction in independent vision-language prompting by learning prompts in a shared embedding space.The paper contrasts this with independently learning vision and language prompts, which do not interact during task-context learning.
  • Vision-language prompt coupling: Vision prompts are conditioned on language prompts through a learned projection, coupling the two modalities during optimization.The projection maps language-prompt dimensions to vision-prompt dimensions and encourages mutual gradient propagation.

4. Experiments

MaPLe is evaluated across base-to-novel generalization, cross-dataset transfer, domain generalization, ablations, and computational complexity. It consistently improves generalization over competing prompt-learning methods while maintaining low computational overhead.

  • Base-to-Novel Generalization: MaPLe improves both base- and novel-class performance over Co-CoOp across 11 datasets, except for a marginal base-class reduction on Caltech101.
  • Base-to-Novel Generalization: 2.72% is MaPLe’s absolute average gain over Co-CoOp when base and novel classes are combined.
  • Base-to-Novel Generalization: MaPLe improves over CoOp on base classes in 6/11 datasets while maintaining average base accuracy around 82.28%.
  • Cross-Dataset Evaluation: 66.30% is MaPLe’s highest averaged accuracy in cross-dataset evaluation, surpassing CoOp in 9/10 and Co-CoOp in 8/10 datasets.
  • Domain Generalization: MaPLe consistently improves against existing approaches on out-of-domain target datasets, supporting stronger domain generalization and robustness.
  • Ablation Experiments: MaPLe adds only 0.1% FLOPS over CoOp and Co-CoOp, uses half as many training epochs as Co-CoOp, and adds about 2.85% training parameters to CLIP.

5. Conclusion

MaPLe adapts both vision and language branches of V-L models, explicitly coupling their prompts across transformer stages. It improves generalization toward novel categories, cross-dataset transfer, and datasets with domain shifts.

  • 5. Conclusion: MaPLe performs prompting for both vision and language branches to appropriately adapt V-L models to downstream tasks.Its vision prompts are explicitly conditioned on textual prompts across different transformer stages.
  • 5. Conclusion: MaPLe explicitly conditions vision prompts on textual prompts across different transformer stages to ensure synergy between modalities.
  • 5. Conclusion: MaPLe improves generalization toward novel categories, cross-dataset transfer, and datasets with domain shifts.

Supplementary Material

The supplementary material provides additional implementation details, alternate prompting designs, analyses of multi-modal prompts, and comparisons with Co-CoOp.

  • Supplementary Material: The supplementary section provides additional implementation details for the main paper.
  • Supplementary Material: It analyzes alternate prompting design choices and multi-modal prompts.
  • Supplementary Material: It compares MaPLe with the heavier Co-CoOp method.

A. Additional Implementation details

The implementation details specify hyper-parameters for deep prompting variants and the Co-CoOp training configuration used for CoOp†.

  • A. Additional Implementation details: The deep-prompting variants use a learning rate of 0.0025 for language and vision prompting and 0.0035 for independent V-L prompting.
  • A. Additional Implementation details: The hyper-parameter table distinguishes independent V-L prompting and identifies K as prompt depth.It denotes vision and language prompt tokens as ˜P and P, respectively.
  • A. Additional Implementation details: CoOp† follows Co-CoOp training configurations, using 10 epochs, batch size 4, and a learning rate of 0.0035.

B. Alternate Design Choices

The supplementary analyses examine prompt initialization, projection direction, and alternative multi-modal prompting designs. They report that MaPLe’s design choices affect performance and that its selected configuration performs best among compared alternatives.

  • B. Alternate Design Choices: Best MaPLe performance uses ‘a photo of a <category>’ initialization in the first layer and random initialization in the remaining layers.
  • B. Alternate Design Choices: MaPLe performance is affected by the choice of prompt initialization.
  • B. Alternate Design Choices: MaPLe conditions vision prompts ˜P on language prompts P through a V-L coupling function F.
  • B. Alternate Design Choices: The reported P →˜P projection direction performs better than conditioning language prompts on vision prompts, attributed to lower information loss.The passage relates this choice to the larger dimension of vision prompts, d_v, than language prompts, d_l.
  • B. Alternate Design Choices: MaPLe provides the best performance among the compared alternate prompting designs.The comparisons include progressive prompting, independent V-L prompting, and their combination.

C. Understanding Multi-modal Prompts

MaPLe’s gains over Co-CoOp vary by dataset, becoming larger as distributional diversity and category rarity increase. Its multimodal prompting also yields better text-category separability, while alternative prompting designs perform worse overall.

  • Dataset and category effects: +13% over Co-CoOp on EuroSAT, FGVCAircrafts, and DTD contrasts with gains below 1% on ImageNet and Caltech101.The reported dataset-dependent differences motivate analyzing when MaPLe is most effective.
  • Design analysis: Projecting from P to ˜P provides the best results, and MaPLe is the strongest variant among the alternative V-L prompting designs.These conclusions are stated in the corresponding design-analysis tables.
  • Dataset and category effects: MaPLe becomes more effective than Co-CoOp as the dataset’s distributional gap from CLIP pretraining increases.EuroSAT and DTD are cited as examples with larger gaps, whereas datasets with little distribution shift show less substantial gains.
  • Dataset and category effects: MaPLe provides larger gains for rare, less-generic concepts, while Co-CoOp remains competitive on frequent categories such as forest, river, and dog.Satellite images are given as an example of a rare visual concept.
  • Embedding analysis: MaPLe produces better separability among text categories according to pairwise cosine similarity and normalized l2 distance analyses.The analysis averages these metrics across category text embeddings.

D. Comparing MaPLe with Heavier Co-CoOp

The comparison tests whether MaPLe’s performance gain can be attributed simply to its larger number of learnable parameters. It therefore evaluates a shallow MaPLe variant and a retrained heavier Co-CoOp model with comparable parameter counts.

  • Parameter-count comparison: MaPLe shallow (J = 1) uses prompts only at the first vision and language layers to test gains with fewer prompting stages.This comparison is designed to address MaPLe’s greater parameter count relative to CoOp and Co-CoOp.
  • Parameter-count comparison: A heavier Co-CoOp model is retrained to match MaPLe’s total parameter count, while MaPLe† uses one shared V-L coupling function across all layers.The table compares parameter-matched alternatives and a shared-coupling MaPLe variant.
Loading 2210.03117v3…