PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain of Thought Prompt Tuning in Vision Language Models

Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu, Shanghang Zhang

arXiv:2304.07919v2cs.CVcs.AI

TL;DR

Vision-language prompt tuning often uses a single prompt, leaving step-to-step reasoning unmodeled, while prompt tuning can also impair generalization to unseen visual concepts. This paper introduces chained visual-language prompts with step-specific networks and an input-dependent controller. The authors report gains across image classification, image-text retrieval, and visual question answering.

  • Problem

    Prompt tuning can impair vision-language models’ generalization to unseen visual concepts, and single-prompt approaches do not model step-to-step reasoning.

    Source: §1 · p. 2

  • Method

    The method chains prompts and step-specific Meta-Nets, with a dynamic controller weighting the chain according to the input.

    Source: §3.3 · p. 4, §3.5 · p. 5

  • Results

    The authors report performance gains across image classification, image-text retrieval, and visual question answering, including base-to-new, cross-dataset, and domain generalization settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates a vision-domain chain-of-thought prompt-tuning approach that combines visual and textual embeddings.

    Source: §1 · p. 2, §5 · p. 8

    The optimal number of prompts may vary across settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language-Image Pre-training has demonstrated promising results on zero-shot and few-shot downstream tasks by prompting visual models with natural language prompts. However, most recent studies only use a single prompt for tuning, neglecting the inherent step-to-step cognitive reasoning process that humans conduct in complex task settings, for example, when processing images from unfamiliar domains. Chain of Thought is a simple and effective approximation to human reasoning process and has been proven useful for natural language processing (NLP) tasks. Based on this cognitive intuition, we believe that conducting effective reasoning is also an important problem in visual tasks, and a chain of thought could be a solution to this problem. In this work, we propose a novel chain of thought prompt tuning for vision-language modeling. Extensive experiments show that our method not only generalizes better in image classification tasks, has greater transferability beyond a single dataset, and has stronger domain generalization performance, but also performs much better in imagetext retrieval and visual question answering, which require more reasoning capabilities. We are the first to successfully adapt chain-of-thought prompting that combines visual and textual embeddings. We will release our codes

1. Introduction

Prompt tuning can impair vision-language models’ generalization to unseen visual concepts, while single-prompt methods do not model step-by-step reasoning. The paper introduces chained visual-language prompts and reports gains across classification, retrieval, and visual question answering.

1. Introduction

  • Prompt tuning can hurt vision-language models’ ability to generalize to unseen visual concepts, motivating work to bridge this generalization gap.The introduction identifies this as a concern even when tuned prompts perform well on their specific downstream tasks.Source: §1 · p. 2
  • The paper frames step-by-step reasoning as missing from prior vision-task prompting and argues that visual reasoning must combine image information with language reasoning.Unlike language, visual reasoning requires combining stepwise visual information extracted from an image with language reasoning.Source: §1 · p. 2
  • The method chains multiple prompts so each receives information from the prior prompt, then uses a chain of networks for step-specific biases and an input-dependent controller.The controller dynamically controls the chain based on the input.Source: §1 · p. 2
  • The authors present the visual-and-textual chain-of-thought prompt-tuning adaptation as the first of its kind in the vision domain.They describe the architecture as modeling relations between prompts, requiring minimal additional computing resources, and being easy to plug in.Source: §1 · p. 2
  • The authors report consistent performance improvements across five tasks, including gains in base-to-new generalization, cross-dataset transfer, domain generalization, image-text retrieval, and VQA.Reported improvements are over 1.27% for base-to-new generalization, 0.5% for cross-dataset transfer, 0.26% for domain generalization, 0.97% for image-text retrieval, and 1.07% for VQA.Source: §1 · p. 2

2. Related Works

The related work situates the method among vision-language models, prompt tuning, and chain-of-thought research. It distinguishes this approach by modeling multiple prompt steps and combining visual and textual embeddings.

2.1. Vision Language Models

  • Vision-language models align images and texts in a joint embedding space, but adapting them efficiently to downstream tasks remains difficult.The section notes that existing prompt-learning studies had not captured implicit reasoning logic in prompts.Source: §2.1 · pp. 2–3

2.2. Prompt Tuning in Vision-Language Models

  • Unlike prior vision-language prompt-tuning work, this method simulates human step-by-step thinking through multiple prompt steps.The section describes CoOp’s learnable context vectors and CoCoOp’s dynamic prompts for handling class shifts.Source: §2.2 · p. 3

2.3. Chain of Thought

  • The approach accounts for differing importance across reasoning steps and the tendency to forget earlier steps, while combining visual and textual embeddings.The related work contrasts it with, which uses pre-obtained image context with GPT-3 [34] and does not use visual features extracted from the image itself.Source: §2.3 · p. 3

3. Method

The method builds a step-by-step chain of prompts and visual-bias networks, with an image-conditioned controller adapting the chain. It combines each prompt with its predecessor and uses the final prompt for prediction.

3. Method

  • The method section introduces three components: Prompt Chaining, Self Adaptive Chain Controller, and Meta-Nets Chaining.It first formulates the problem and provides an overview of the approach.Source: §3 · pp. 3–4

3.1. Problem Formulation

  • Class probabilities are computed from cosine similarity between image and text embeddings, with temperature scaling; the encoders remain frozen during prompt training.Cross-entropy loss back-propagates gradients to update the prompt.Source: §3.1 · p. 4

3.2. Method Overview

  • The method chains prompts, passes information forward, dynamically controls the chain from image inputs, and uses the final prompt for prediction.The chain of networks also uses residual connections to preserve initial information and avoid vanishing gradients.Source: §3.2 · p. 4

3.3. Chain of Thought: Prompt Chaining

  • Each prompt combines its own embedding with the preceding prompt’s embedding, using a step-specific parameter generated by the chain controller.This lets each step use prior reasoning while conducting its own reasoning.Source: §3.3 · p. 4

3.4. Self Adaptive Chain Controller

  • The self-adaptive controller takes an image’s visual features and outputs chain parameters λ1 through λn to dynamically control the chain.It uses a lightweight linear-ReLU-linear-sigmoid structure with few additional parameters.Source: §3.4 · p. 4

3.5. Meta-Nets Chaining

  • Unlike CoCoOp’s single network, Meta-Nets Chaining assigns a network to each step to add a visual bias to that prompt’s word embeddings before text encoding.Each network receives information from the previous network and the original image; a residual-like design is preferred for information preservation, gradient flow, and stability.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate the method across five tasks and 18 datasets, with comparisons to prompt-tuning and zero-shot baselines. Results cover base-to-new generalization, cross-dataset transfer, domain generalization, retrieval, and VQA; ablations examine chain length and architecture.

4.1. Tasks

  • Experiments span five primary tasks and 18 datasets, including base-to-new evaluation, retrieval, and VQA; the base-to-new setting reports base accuracy, novel accuracy, and their harmonic mean.The harmonic mean is used to measure the generalization trade-off.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 5

4.4. Results and Analysis

  • The method exceeds CoCoOp, CoOp, and CLIP in harmonic-mean score on all 11 base-to-new datasets, and surpasses CoCoOp on novel-class accuracy across all 11.The harmonic mean summarizes the trade-off between base- and novel-class accuracy.Source: §4.4 · p. 6, Table 1 · p. 6
  • It achieves the highest accuracy on eight of 10 cross-dataset transfer datasets and is best on all four tested ImageNet domain-shift datasets.Both settings train on full ImageNet before evaluation on other datasets.Source: §4.4 · pp. 6–7, Table 2 · p. 6, Table 3 · p. 7
  • On Flickr30k and MSCOCO image-text retrieval, the method surpasses zero-shot CLIP and CoCoOp in all tests; on VQAv2, it outperforms CLIP and CoCoOp in all settings.Retrieval trains on 1%, 1.5%, and 2% of training data; VQAv2 uses 0.25%, 0.5%, and 0.75%.Source: §4.4 · p. 7, Table 5 · p. 7

4.5. Ablation Study

  • A chain length of three gives the best average harmonic-mean performance across nine datasets; longer chains are less stable and dataset-sensitive.The reported comparison tests lengths two through five, with length one equivalent to CoCoOp.Source: §4.5 · p. 7, Table 8 · p. 8, Figure 3 · p. 8
  • Chained Meta-Nets outperform unchained Meta-Nets on eight of nine datasets when both use a chain length of three.The authors associate this result with information flow between Meta-Nets.Source: Table 6 · p. 8, §4.5 · p. 8
  • Chained prompts outperform parallel averaged prompts, including a heavier CoCoOp variant with 66.7% more parameters.The comparison addresses whether gains come merely from adding parameters.Source: §4.5 · pp. 7–8, Figure 4 · p. 8
  • The dynamic chain controller performs better than fixed-chain settings, which the authors attribute to different images preferring different reasoning processes.The fixed-chain comparisons use parameters of 0.5 and 0.7.Source: §4.5 · p. 8

5. Conclusion and Limitation

The conclusion presents Chain of Thought prompt tuning as a framework for step-to-step reasoning in vision and reports experimental effectiveness. It also identifies task- and dataset-dependent prompt-chain length as a limitation.

5. Conclusion and Limitation

  • The framework models step-to-step reasoning in vision, and experiments cover image classification, image-text retrieval, and visual question answering.The conclusion also reports ablations supporting the architecture's design.Source: §5 · pp. 8–9
  • The dynamic chain controller attains the best H-score performance on 8 of 9 datasets in the base-to-new experiment.The comparison is against fixed chains with parameters 0.5 and 0.7.Source: Figure 5 · p. 9, Table 9 · p. 9
  • The optimal number of prompts may vary across downstream tasks and datasets.The authors suggest a mechanism to choose chain length based on the downstream task as a possible future direction.Source: §5 · p. 9

A. Supplementary Experimental Analysis

The supplementary analyses examine prompt concatenation, pretrained backbones, and prompt length. Their reported outcomes indicate that longer concatenated chains are not consistently advantageous, backbone choice matters, and longer randomly initialized prompts can perform promisingly.

A.1. Concatenation of Prompts

  • Concatenating additional prompts can improve some datasets and reduce performance on others, so a very lengthy chain may be less advantageous for vision tasks than for NLP.The tested variants concatenate the last two prompts or all three prompts across nine datasets.Source: §A.1 · p. 11

A.2. Effect of the Model BackBone

  • In this experiment, pretrained backbone architecture has a significant impact on model performance: the study evaluates ViT-B/32, RN50, and RN101, following prior experiments using ViT-B/16.Source: §A.2 · p. 11

A.3. Effect of the Prompt Length

  • A prompt length of 16 with random initialization attains promising results compared with a randomly initialized prompt length of 8.The authors suggest that finding a general initialization pattern for longer prompts could be beneficial.Source: §A.3 · p. 11

B.1. Model’s Confidence Visualization

The visualizations compare model confidence and class-level predictions. The authors report higher confidence than CoCoOp and a more focused confusion-matrix diagonal with their method.

B.1. Model’s Confidence Visualization

  • The method is more confident than CoCoOp, with confidence measured by the largest class probability for each image.The comparison uses the base-to-new experiment on nine datasets.Source: Figure 6 · p. 11, §B.1 · p. 11

B.2. Confusion Matrix Visualization

  • The confusion matrices show a more focused diagonal, which the authors interpret as evidence that their method can help discriminate between classes.The x-axis represents classes and the y-axis represents model predictions; the matrices can also indicate likely class confusions.Source: §B.2 · p. 11

C.1. Implementation details

The implementation details specify training and prompt settings, while accompanying tables describe comparisons of prompt aggregation and pretrained backbones.

C.1. Implementation details

  • Training uses a learning rate of 0.002 and 10 epochs, with prompt length 4 and initialization to “a photo of a”.Each Meta-Net and the controller use linear layers sized by the image-encoder output dimension and chain length.Source: §C.1 · p. 11
  • Using the final prompt yields the highest average H score and best performance on 6 of 9 datasets, while concatenating prompts is less stable.The comparison covers the final prompt, the last two prompts, and the last three prompts.Source: Table 10 · p. 12
  • Different pretrained backbones have a substantial impact on H-score performance in the base-to-new experiment.The table caption identifies a comparison across backbones on nine datasets.Source: Table 11 · p. 12

C.2. Training and testing details

The experiments cover image classification, image-text retrieval, and visual question answering, with additional comparisons of confidence, backbones, prompt choices, and hyperparameters.

C.2. Training and testing details

  • Image-classification experiments cover base-to-new and cross-dataset transfer, plus domain generalization on ImageNetV2, ImageNet-A, ImageNet-R, and ImageNet-sketch.The classification datasets span generic objects, textures, satellite imagery, fine-grained categories, scenes, and actions.Source: §C.2 · p. 12
  • Image-text retrieval uses MSCOCO and Flickr30k, with 0.5%, 1.0%, and 1.5% few-shot training samples and the first image caption as the prompt.Sampling uses seed 1.Source: §C.2 · p. 12
  • VQA v2.0 uses 0.25%, 0.5%, and 0.75% few-shot training samples; each question is classified among 3,129 answers using concatenated question-answer prompts.The experiments use the official split and seed 1 for sampling.Source: §C.2 · p. 12
  • Our method outperforms CoCoOp on 7 of 9 datasets for confidence on correct predictions and on 7 of 9 for confidence on incorrect predictions.The first metric is intended to be higher; the second, lower.Source: Table 12 · p. 13, Table 13 · p. 13
  • The backbone comparison uses base-to-new classification across 9 datasets and H score, and reports that backbone choice can substantially affect performance.The authors state that finding a proper backbone remains important.Source: Table 15 · p. 14
  • Using the final prompt gives the highest average H score and best performance on 6 of 9 datasets; concatenating more prompts is less stable.The authors suggest that feeding the entire chain of thought may be less beneficial in vision than in NLP.Source: Table 16 · p. 14
  • Randomly initialized prompts of length 4, 8, or 16 attain promising results, and the authors suggest general initialization patterns for longer prompts could help.The prompt-length comparison uses base-to-new classification on 9 datasets and H score.Source: Table 17 · p. 14
  • The confusion matrices for CoCoOp and the proposed method plot classes on the x-axis and predicted classes on the y-axis.Source: Figure 7 · p. 13, Figure 8 · p. 15

Related papers

Loading 2304.07919v2…