PaperlayerView exact revision on arXiv

Source-linked AI summary

From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, Steven C. H. Hoi

arXiv:2212.10846v3cs.CVcs.MM

TL;DR

Zero-shot VQA remains difficult for LLMs because vision-language and task disconnections require costly end-to-end alignment. Img2LLM instead supplies image-derived captions and synthetic QA exemplars to frozen LLMs, achieving competitive or superior zero-shot performance while avoiding that training cost.

  • Problem

    LLMs face modality and task disconnections when applied to zero-shot VQA, while end-to-end alignment is costly and inflexible.

    Source: §1 · p. 1

  • Method

    Img2LLM uses LLM-agnostic models to generate image-derived captions and synthetic question-answer exemplars from the current image and question.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 3

  • Results

    Img2LLM achieves competitive or superior zero-shot VQA performance to costly end-to-end methods, including a 5.6% improvement over Flamingo on VQAv2.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Frozen off-the-shelf LLMs can be used for flexible, lower-cost zero-shot VQA deployment without expensive end-to-end multimodal alignment.

    Source: §1 · p. 2, §6 · p. 8

    Caption and QA generation adds 24.4% computational time on top of 175B OPT inference, although shorter prompts can trade accuracy for speed.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answering (VQA) remains challenging, primarily due to the modality disconnection and task disconnection between LLM and VQA task. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. To address this issue, we propose \emph{Img2Prompt}, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training. In order to provide such prompts, we further employ LLM-agnostic models to provide prompts that can describe image content and self-constructed question-answer pairs, which can effectively guide LLM to perform zero-shot VQA tasks. Img2Prompt offers the following benefits: 1) It can flexibly work with various LLMs to perform VQA. 2)~Without the needing of end-to-end training, it significantly reduces the cost of deploying LLM for zero-shot VQA tasks. 3) It achieves comparable or better performance than methods relying on end-to-end training. For example, we outperform Flamingo \cite{Deepmind:Flamingo2022} by 5.6\% on VQAv2. On the challenging A-OKVQA dataset, our method even outperforms few-shot methods by as much as 20\%.

1. Introduction

The paper targets zero-shot VQA with frozen LLMs, whose language-only training leaves vision-language and task mismatches unresolved. Img2LLM addresses both through image-derived synthetic QA prompts, achieving competitive or superior performance without costly end-to-end training.

  • VQA supports applications such as helping blind individuals understand their environments, but human annotations can be expensive and introduce biases.Source: §1 · p. 1
  • LLMs generalize well in language tasks, yet applying them to VQA is difficult because vision and language, and language modeling and question answering, are disconnected.Source: §1 · p. 1
  • End-to-end vision-language alignment can bridge these gaps but requires prohibitive computational and data resources, including billions of image-text pairs and thousands of TPUs for Flamingo.Source: §1 · p. 1
  • Img2LLM generates synthetic question-answer pairs from the current image and question, using them to describe image content and demonstrate the VQA task to an LLM.Source: §1 · p. 2
  • The module enables off-the-shelf LLMs to perform zero-shot VQA without costly end-to-end training or specialized textual QA networks, supporting flexible deployment and upgrades.Source: §1 · p. 2
  • 5.6%: Img2LLM outperforms Flamingo on VQAv2, while also achieving competitive or superior performance to end-to-end trained models and outperforming many few-shot methods.Source: §1 · p. 2

2. Related Work

Related work spans multimodal pretraining and language-mediated VQA. Multimodal approaches align visual and language representations at high cost, while language-mediated methods use textual image descriptions but remain sensitive to exemplar availability.

  • VQA requires answering natural-language questions from images, with recent advances driven by image-text pretraining and VQA fine-tuning.Source: §2.1 · p. 2
  • Knowledge-based VQA methods incorporate resources such as ConceptNet or Wikipedia, but they still struggle with questions requiring complex reasoning.Source: §2.1 · p. 2
  • LLMs generate target tokens autoregressively from prompts and task inputs; prior LLM-based VQA methods mainly use multimodal pretraining or language mediation.Source: §2.2 · p. 2

Multimodal pretraining

  • Multimodal pretraining aligns vision and language embeddings through trainable visual encoders or cross-attention modules, requiring substantial computational resources.Source: §2.2 · p. 2

Language-mediated VQA

  • Language-mediated VQA converts images into textual descriptions for frozen LLMs, avoiding expensive multimodal pretraining.Source: §2.2 · p. 2
  • PICa uses image captions and training examples as in-context exemplars, but its performance degrades substantially when exemplars are omitted.Source: §2.2 · p. 2

3. Method

Img2LLM bridges VQA’s modality and task gaps by generating question-relevant captions and synthetic QA exemplars from the current image. These textual prompts guide frozen LLMs while fitting within their context limits.

Overview

  • Img2LLM generates image-relevant exemplar prompts that communicate image content and demonstrate question answering to generic LLMs in zero-shot VQA.Source: §3 · p. 3

Answer candidates

  • The method extracts candidate answers from generated captions, including noun, verb, adjective, number, named-entity, and boolean-typed phrases.Source: §3.1 · p. 3

Question Generation

  • Question-generation networks convert extracted answer candidates and their caption contexts into synthetic questions without using VQA data.Source: §3.2 · p. 3
  • The neural question generator is a T5-large model fine-tuned on five textual QA datasets and applied to extracted answers paired with generated captions.Source: §3.2 · p. 3

Exemplar Prompt

  • Synthetic QA pairs serve as in-context exemplars that guide the LLM on the QA task while conveying image content.Source: §3.2 · p. 3

Prompt design

  • Exemplar prompts perform considerably better than caption-only prompts, while adding captions provides only limited additional gains.Source: §3.2 · p. 4

Caption Prompt

  • Question-relevant captions target image regions needed by the question, addressing cases where generic whole-image captions omit relevant information.Source: §3.3 · p. 4
  • GradCAM uses question-image similarity to produce a coarse localization map, after which relevant patches are sampled for caption generation.Source: §3.3 · p. 4

4. Experiment

Experiments evaluate Img2LLM across three VQA datasets, compare it with zero-shot, few-shot, pretrained, and supervised methods, and test LLM scaling, generalization, prompt construction, and caption selection. Results show strong zero-shot performance, broad compatibility with LLMs, and sensitivity to prompt design and selected visual information.

Experimental Setup

  • Img2LLM is evaluated on VQAv2, OK-VQA, and A-OKVQA, covering perception, reasoning, and commonsense questions.The evaluation uses official VQA protocols and reports scores for each dataset.Source: §4.1 · p. 5

Main Results

  • 45.6 versus 17.7 on OK-VQA: Img2LLM surpasses PICa, the strongest prior zero-shot frozen-LLM method.Unlike PICa, Img2LLM constructs question-answer prompts without access to VQA samples.Source: §4.2 · p. 5
  • 3-10 points improve VQA performance across datasets when LLM size increases from 6.7B to 175B.The trend is clearer on OK-VQA and A-OKVQA, whose questions require commonsense reasoning and external knowledge.Source: §4.2 · p. 5
  • Scaling effects become obvious mainly with 30B-or-larger models, while improvements are not entirely predictable for 6.7B and 13B models.This extends observations about emergent abilities to frozen LLMs used for vision-language tasks.Source: §4.2 · p. 5
  • 5.6 points: Img2LLM surpasses Flamingo80B on VQAv2 despite Flamingo's billion-scale training data and over 500K TPU hours.Img2LLM also more than doubles Clip-Clap's best reported A-OKVQA result, but trails Flamingo80B on OK-VQA.Source: §4.2 · p. 6

Experimental Results of Different LLMs

  • All tested alternative LLMs—GPT-J, GPT-Neo, and BLOOM—outperform zero-shot PICa and Frozen, supporting Img2LLM's generalization across LLMs.The experiments specifically test open-source LLMs other than OPT.Source: §4.3 · pp. 6–7

Question Selection

  • Neural-generated questions perform best, while image-agnostic questions perform worst; maximum-frequency answer selection beats random selection.The authors attribute differences to QA-pair relevance, linguistic diversity, and the salience of frequent answers.Source: §4.4 · p. 7

Prompt Design

  • The CCC-QAQAQA prompt layout significantly outperforms CQA-CQA-CQA, highlighting the importance of QA-prompt positioning.The authors hypothesize that paired captions may cause the LLM to read only one caption before answering.Source: §4.6 · p. 7

5. Limitation

Img2LLM incurs additional inference overhead from generating image captions and question-answer pairs, though shortening prompts can trade accuracy for speed.

  • 24.4% additional computational time is incurred beyond 175B OPT inference on an 8×A100 machine.The overhead comes from generating image captions and question-answer pairs.Source: §5 · p. 8
  • Shortening the prompt can reduce overhead, but this trades accuracy for speed.Source: §5 · p. 8
  • Img2LLM avoids expensive end-to-end multimodal representation alignment, which took more than 500K TPU hours for Flamingo.Source: §5 · p. 8

6. Conclusion

Img2LLM uses prompts to provide visual information and task guidance to frozen LLMs, avoiding expensive end-to-end alignment while supporting flexible deployment across models.

  • Img2LLM provides visual information and task guidance to off-the-shelf LLMs through easily digestible prompts.Source: §6 · p. 8
  • The method eliminates expensive end-to-end vision-language alignment, increasing deployment flexibility and decreasing deployment cost.Source: §6 · p. 8
  • Experiments show that Img2LLM enables different LLMs to achieve comparable or superior zero-shot VQA performance to methods requiring costly end-to-end training.Source: §6 · p. 8

A.1. Reproducibility Statement

The reproducibility statement commits to providing implementation details and publicly releasing the code after paper acceptance.

  • Implementation details, including hyperparameters, are provided in Section 4.1 of the main paper.Source: §A.1 · p. 13
  • The authors plan to publicly release all code after acceptance.Source: §A.1 · p. 13

A.2. Broader Impact Statement

Img2LLM does not reduce the inherent social-economic biases present in the datasets, LLMs, and VQA systems used in the work.

  • Img2LLM has not reduced the inherent bias of the zero-shot VQA systems considered.Source: §A.2 · p. 13
  • The acknowledged biases concern gender, age, race, and ethnicity across datasets, LLMs, and VQA systems.Source: §A.2 · p. 13
  • Future work could assess the magnitude of these biases and mitigate their impact.Source: §A.2 · p. 13

A.3. Details about Question-Relevant Caption Generation

Question-relevant captions are generated by using BLIP cross-attention to identify image patches associated with the question. GradCAM aggregates attention-based gradients to locate less relevant patches for caption generation.

Feature extraction

  • BLIP image-grounded text-encoder features represent image patches and question tokens across its layers.The patch and textual feature dimensions are denoted D_v^i and D_q^i, respectively.Source: §A.3 · p. 13

Cross-attention

  • Cross-attention scores are computed between each question token and every image patch using query, key, and value features.The resulting attention matrix has shape R^L×K, with rows representing token-to-patch scores.Source: §A.3 · p. 13

Question-relevant patch selection

  • GradCAM averages over attention heads and aggregates textual-token contributions to estimate patch relevance for the image-question similarity.This relevance computation is used to identify image patches that contribute less and may be uninformative.Source: §A.3 · p. 13

A.4. Experimental Results of Supervised Learning Methods in A-OKVQA

The experiments compare the method with supervised approaches on A-OKVQA and examine prompt construction choices. Results indicate strong performance with smaller language models and robustness to different prompt sizes, alongside qualitative success and failure cases.

Supervised-model comparison

  • Our method outperforms almost all supervised models on A-OKVQA while using a smaller language model.The comparison is reported in Table 10 and is presented as evidence that the method leverages large language models’ reasoning power.Source: §A.4 · p. 13, Table 8 · p. 14

Prompt sensitivity

  • Prompt performance on OK-VQA changes by no more than 1 in QA score across the tested numbers of QA pairs and captions.The experiments use OPT 30B, and the corresponding table reports different numbers of captions and QA pairs as prompts.Source: §A.6 · p. 13, Table 10 · p. 14, Table 11 · p. 14

Qualitative analysis

  • The qualitative analyses distinguish successful and failed predictions for OK-VQA and A-OKVQA using green answer cues and red incorrect-prediction markers.The examples include question, ground-truth answer, captions, synthetic questions, and predicted answers.Source: Figure 4 · p. 15, Figure 5 · p. 16, Figure 6 · p. 17

Related papers

Loading 2212.10846v3…