PaperlayerView exact revision on arXiv

Source-linked AI summary

From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, Steven C. H. Hoi

arXiv:2212.10846v3cs.CVcs.MM

TL;DR

Zero-shot VQA remains difficult for LLMs because vision-language and task disconnections require costly end-to-end alignment. Img2LLM instead supplies image-derived captions and synthetic QA exemplars to frozen LLMs, achieving competitive or superior zero-shot performance while avoiding that training cost.

  • Problem

    LLMs face modality and task disconnections when applied to zero-shot VQA, while end-to-end alignment is costly and inflexible.

  • Method

    Img2LLM uses LLM-agnostic models to generate image-derived captions and synthetic question-answer exemplars from the current image and question.

  • Results

    Img2LLM achieves competitive or superior zero-shot VQA performance to costly end-to-end methods, including a 5.6% improvement over Flamingo on VQAv2.

  • Takeaways & Limitations

    Frozen off-the-shelf LLMs can be used for flexible, lower-cost zero-shot VQA deployment without expensive end-to-end multimodal alignment.

  • Takeaways & Limitations

    Caption and QA generation adds 24.4% computational time on top of 175B OPT inference, although shorter prompts can trade accuracy for speed.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answering (VQA) remains challenging, primarily due to the modality disconnection and task disconnection between LLM and VQA task. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. To address this issue, we propose \emph{Img2Prompt}, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training. In order to provide such prompts, we further employ LLM-agnostic models to provide prompts that can describe image content and self-constructed question-answer pairs, which can effectively guide LLM to perform zero-shot VQA tasks. Img2Prompt offers the following benefits: 1) It can flexibly work with various LLMs to perform VQA. 2)~Without the needing of end-to-end training, it significantly reduces the cost of deploying LLM for zero-shot VQA tasks. 3) It achieves comparable or better performance than methods relying on end-to-end training. For example, we outperform Flamingo \cite{Deepmind:Flamingo2022} by 5.6\% on VQAv2. On the challenging A-OKVQA dataset, our method even outperforms few-shot methods by as much as 20\%.

1. Introduction

The paper targets zero-shot VQA with frozen LLMs, whose language-only training leaves vision-language and task mismatches unresolved. Img2LLM addresses both through image-derived synthetic QA prompts, achieving competitive or superior performance without costly end-to-end training.

  • VQA supports applications such as helping blind individuals understand their environments, but human annotations can be expensive and introduce biases.
  • LLMs generalize well in language tasks, yet applying them to VQA is difficult because vision and language, and language modeling and question answering, are disconnected.
  • End-to-end vision-language alignment can bridge these gaps but requires prohibitive computational and data resources, including billions of image-text pairs and thousands of TPUs for Flamingo.
  • Img2LLM generates synthetic question-answer pairs from the current image and question, using them to describe image content and demonstrate the VQA task to an LLM.
  • The module enables off-the-shelf LLMs to perform zero-shot VQA without costly end-to-end training or specialized textual QA networks, supporting flexible deployment and upgrades.
  • 5.6%: Img2LLM outperforms Flamingo on VQAv2, while also achieving competitive or superior performance to end-to-end trained models and outperforming many few-shot methods.

2. Related Work

Related work spans multimodal pretraining and language-mediated VQA. Multimodal approaches align visual and language representations at high cost, while language-mediated methods use textual image descriptions but remain sensitive to exemplar availability.

  • VQA requires answering natural-language questions from images, with recent advances driven by image-text pretraining and VQA fine-tuning.
  • Knowledge-based VQA methods incorporate resources such as ConceptNet or Wikipedia, but they still struggle with questions requiring complex reasoning.
  • LLMs generate target tokens autoregressively from prompts and task inputs; prior LLM-based VQA methods mainly use multimodal pretraining or language mediation.
  • Multimodal pretraining: Multimodal pretraining aligns vision and language embeddings through trainable visual encoders or cross-attention modules, requiring substantial computational resources.
  • Language-mediated VQA: Language-mediated VQA converts images into textual descriptions for frozen LLMs, avoiding expensive multimodal pretraining.
  • Language-mediated VQA: PICa uses image captions and training examples as in-context exemplars, but its performance degrades substantially when exemplars are omitted.

3. Method

Img2LLM bridges VQA’s modality and task gaps by generating question-relevant captions and synthetic QA exemplars from the current image. These textual prompts guide frozen LLMs while fitting within their context limits.

  • Overview: Img2LLM generates image-relevant exemplar prompts that communicate image content and demonstrate question answering to generic LLMs in zero-shot VQA.
  • Answer candidates: The method extracts candidate answers from generated captions, including noun, verb, adjective, number, named-entity, and boolean-typed phrases.
  • Question Generation: Question-generation networks convert extracted answer candidates and their caption contexts into synthetic questions without using VQA data.
  • Question Generation: The neural question generator is a T5-large model fine-tuned on five textual QA datasets and applied to extracted answers paired with generated captions.
  • Exemplar Prompt: Synthetic QA pairs serve as in-context exemplars that guide the LLM on the QA task while conveying image content.
  • Prompt design: Exemplar prompts perform considerably better than caption-only prompts, while adding captions provides only limited additional gains.
  • Caption Prompt: Question-relevant captions target image regions needed by the question, addressing cases where generic whole-image captions omit relevant information.
  • Caption Prompt: GradCAM uses question-image similarity to produce a coarse localization map, after which relevant patches are sampled for caption generation.

4. Experiment

Experiments evaluate Img2LLM across three VQA datasets, compare it with zero-shot, few-shot, pretrained, and supervised methods, and test LLM scaling, generalization, prompt construction, and caption selection. Results show strong zero-shot performance, broad compatibility with LLMs, and sensitivity to prompt design and selected visual information.

  • Experimental Setup: Img2LLM is evaluated on VQAv2, OK-VQA, and A-OKVQA, covering perception, reasoning, and commonsense questions.The evaluation uses official VQA protocols and reports scores for each dataset.
  • Main Results: 45.6 versus 17.7 on OK-VQA: Img2LLM surpasses PICa, the strongest prior zero-shot frozen-LLM method.Unlike PICa, Img2LLM constructs question-answer prompts without access to VQA samples.
  • Main Results: 3-10 points improve VQA performance across datasets when LLM size increases from 6.7B to 175B.The trend is clearer on OK-VQA and A-OKVQA, whose questions require commonsense reasoning and external knowledge.
  • Main Results: Scaling effects become obvious mainly with 30B-or-larger models, while improvements are not entirely predictable for 6.7B and 13B models.This extends observations about emergent abilities to frozen LLMs used for vision-language tasks.
  • Main Results: 5.6 points: Img2LLM surpasses Flamingo80B on VQAv2 despite Flamingo's billion-scale training data and over 500K TPU hours.Img2LLM also more than doubles Clip-Clap's best reported A-OKVQA result, but trails Flamingo80B on OK-VQA.
  • Experimental Results of Different LLMs: All tested alternative LLMs—GPT-J, GPT-Neo, and BLOOM—outperform zero-shot PICa and Frozen, supporting Img2LLM's generalization across LLMs.The experiments specifically test open-source LLMs other than OPT.
  • Question Selection: Neural-generated questions perform best, while image-agnostic questions perform worst; maximum-frequency answer selection beats random selection.The authors attribute differences to QA-pair relevance, linguistic diversity, and the salience of frequent answers.
  • Prompt Design: The CCC-QAQAQA prompt layout significantly outperforms CQA-CQA-CQA, highlighting the importance of QA-prompt positioning.The authors hypothesize that paired captions may cause the LLM to read only one caption before answering.

5. Limitation

Img2LLM incurs additional inference overhead from generating image captions and question-answer pairs, though shortening prompts can trade accuracy for speed.

  • 24.4% additional computational time is incurred beyond 175B OPT inference on an 8×A100 machine.The overhead comes from generating image captions and question-answer pairs.
  • Shortening the prompt can reduce overhead, but this trades accuracy for speed.
  • Img2LLM avoids expensive end-to-end multimodal representation alignment, which took more than 500K TPU hours for Flamingo.

6. Conclusion

Img2LLM uses prompts to provide visual information and task guidance to frozen LLMs, avoiding expensive end-to-end alignment while supporting flexible deployment across models.

  • Img2LLM provides visual information and task guidance to off-the-shelf LLMs through easily digestible prompts.
  • The method eliminates expensive end-to-end vision-language alignment, increasing deployment flexibility and decreasing deployment cost.
  • Experiments show that Img2LLM enables different LLMs to achieve comparable or superior zero-shot VQA performance to methods requiring costly end-to-end training.

A.1. Reproducibility Statement

The reproducibility statement commits to providing implementation details and publicly releasing the code after paper acceptance.

  • Implementation details, including hyperparameters, are provided in Section 4.1 of the main paper.
  • The authors plan to publicly release all code after acceptance.

A.2. Broader Impact Statement

Img2LLM does not reduce the inherent social-economic biases present in the datasets, LLMs, and VQA systems used in the work.

  • Img2LLM has not reduced the inherent bias of the zero-shot VQA systems considered.
  • The acknowledged biases concern gender, age, race, and ethnicity across datasets, LLMs, and VQA systems.
  • Future work could assess the magnitude of these biases and mitigate their impact.

A.3. Details about Question-Relevant Caption Generation

Question-relevant captions are generated by using BLIP cross-attention to identify image patches associated with the question. GradCAM aggregates attention-based gradients to locate less relevant patches for caption generation.

  • Feature extraction: BLIP image-grounded text-encoder features represent image patches and question tokens across its layers.The patch and textual feature dimensions are denoted D_v^i and D_q^i, respectively.
  • Cross-attention: Cross-attention scores are computed between each question token and every image patch using query, key, and value features.The resulting attention matrix has shape R^L×K, with rows representing token-to-patch scores.
  • Question-relevant patch selection: GradCAM averages over attention heads and aggregates textual-token contributions to estimate patch relevance for the image-question similarity.This relevance computation is used to identify image patches that contribute less and may be uninformative.

A.4. Experimental Results of Supervised Learning Methods in A-OKVQA

The experiments compare the method with supervised approaches on A-OKVQA and examine prompt construction choices. Results indicate strong performance with smaller language models and robustness to different prompt sizes, alongside qualitative success and failure cases.

  • Supervised-model comparison: Our method outperforms almost all supervised models on A-OKVQA while using a smaller language model.The comparison is reported in Table 10 and is presented as evidence that the method leverages large language models’ reasoning power.
  • Prompt sensitivity: Prompt performance on OK-VQA changes by no more than 1 in QA score across the tested numbers of QA pairs and captions.The experiments use OPT 30B, and the corresponding table reports different numbers of captions and QA pairs as prompts.
  • Qualitative analysis: The qualitative analyses distinguish successful and failed predictions for OK-VQA and A-OKVQA using green answer cues and red incorrect-prediction markers.The examples include question, ground-truth answer, captions, synthetic questions, and predicted answers.
Loading 2212.10846v3…