PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptCap: Prompt-Guided Task-Aware Image Captioning

Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, Jiebo Luo

arXiv:2211.09699v4cs.CVcs.CL

TL;DR

Generic image captions can omit visual details needed for language models to answer visual questions, and black-box models limit access to internal adaptation. PromptCap conditions captions on a natural-language question, using GPT-3-synthesized training examples, and achieves state-of-the-art results on OK-VQA and A-OKVQA while also generalizing to WebQA.

  • Problem

    Generic captions can omit visual details needed to answer questions, while many high-performing language models are accessible only through APIs and cannot be fine-tuned internally.

    Source: §1 · p. 2

  • Method

    PromptCap uses a natural-language question prompt to guide caption content and is trained with examples synthesized using GPT-3 and existing datasets.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    60.4% on OK-VQA and 59.6% on A-OKVQA; PromptCap achieves state-of-the-art results on both and also outperforms generic captions and supervised baselines on WebQA without training on its compositional questions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PromptCap works as a simple, general module for converting question-related visual information into text.

    Source: §7 · p. 9

    Some image information cannot be abstracted as text, so PromptCap should be used alongside other methods for a comprehensive understanding of visual data.

    Source: §6 · p. 9

Abstract

from arXiv · show

Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's effectiveness on an existing pipeline in which GPT-3 is prompted with image captions to carry out VQA. PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that PromptCap generalizes well to unseen domains.

1. Introduction

Black-box language models can use text descriptions of images, but generic captions may omit visual details needed to answer a question. PROMPTCAP conditions captions on a natural-language question and uses GPT-3-synthesized training examples to improve knowledge-based VQA.

1. Introduction

  • Generic captions can omit task-essential visual details, while many high-performing language models are accessible only through APIs and cannot be fine-tuned internally.For example, a generic caption may miss a sign such as “McDonald’s,” which could be needed to answer a question.Source: §1 · p. 2
  • PROMPTCAP uses a natural-language question prompt to control which visual content its caption describes for a black-box language model.The prompt contains the question the caption should help answer.Source: §1 · p. 2
  • 60.4% on OK-VQA and 59.6% on A-OKVQA accompany PROMPTCAP’s state-of-the-art results on both knowledge-based VQA tasks.The reported comparison also shows consistent gains over a generic captioning model with the same architecture and training data.Source: §1 · p. 2
  • The authors propose synthesizing and filtering training samples with GPT-3, and report that PROMPTCAP generalizes to WebQA without training on its compositional questions.On WebQA, it outperforms the generic-caption approach and supervised baselines.Source: §1 · p. 2

2. Related Work

Related work draws on external knowledge resources and language models for knowledge-based VQA, while identifying generic captions as a bottleneck. Approaches that tune visual encoders for frozen language models require access to internal parameters, unlike black-box systems such as GPT-3.

2. Related Work

  • Prior knowledge-based VQA systems use resources such as Wikipedia, ConceptNet, and ASER, while later work uses GPT-3 as an implicit knowledge base.The text identifies PICa as the first system using GPT-3 as a knowledge source and says KAT and REVIVE build on that direction.Source: §2 · pp. 2–3
  • Generic captions used to prompt GPT-3 can miss critical visual details for VQA, a bottleneck PROMPTCAP is designed to address.Source: §2 · p. 3
  • Frozen, Flamingo, and BLIP-2 keep language models frozen and tune a visual encoder, but require access to internal language-model parameters.The passage contrasts this requirement with black-box LMs such as GPT-3.Source: §2 · p. 3
  • Prompting adapts pretrained language models to tasks without modifying parameters, and GPT-3 has demonstrated zero-shot and few-shot abilities via prompting.Source: §2 · p. 3

3. PROMPTCAP

PROMPTCAP generates captions conditioned on a natural-language prompt, enabling descriptions to focus on visual details relevant to a question. GPT-3-synthesized examples train the captioner without requiring new question-aware caption annotations.

3. PROMPTCAP

  • PROMPTCAP conditions image captioning on a natural-language prompt so captions cover requested visual details, describe main objects, and use auxiliary prompt information when needed.For VQA, the prompt contains the question the caption should help answer.Source: §3 · p. 3
  • Because black-box language models cannot be fine-tuned end-to-end for VQA and question-aware caption training data are unavailable, PROMPTCAP training uses GPT-3-synthesized examples.This avoids annotating question-aware captions.Source: §3 · p. 3, §3.1 · p. 3

3.1.1 Training Example Generation with GPT-3

  • GPT-3 uses existing VQA question-answer pairs, five COCO captions per image, and 20 human-written demonstrations to synthesize question-aware captions.The images are shown for clarity but are not used in the data synthesis procedure.Source: §3.1.1 · p. 3, Figure 3 · p. 3
  • The generated examples follow three prompt-guided caption principles, with GPT-3 using commonsense reasoning to synthesize captions from caption and question text.The source captions provide GPT-3 with a text representation of each image.Source: §3.1.1 · p. 3

3.1.2 Training Example Filtering

  • Five candidate captions are generated per question-answer pair and scored by whether the GPT-3 in-context VQA system answers correctly from each caption.The candidate with the best filtering score is selected for training.Source: §3.1.2 · p. 4
  • Soft VQA accuracy ranks candidate captions using the three ground-truth answers with lowest character error rate, avoiding exact-match penalties for slight wording differences.When captions tie on soft accuracy, the one with the highest CIDEr score against COCO ground-truth captions is selected.Source: §3.1.2 · p. 4

3.2. PROMPTCAP Training

  • PROMPTCAP starts from OFA and trains an encoder-decoder to generate synthesized captions from concatenated prompt and image-patch inputs.The model is trained end-to-end with negative log-likelihood loss.Source: §3.2 · p. 4

4. VQA with PROMPTCAP and GPT-3

PROMPTCAP converts image-question examples into question-aware text contexts that GPT-3 can use. GPT-3 then predicts answers through in-context learning from demonstrations and a test example.

4. VQA with PROMPTCAP and GPT-3

  • PROMPTCAP and GPT-3 form the two components of the VQA pipeline.The pipeline is adopted from PICa.Source: §4 · p. 4

Step 1: Converting images into texts via PROMPTCAP

Step 2: GPT-3 in-context learning for VQA

Example retrieval

Step 1: Converting images into texts via PROMPTCAP

Example retrieval

  • With more training data, CLIP ranks examples by the summed cosine similarities of their question and image embeddings, and the n most similar are selected.The paper calls this selection method “CLIP.”Source: Step 1: Converting images into texts… · p. 5

5. Experiments

PROMPTCAP paired with GPT-3 outperforms prior methods on OK-VQA and A-OKVQA, while using no additional knowledge source, visual-feature ensemble, or end-to-end fine-tuning. The experiments also describe the datasets and comparison settings used.

5.1. Experimental Setup

  • The experiments evaluate OK-VQA, A-OKVQA, and WebQA, including direct-answer scoring on both OK-VQA datasets and multiple-choice scoring on A-OKVQA.WebQA is described as a multimodal multi-hop reasoning benchmark combining multiple text and image sources.Source: §5.1 · p. 5
  • PROMPTCAP is initialized from OFA and fine-tuned on GPT-3-synthesized examples from VQAv2, which is already included in OFA pre-training.The authors therefore do not add annotated training data compared with OFA.Source: §5.1 · p. 5
  • Table 1 compares OK-VQA methods by image representation and knowledge source, separating end-to-end fine-tuned methods from in-context or zero-shot methods.GPT-3 is frozen for all methods in the table.Source: Table 1 · p. 6, §5.2 · p. 6

5.2. Results on OK-VQA and A-OKVQA

  • Prior systems use GPT-3 or Chinchilla, and the described methods rely on generic captions; the paper identifies those captions as a bottleneck for VQA.The paper presents PROMPTCAP as addressing that bottleneck.Source: §5.2 · p. 6
  • 60.4% accuracy on OK-VQA and 73.1% multiple-choice and 59.6% direct-answer accuracy on A-OKVQA: PROMPTCAP + GPT-3 outperforms prior methods.The reported comparison uses no additional knowledge source, ensemble with visual features, or end-to-end fine-tuning.Source: §5.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6

5.3. Ablation Study

The ablations assess PROMPTCAP against generic OFA-Cap captions, compare GPT-3 with Flan-T5-XXL, and vary GPT-3’s in-context examples. PROMPTCAP improves results across VQA benchmarks and also outperforms generic captions on WebQA.

5.3. Ablation Study

  • The ablations evaluate PROMPTCAP, the language model, and the prompting method, and include VQAv2 to assess gains on traditional VQA.OFA-Cap is the generic-caption baseline, and PROMPTCAP shares its architecture and training data.Source: §5.3 · p. 7, §5.3.1 · p. 7

5.3.1 Performance Benefit from PROMPTCAP

  • With GPT-3, PROMPTCAP improves absolute accuracy over OFA-Cap by 3.8% on OK-VQA, 5.3% on A-OKVQA, and 9.2% on VQAv2.Table 3 measures the captioning-model contribution using GPT-3.Source: Table 3 · p. 7, §5.3.1 · p. 7

5.3.2 Performance Benefit from Language Model

  • GPT-3 improves absolute accuracy over Flan-T5-XXL by 18.4% on OK-VQA and 14.8% on A-OKVQA, but by 3.2% on VQAv2.The authors interpret external knowledge as critical for knowledge-based VQA, but not VQAv2.Source: Table 4 · p. 7, §5.3.2 · p. 7

5.3.3 Ablation on GPT-3 Prompting

  • On OK-VQA, more in-context examples improve performance for both selection strategies, and CLIP-retrieved examples add 5.2% absolute accuracy for PROMPTCAP at n = 32.The study varies n across 1, 4, 16, and 32, comparing random examples with examples retrieved by CLIP.Source: Table 5 · p. 7, §5.3.3 · p. 7

5.4. Domain Transfer on WebQA

  • On WebQA, PROMPTCAP outperforms generic OFA captions and all official baselines despite using only 8-shot in-context learning.The evaluation uses web images and longer compositional questions; PROMPTCAP was not trained on those questions.Source: §5.4 · pp. 7–8, Table 6 · p. 8

5.5. Qualitative Analysis

The qualitative examples show PROMPTCAP captions supplying question-relevant details that help GPT-3 answer correctly, while also documenting failures. Caption comparisons find high similarity to synthesized question-aware captions and strong generic-caption scores on COCO.

5.5. Qualitative Analysis

  • For the illustrated OK-VQA questions, GPT-3 answers correctly from PROMPTCAP captions but fails with generic captions, which can omit queried attributes or objects.Examples include brand, color, and the clothing a person is putting on.Source: §5.5 · p. 8
  • PROMPTCAP failures include captions that provide unhelpful or unfactual information, as well as occasional GPT-3 answer mistakes.The authors describe unhelpful or unfactual captions as the majority of the shown failure cases.Source: §5.5 · p. 9

5.6. Analysis: How Do Question-Aware Captions Differ from Generic Captions?

  • GPT-3-synthesized question-aware captions are similar to MSCOCO ground-truth captions, and PROMPTCAP scores highly against the synthesized captions on CIDEr and SPICE.The authors suggest synthesized captions are shorter, less diverse, and cover fewer visual entities, making the prompt-guided captioning task less ambiguous.Source: §5.6 · p. 9, Table 7 · p. 9
  • When prompted with “what does the image describe?”, PROMPTCAP produces generic captions that achieve state-of-the-art automatic-metric scores on COCO validation, exceeding OFA-Cap.The cited metrics are BLEU-4, METEOR, CIDEr, and SPICE.Source: §5.6 · p. 9

6. Limitations and Broader Impact

The authors identify two limitations: PROMPTCAP currently focuses on knowledge-based VQA, and some image information cannot be abstracted as text. They show an NLVR2 example as a possible extension beyond VQA.

6. Limitations and Broader Impact

  • The current model focuses on knowledge-based VQA, although the authors suggest extending it to other vision-language tasks.They identify broader applications and more diverse training tasks and instructions as future work.Source: §6 · p. 9
  • Some image information cannot be abstracted as text, so the authors recommend using PROMPTCAP alongside other methods for comprehensive visual understanding.Source: §6 · p. 9
  • An NLVR2 example illustrates a possible use of PROMPTCAP with ChatGPT through a series of vision and reasoning steps.Source: Figure 7 · p. 9, §6 · p. 9

7. Conclusion

The paper presents prompt-controlled, question-aware captioning and a GPT-3-based pipeline for training it without extra annotation. It reports state-of-the-art results on knowledge-based VQA, consistent gains over generic captions, and generalization to WebQA.

7. Conclusion

  • PROMPTCAP is a question-aware captioning model controlled by a natural-language prompt.Source: §7 · p. 9
  • GPT-3 synthesizes and filters PROMPTCAP training examples without extra annotation.Source: §7 · p. 9
  • The system achieves state-of-the-art performance on OK-VQA and A-OKVQA, with ablations showing consistent gains over generic captions and evaluation examining WebQA generalization.Source: §7 · p. 9

A. More Implementation Details

PROMPTCAP is fine-tuned on GPT-3-synthesized examples, with training and evaluation-run details specified.

A. More Implementation Details

B. Common Q & A

PROMPTCAP’s gains over OFA are attributed to instruction-guided caption control rather than additional annotated data, and it can also produce generic captions.

B. Common Q & A

  • PROMPTCAP’s performance gain over OFA is attributed to controlling image captions with natural-language instructions, not to more training data.VQAv2 was already included in OFA’s training data; PROMPTCAP uses it to fine-tune OFA for prompt-guided captioning.Source: B. Common Q & A · p. 14
  • PROMPTCAP’s prompt-guided fine-tuning uses the same amount of annotated data as OFA while yielding significant performance gains.Source: B. Common Q & A · p. 14
  • CIDEr is 150.1 on the COCO validation set when PROMPTCAP is prompted with “what does the image describe?” for generic captioning.Source: B. Common Q & A · p. 14

C. Prompt Examples

The appendix illustrates GPT-3 prompts for synthesizing question-aware captions and for in-context VQA, with examples showing captions tailored to question-specific visual details.

C.1. Prompt for example synthesis

  • The synthesis prompt frames the task as summarization, includes human-written examples, and places the question-answer pair at the end for GPT-3 to generate a prompt-guided caption.Source: §C.1 · p. 14
  • Examples turn questions about counts, materials, and colors into captions that state the corresponding visual detail, such as four cabinets, a tiled backsplash, or silver objects.Source: §C.1 · p. 14
  • Other examples include question-specific context such as a summer scene, a watermelon-style helmet, or a red car driving north.Source: §C.1 · pp. 15–16
  • Some synthesized captions include details tied to broader questions, including a dish cooked at IHOP and a man wearing a hat.Source: §C.1 · pp. 15–16

C.2. GPT-3 In-Context Learning for VQA

  • For GPT-3 in-context VQA, the appendix uses the PICa prompt template and eight closest examples retrieved from the training set.Source: §C.2 · p. 17
  • The VQA demonstrations pair a caption-like context with a question and answer, including a fruit-bowl material question answered “glass.”Source: §C.2 · p. 17
Loading 2211.09699v4…