Source-linked AI summary
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, Lijuan Wang
TL;DR
Knowledge-based VQA requires external knowledge, but retrieve-then-reason pipelines can introduce mismatches between retrieval and reasoning. PICa instead prompts GPT-3 with textual image descriptions and a few in-context examples, jointly acquiring and processing relevant knowledge. On OK-VQA, it reaches 48.0% accuracy in a few-shot setting, surpassing the supervised state of the art, while its image-to-text abstraction limits detailed visual understanding.
Problem
Retrieve-then-reason knowledge-based VQA can select noisy or irrelevant knowledge and alter its meaning during re-embedding.
Method
PICa prompts GPT-3 as an implicit, unstructured knowledge base using image captions or tags and a few in-context VQA examples.
Results
48.0% accuracy on OK-VQA surpasses the supervised state of the art at 39.4% in a few-shot setting.
Takeaways & Limitations
PICa demonstrates that GPT-3 can implicitly acquire relevant knowledge and reason over textual image context for knowledge-based VQA.
Takeaways & Limitations
Captions or tags provide only partial image descriptions and may miss detailed visual attributes needed for answering.
Abstract
from arXiv · showhide
Knowledge-based visual question answering (VQA) involves answering questions that require external knowledge not present in the image. Existing methods first retrieve knowledge from external resources, then reason over the selected knowledge, the input image, and question for answer prediction. However, this two-step approach could lead to mismatches that potentially limit the VQA performance. For example, the retrieved knowledge might be noisy and irrelevant to the question, and the re-embedded knowledge features during reasoning might deviate from their original meanings in the knowledge base (KB). To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by GPT-3's power in knowledge retrieval and question answering, instead of using structured KBs as in previous work, we treat GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge. Specifically, we first convert the image into captions (or tags) that GPT-3 can understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just providing a few in-context VQA examples. We further boost performance by carefully investigating: (i) what text formats best describe the image content, and (ii) how in-context examples can be better selected and used. PICa unlocks the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa surpasses the supervised state of the art by an absolute +8.6 points on the OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent few-shot performance.
Introduction
PICa addresses limitations of retrieve-then-reason knowledge-based VQA by prompting GPT-3 to jointly acquire and reason over knowledge represented as image text. It achieves strong few-shot results on OK-VQA using captions or tags and in-context examples.
- Existing knowledge-based VQA methods retrieve external knowledge before reasoning over it with the image-question pair.
- Retrieved knowledge can be noisy or mismatched, while re-embedding may alter its original meaning and hinder transfer to new question types.
- PICa treats GPT-3 as an implicit, unstructured knowledge base that jointly acquires and reasons over relevant knowledge.
- PICa converts images into captions or tags and adapts GPT-3 through a few in-context VQA examples without supervised fine-tuning.
- 46.9% accuracy on OK-VQA represents a 7.5-point absolute improvement over the supervised state of the art, rising to 48.0 with predicted image tags.
- PICa also reports decent few-shot performance on VQAv2 and investigates image-text formats, example selection, and multi-query ensemble.
Related Work
Prior knowledge-based VQA work retrieves information from external resources and then reasons over it, while related multimodal few-shot work extends language-model abilities to vision and language.
- Knowledge-based VQA requires external knowledge in addition to image content, motivating retrieval from structured resources and subsequent multimodal reasoning.
- Recent methods combine structured knowledge bases with implicit knowledge retrieved from pretrained language models or use answer validation after retrieval.
- GPT-3 established strong in-context few-shot learning, adapting to new tasks from examples without parameter updates.
- Frozen extends few-shot language-model capabilities to vision-language tasks by training a visual encoder while freezing a GPT-like language model.
Approach
PICa prompts GPT-3 with textual image representations, a prompt head, in-context examples, and the VQA input, then generates answers openly. Example selection and multi-query ensembling improve use of limited context capacity.
- GPT-3 for VQA: The prompt contains a fixed head, n image-question-answer examples, and the new VQA input; GPT-3 produces an open-ended answer.
- GPT-3 for VQA: GPT-3 cannot inherently understand images, so PICa translates each VQA image into captions or tags before prompting the language model.
- GPT-3 for VQA: PICa uses captioning or tagging models to form the image context and concatenates it with the question and answer prefix.
- In-context Examples: More examples generally improve few-shot performance, but available context length limits how many examples can fit in a prompt.
- In-context Examples: Example selection retrieves demonstrations with question features similar to the inference input using CLIP similarity.
- In-context Examples: Multi-query ensemble distributes n*k examples across k prompts and selects the prediction with the highest summed log-probability.
Experiments on OK-VQA
PICa evaluates few-shot knowledge-based VQA on OK-VQA by representing images as text and prompting GPT-3, with experiments on representations, example selection, and qualitative behavior. Using captions and tags, PICa-Full reaches 48.0% accuracy with 16 examples, while ablations show benefits from richer descriptions and similarity-guided selection.
- Setup and overall results: PICa represents images with VinVL captions or captions enhanced by tags, while PICa-Full additionally selects examples and uses multi-query ensemble.PICa-Base randomly selects in-context examples; PICa-Full includes example selection and multi-query ensemble.
- Setup and overall results: PICa-Full reaches 48.0% accuracy with 16 in-context examples, compared with the supervised state-of-the-art accuracy of 39.4%.The method uses no model fine-tuning and represents images with captions and tags.
- Few-shot Ability: More shots generally improve performance, increasing accuracy from 40.8% with n = 1 to 48.0% with n = 16.The experiments vary n from 0 to 16, with n = 16 roughly matching GPT-3's maximum example capacity.
- Textual Representation for Images: Richer textual image descriptions improve performance: VinVL-Caption-CC reaches 37.0% with 16 shots, while selection and ensemble raise it to 44.0%.VinVL-Caption-COCO performs best among predicted captions, and concatenated ground-truth captions reach 53.3% oracle accuracy.
- Example Selection and Multi-query Ensemble: Question-similarity example selection helps few-shot VQA, whereas dissimilar examples produce 40.1% accuracy and jointly selected question-image examples reach 46.5%.The oracle selection condition reaches 49.1%, and image similarity alone improves the random baseline less than question similarity alone.
- Qualitative Analysis: PICa qualitatively answers questions requiring factual, commonsense, and implicit knowledge, and GPT-3 generates reasonable zero-shot answer rationales.Examples include transportation invention dates, obtaining bananas from grocery stores, train stops, and possible sharks in the sea.
Experiments on VQAv2
On VQAv2, PICa is evaluated on questions closely tied to detailed image content, where textual image descriptions provide useful but incomplete visual information. PICa achieves strong few-shot accuracy, but remains below supervised performance and fails on some detailed visual questions.
- Dataset and setup: VQAv2 emphasizes detailed image contents, and human accuracy rises from 57.5% with questions and captions to 83.3% with questions and images.Questions alone yield 40.8% human performance.
- Results: 56.1% accuracy makes PICa-Full substantially stronger than the previous few-shot result of 38.2% on VQAv2.PICa-Base reaches 54.3%, while PICa-Full reaches 56.1%.
- Results: PICa-Full remains around 17% lower than Oscar’s supervised accuracy of 73.8%.
- Qualitative analysis: PICa generally handles commonsense questions, such as interpreting exit signs or why a cow is lying down.
- Limitations: Incomplete textual descriptions cause failures on detailed attributes and counts, including object color and the number of giraffes.The authors expect end-to-end vision encoder tuning could improve these cases.
Conclusion
PICa uses GPT-3 for few-shot knowledge-based VQA, jointly acquiring and processing relevant knowledge through prompting rather than explicit structured knowledge bases. Analyses indicate that PICa implicitly acquires knowledge relevant to answering questions.
- Conclusion: PICa uses GPT-3 for few-shot knowledge-based VQA instead of explicit structured knowledge bases.Prompting lets PICa jointly acquire and process relevant knowledge.
- Conclusion: PICa surpasses the supervised state of the art on OK-VQA by a significant margin.
- Conclusion: Analyses show that PICa implicitly acquires relevant knowledge to answer questions.