PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

Yunxin Li, Longyue Wang, Baotian Hu, Xinyu Chen, Wanqi Zhong, Chenyang Lyu, Wei Wang, Min Zhang

arXiv:2311.07536v3cs.CL

TL;DR

Knowledge-intensive VQA asks models to combine image understanding with learned knowledge, but comprehensive evidence about how MLMs handle these questions has been limited. This paper evaluates models across commonsense, fine-grained world knowledge, and rationales, and examines visual-knowledge training and composite-image prompting. The evaluation finds substantial world-knowledge challenges alongside gains from composite-image prompting and rationale generation.

  • Problem

    Knowledge-intensive VQA requires combining visual information with learned knowledge, and comprehensive evaluation of MLMs on such information-seeking questions remains limited.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    The paper builds a benchmark across commonsense, fine-grained world knowledge, and decision-making rationales, and explores visual-knowledge-enhanced training and composite-image prompting.

    Source: §1 · p. 2, §7 · p. 16, §8 · p. 19

  • Results

    GPT-4V and other MLMs struggle with fine-grained world-knowledge questions, while composite-image prompting enables GPT-4V to generate more accurate answers and rationales.

    Source: §5.3 · p. 13, §8 · p. 19

  • Takeaways & Limitations

    The evaluation identifies domain-specific model weaknesses and reports that visual-knowledge training and prompting approaches have potential to improve performance.

    Source: §1 · p. 3, §7 · p. 16

    Models show excessive dependence on visual content, which is a drawback when images lack information pertinent to the question.

    Source: §8 · p. 18

Abstract

from arXiv · show

The emergence of multimodal large models (MLMs) has significantly advanced the field of visual understanding, offering remarkable capabilities in the realm of visual question answering (VQA). Yet, the true challenge lies in the domain of knowledge-intensive VQA tasks, which necessitate not just recognition of visual elements, but also a deep comprehension of the visual information in conjunction with a vast repository of learned knowledge. To uncover such capabilities of MLMs, particularly the newly introduced GPT-4V and Gemini, we provide an in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which assesses how well models can understand visual cues and connect to general knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in reasoning out specific knowledge from images, showcasing their proficiency across various specialized fields; 3) Comprehensive Knowledge with Decision-making Rationales, which examines model's capability to provide logical explanations for its inference, facilitating a deeper analysis from the interpretability perspective. Additionally, we utilize a visual knowledge-enhanced training strategy and multimodal retrieval-augmented generation approach to enhance MLMs, highlighting the future need for advancements in this research direction. Extensive experiments indicate that: a) GPT-4V demonstrates enhanced explanation generation when using composite images as few-shots; b) GPT-4V and other MLMs produce severe hallucinations when dealing with world knowledge; c) Visual knowledge enhanced training and prompting technicals present potential to improve performance. Codes: https://github.com/HITsz-TMG/Cognitive-Visual-Language-Mapper

1 Introduction

The paper asks how MLM-based VQA handles information-seeking questions that require combining visual reasoning with learned knowledge. It evaluates models across commonsense, fine-grained world knowledge, and decision-making rationales, finding domain-specific variation and challenges alongside gains from composite-image prompting.

1 Introduction

  • The paper motivates this evaluation by noting that knowledge-intensive VQA requires visual interpretation alongside learned knowledge for information-seeking questions.Source: §1 · p. 2
  • The study evaluates MLMs on a knowledge-intensive VQA benchmark spanning commonsense, fine-grained world knowledge, and decision-making rationales.It compares prompting methods and probes GPT-4V and Gemini using decision-making rationales.Source: §1 · p. 2
  • MLMs show significant variation in understanding and reasoning across commonsense and fine-grained world-knowledge categories.Source: §1 · p. 2
  • GPT-4V and other MLMs face challenges on fine-grained world-knowledge questions, while composite-image reference examples improve answer accuracy and rationale quality.Source: §1 · p. 3

2 Related Work

Prior work established knowledge-based VQA datasets spanning closed and open-domain information, while multimodal large models advanced visual-language tasks. The paper positions comprehensive evaluation of knowledge-intensive VQA as an outstanding gap.

Knowledge-based Visual Question Answering

  • Knowledge-based VQA ranges from closed-knowledge benchmarks to open-domain datasets, including INFOSEEK’s fine-grained entity knowledge and A-OKVQA’s multiple knowledge types and rationales.OK-VQA and INFOSEEK cover extensive knowledge categories and scopes for open-domain VQA evaluation.Source: §2 · p. 3

Multimodal Large Models

  • Multimodal large models combine visual features with language models and are refined on multimodal instruction-following data.The passage names projection layers such as a linear layer or Q-former as ways to integrate visual features.Source: §2 · pp. 3–4

Knowledge-based Visual Question Answering

  • Figure 3 reports the percentage of questions in each of 11 knowledge categories for commonsense and world-knowledge types.Source: Figure 3 · p. 4

Multimodal Large Models

  • Existing multimodal benchmarks had not yet provided a comprehensive evaluation of knowledge-intensive VQA.Source: §2 · p. 4

3 Experimental Setup

The benchmark draws on OK-VQA, INFOSEEK, and A-OKVQA to cover commonsense, fine-grained world knowledge, and decision-making rationales. It evaluates short answers with exact matching and rationales with automatic and human judgments.

3.1 Dataset

  • The benchmark combines OK-VQA, INFOSEEK, and A-OKVQA, sampling 150 questions per knowledge category when available and using the full A-OKVQA validation set for rationale evaluation.For INFOSEEK, selected questions and entities were unseen in training to reduce inadvertent training-data overlap.Source: §3.1 · pp. 4–5
  • The benchmark covers more than ten knowledge categories, mainly commonsense and fine-grained world knowledge, and uses rationales to assess how models apply knowledge in reasoning.Source: §3.1 · p. 5

3.2 Evaluation Method

  • Short-answer accuracy is measured by exact matching against predefined correct answers.Source: §3.2 · p. 5
  • Rationale assessment combines BLEU, CIDER, and METEOR with human judgments of consistency, sufficiency, and factual correctness.Source: §3.2 · p. 5

4 Assessing MLMs in Commonsense Knowledge Q&A

The commonsense evaluation compares GPT-4V and other models across 11 OK-VQA categories using zero-shot and few-shot prompting. GPT-4V outperforms open-source peers across many categories, while composite-image examples can improve its performance and it still makes visual interpretation errors.

4.1.1 Dataset

  • The OK-VQA evaluation uses 1,495 questions across 11 categories, with some categories represented by fewer than 150 samples.Source: §4.1.1 · p. 6

4.1.2 Results and Analysis

  • The four-shot composite-image prompt can improve GPT-4V in categories including Vehicles and Transportation and Objects, Material and Clothing, while few-shot effects vary across models.The authors note that chosen examples may contribute to category-level declines.Source: Figure 4 · p. 6, §4.1.2 · p. 7
  • GPT-4V remains significantly ahead across a wide range of commonsense categories, scoring 70.77 in Science and Technology versus 55.33 in Vehicles and Transportation.Llava-v1.5-13b is the strongest open-source model, but is lower than GPT-4V especially in Science and Technology, Objects, Material and Clothing, and Other.Source: §4.1.2 · pp. 6–7, Table 2 · p. 7

4.2 Conclusion

  • GPT-4V’s responses are often more comprehensive and precise than Llava-v1.5-13b’s, but visual misinterpretations and confusion between similar species remain.Examples include misidentifying an FH16 510 and mistaking a blue songbird.Source: §4.1.2 · p. 7, §4.2 · p. 7, Figure 5 · p. 8

5 Evaluating MLMs on Fine-grained World Knowledge

Fine-grained world-knowledge VQA tests whether models can connect visual clues to specific stored knowledge. The evaluation finds that models remain limited on these questions, despite GPT-4V outperforming open-source MLMs.

5.1 Dataset

  • The INFOSEEK validation subset contained 1,650 examples, sampled as 150 questions from each of 11 knowledge domains.INFOSEEK is grounded in Wikipedia knowledge.Source: §5.1 · p. 8

5.2 Results and Analysis

  • Four INFOSEEK image–question-answer examples served as fine-grained in-context references; composite-image few-shot prompting reduced context-token length, contingent on robust visual comprehension.Some categories declined in accuracy, which the authors link to the specific in-context examples used.Source: §5.2 · pp. 9–10
  • GPT-4V averaged under 30% accuracy, while open-source MLMs did not exceed 15% on fine-grained world-knowledge VQA.The comparison is reported in Table 3.Source: §5.2 · p. 9, Table 3 · p. 9
  • GPT-4V reached 57.33% accuracy on Sport but 10.28% on Animal; Llava-v1.5-13b zero-shot scored 0 on Plant and 26.0% on Vehicle.The authors describe substantial performance gaps across knowledge categories, with lower effectiveness in eight domains.Source: §5.2 · pp. 9–10
  • Case studies identify refusals from insufficient context, confusion between similar objects, weak visual–knowledge integration, and reliance on visual clues over question text.Figures 7–11 show error instances; yellow denotes incorrect responses and green denotes refusals.Source: §5.2 · pp. 10, 13

5.3 Conclusion

  • The authors conclude that models need better integration of detailed visual data with contextual knowledge to improve fine-grained visual object knowledge tasks.They note that images without information pertinent to a question can expose weaknesses in current models.Source: §5.3 · p. 13

6 Comprehensive Knowledge with Decision-Making Rationale

The section evaluates whether multimodal models can answer A-OKVQA questions and generate decision-making rationales. GPT-4V benefits from few-shot prompting, while human evaluation favors its answers and rationales over open-source models.

6.1 Dataset

  • The evaluation uses the A-OKVQA validation set, a 25,000-question benchmark spanning commonsense and world knowledge with training-set rationales.The questions can require knowledge beyond the image; the training subset supplies facts and deductions supporting answers.Source: §6.1 · p. 13

6.2 Results and Analysis

  • GPT-4V with four-shot prompting outperforms other models, and image-based in-context examples further improve its performance.Other MLMs generally cannot generate rationales without rationale-bearing examples; open-source models’ automatic rationale metrics may benefit from their shorter outputs.Source: §6.2 · pp. 13–14, Table 4 · p. 14, Figure 12 · p. 14
  • In human evaluation, GPT-4V excels in answer accuracy and rationale quality compared with open-source MLMs.Four annotators blindly scored 100 randomly selected test examples for answer accuracy, consistency, sufficiency, and factual correctness.Source: §6.2 · pp. 14–15, Table 5 · p. 15
  • GPT-4V’s case-study rationales are more detailed, while Llava-v1.5 sometimes hallucinates visual details and both models can misread images.The figure caption identifies eight examples from GPT-4V and Llava-v1.5-13b; highlighted words mark incorrect or inadequate responses.Source: §6.2 · p. 15, Figure 13 · p. 15

6.3 Conclusion

  • GPT-4V’s few-shot setting improves answer accuracy and rationale quality, while open-source models struggle to generate rationales and lose accuracy under few-shot prompting.Open-source models can achieve answer accuracy comparable to GPT-4V, which the authors attribute primarily to their use of relevant datasets during multimodal instruction tuning.Source: §6.2 · p. 15, §6.3 · p. 16

7 Improving MLMs with Visual Knowledge Enhanced Approaches

The paper proposes visual knowledge-enhanced training and multimodal retrieval-augmented generation to improve MLM performance on knowledge-intensive VQA. Both approaches improve performance, with retrieved-knowledge precision and noise handling affecting reasoning accuracy.

Visual Knowledge Enhanced Training

  • Visual knowledge-enhanced training adds image-associated background knowledge during pretraining and combines a knowledge representation with the image representation.The approach uses image-context descriptions sourced from WiT’s Wikipedia; a four-layer Q-former abstracts visual knowledge into a fixed-length representation.Source: §7 · pp. 16–17

Multimodal Retrieval-Augmented Generation

  • The retrieval approach uses image similarity from a pretrained CLIP visual encoder to retrieve corresponding knowledge from image-knowledge pairs.The retrieved knowledge is used in knowledge-based reasoning alongside the image and question.Source: §7 · p. 18

Comparison and Analysis

  • Both visual knowledge-enhanced training and multimodal retrieval-augmented generation improve VQA performance on commonsense and fine-grained world-knowledge tasks.Retrieved-knowledge precision critically influences reasoning accuracy, and multi-step reasoning mitigates the effects of retrieval noise.Source: Table 6 · p. 17, §7 · p. 18

8 Conclusion and Future Work

The conclusion reports uneven MLM reasoning across knowledge domains and persistent weaknesses in fine-grained world-knowledge questions. It also highlights composite-image prompting for GPT-4V and recommends broader integration of visual-object knowledge in training.

Conclusion and Future Work

  • MLMs’ reasoning varies across knowledge domains, with the greatest difficulty in long-tail, human-challenging entity knowledge.The authors suggest, but do not establish, that imbalanced training data may help explain this disparity.Source: §8 · p. 18
  • GPT-4V and other MLMs show deficiencies in fine-grained world-knowledge QA, including similar-object recognition and integrating visual knowledge with textual queries.The authors also report reluctance to answer when context is insufficient and overreliance on visual clues.Source: §8 · p. 18
  • In the predominantly single-hop evaluation, models often identify depicted objects but miss their broader context, prompting a recommendation to expand visual-language training with object knowledge.The authors suggest this could improve answer accuracy and nuanced visual understanding.Source: §8 · p. 18
  • Composite-image in-context learning improves GPT-4V’s answers and rationales while reducing input tokens, but its effectiveness depends on image-understanding ability.The authors report that open-source MLMs do not excel at understanding information-rich composite images.Source: §8 · p. 19

Related papers

Loading 2311.07536v3…