PaperlayerView exact revision on arXiv

Source-linked AI summary

II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models

Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, Hongquan Lin, Jiaming Li, Yuansheng Ni, Haihong Wu, Yaswanth Narsupalli, Zhigang Zheng, Chengming Li, Xiping Hu, Ruifeng Xu, Xiaojun Chen, Min Yang, Jiaheng Liu, Ruibo Liu, Wenhao Huang, Ge Zhang, Shiwen Ni

arXiv:2406.05862v3cs.CLcs.AIcs.CV

TL;DR

Higher-order perceptual capabilities of MLLMs, including nuanced emotional understanding and deeper meaning extraction, remain underexplored. II-Bench evaluates image implication understanding, and its results show a human–model accuracy gap, difficulty with abstract images, and generally improved scores with sentiment hints.

  • Problem

    Higher-order image perception—including nuanced emotional understanding and profound meaning extraction—has received limited evaluation despite the need to assess MLLMs’ broader capabilities.

    Source: §1 · p. 2

  • Method

    II-Bench evaluates MLLMs’ higher-order perception, reasoning, and comprehension using complex implication images and multiple-choice questions.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    74.8% is the highest MLLM accuracy, compared with 90% average human accuracy and a 98% human peak; models perform worse on abstract, complex images, while sentiment hints generally improve scores.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Current MLLMs handle surface image content better than image implication, and sentiment-prompt gains suggest insufficient emotional understanding.

    Source: §5 · p. 9

    Subjective elements may produce varying interpretations, the benchmark covers only part of human knowledge, and its metrics may not fully reflect advanced AI understanding and reasoning.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The rapid advancements in the development of multimodal large language models (MLLMs) have consistently led to new breakthroughs on various benchmarks. In response, numerous challenging and comprehensive benchmarks have been proposed to more accurately assess the capabilities of MLLMs. However, there is a dearth of exploration of the higher-order perceptual capabilities of MLLMs. To fill this gap, we propose the Image Implication understanding Benchmark, II-Bench, which aims to evaluate the model's higher-order perception of images. Through extensive experiments on II-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMs attains 74.8%, whereas human accuracy averages 90%, peaking at an impressive 98%. Subsequently, MLLMs perform worse on abstract and complex images, suggesting limitations in their ability to understand high-level semantics and capture image details. Finally, it is observed that most models exhibit enhanced accuracy when image sentiment polarity hints are incorporated into the prompts. This observation underscores a notable deficiency in their inherent understanding of image sentiment. We believe that II-Bench will inspire the community to develop the next generation of MLLMs, advancing the journey towards expert artificial general intelligence (AGI). II-Bench is publicly available at https://huggingface.co/datasets/m-a-p/II-Bench.

1 Introduction

Although MLLMs have advanced across multimodal tasks, higher-order image perception remains underexplored. II-Bench evaluates this capability and reveals a substantial human–model performance gap, weaker performance on abstract images, and gains from sentiment hints.

1 Introduction

  • The benchmark addresses a gap in evaluating nuanced emotional understanding and deeper meaning extraction, which are needed to interpret images that convey emotions and cultural context.Abstract and complex images pose a challenge because models must understand conveyed emotions and infer creators’ intended meanings.Source: §1 · p. 2
  • II-Bench evaluates higher-order perception, reasoning, and comprehension across 1,222 images spanning six domains and varied image categories.The domains include life, art, society, psychology, environment, and others; categories include illustrations, memes, posters, comics, logos, and paintings.Source: §1 · p. 2, Figure 2 · p. 2
  • 74.8% is the highest MLLM accuracy, versus 90% average human accuracy and a human peak of 98%; models also struggle more with abstract, complex images, while sentiment hints generally improve scores.The benchmark experiments cover 20 MLLMs.Source: §1 · p. 2
  • The authors hope II-Bench will encourage development of next-generation MLLMs toward expert artificial general intelligence.Source: §1 · p. 2

2 Related Work

Prior work has developed MLLM architectures and benchmarks for multimodal tasks, but image implication understanding remains comparatively underexplored. II-Bench is presented as a benchmark for evaluating image understanding and reasoning about implications.

2.1 Multimodal Large Language Models

  • MLLM architectures such as BLIP-2 and LLaVA connect visual representations to language models, while preserving the language backbones’ capabilities.The cited examples use a Q-Former and an MLP connector, respectively.Source: §2.1 · p. 3

2.2 MLLM Benchmarks

  • Existing benchmarks span common-sense questions and specialized exam or textbook content, alongside earlier single-task VQA and image-captioning evaluations.Source: §2.2 · p. 3
  • II-Bench targets image implication understanding, a task requiring multi-hop reasoning and theory of mind beyond simpler image understanding or knowledge tests.The passage notes concerns that some benchmark questions can be answered without images and that training data leakage is a risk.Source: §2.2 · p. 3

3 The II-Bench

II-Bench is a manually curated benchmark of complex images and fine-grained questions designed to assess image implication understanding. Its collection, filtering, annotation, and dataset statistics define the benchmark’s scope and composition.

3.1 Overview of II-Bench

  • II-Bench tests higher-order perception and reasoning on implication-rich images; its 1,222 images carry 1,434 multiple-choice questions about metaphors, symbolism, and visual details.The questions have six answer options, with one correct answer.Source: §3.1 · p. 3

3.2 Data Curation Process

  • Curation begins with 20,150 raw images and removes over 90% through deduplication, text-ratio screening, and human review before annotators add labels, implication explanations, and questions.Images are screened for pertinent metaphorical or suggestive implications; annotation includes one to three questions per image.Source: §3.2 · p. 4

3.3 Dataset Statistics

  • Images are classified by six domains, image type, difficulty, emotional tone, and rhetorical device, including metaphor, symbolism, contrast, and analogy.Source: §3.3 · p. 5

Statistics

  • 708 questions are easy, 385 medium, and 129 hard; the dev, validation, and test split is 15:20:1,187.The benchmark reports average question, option, and explanation lengths of 16.91, 14.05, and 170.47 words, respectively.Source: Statistics · p. 5

4 Experiment

The experiments compare MLLMs and humans on II-Bench across models, image domains, emotions, and prompting conditions. Results show a substantial human–model gap, weaker performance on abstract or emotionally negative content, and benefits from emotion-label prompts.

4 Experiment

  • Experiments evaluated open- and closed-source MLLMs under eight uniform prompting settings, including 1–3-shot, zero-shot, CoT, Domain, Emotion, and Rhetoric.Emotion prompts supplied image emotional polarity; Domain and Rhetoric prompts supplied domain and rhetorical-device information, respectively.Source: §4 · p. 5

4.1 Baselines

  • Table 2 describes baseline architectures and sizes; accuracy is the main metric, and responses without extractable answer options count as incorrect.Human performance was also assessed.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2.1 Gap between Humans and MLLMs

  • 90.3% average human accuracy (98.2% highest) exceeds Qwen-VL-MAX’s 74.8%, the best closed-source score, and LLaVA-1.6-34B’s 73.8%, the best open-source score.Table 3 reports results across models, humans, domains, and emotions.Source: Table 3 · p. 6, §4.2.1 · p. 7

4.2.2 Disparity between Open-source and Closed-source Models

  • Closed-source models generally performed better, but top open-source models exceeded GPT-4V’s 65.9% accuracy while remaining below GPT-4o’s 72.6%.The best open-source model scored 73.8%, compared with 74.8% for the best closed-source model; the authors also relate weaker performance to multi-hop reasoning, with InstructBLIP-T5-XL lowest at 47.3%.Source: §4.2.2 · p. 7

4.2.3 Model Performance across Different Domains and Emotions

  • Models performed better in Environment, Other, Life, and Society than in Art and Psychology, and accuracy was higher for positive-emotion metaphors than negative-emotion images.The authors suggest a possible positive-response bias in models and note that this differs from humans’ greater sensitivity to negative implications.Source: §4.2.3 · p. 7

4.2.4 Analysis on different prompt skills

  • CoT did not significantly improve accuracy and sometimes reduced it, while Emotion labels improved accuracy most, followed by similarly effective Domain and Rhetoric labels.For example, CogVLM2-Llama3-Chat-19B fell from 70.3% without CoT to 69.3% with CoT, and InternVL-Chat-1.5 fell from 66.3% to 63.3%. Manual checks found answer-format failures, including 15.8% more regex-mismatched BLIP-2 FLAN-T5-XXL responses under CoT than direct answering.Source: §4.2.4 · pp. 7–8
  • Few-shot examples did not improve accuracy, which tended to decline as the number of shots increased.The authors attribute the decline to weaker multi-image and long-context capabilities; Qwen-VL-Max produced errors when inputs exceeded 6,000 tokens.Source: §4.2.4 · p. 8, Table 5 · p. 9

4.3 Error Analysis

  • 36% of GPT-4V errors were metaphorical misunderstandings and 21% were detail misunderstandings; other listed categories ranged from 1% answer extraction error to 15% surface-level interpretation.The analysis used expert annotators to examine 100 errors per domain, sampled in proportion to domain representation. Metaphorical errors included incorrect theme interpretation or missing specific knowledge; detail errors could affect image-meaning interpretation.Source: §4.3 · pp. 8–9, Figure 4 · p. 9

5 Conclusion

II-Bench results indicate a large gap between MLLMs and humans in understanding image implications, alongside weaknesses in image-emotion understanding. The authors hope the benchmark will spur next-generation multimodal models toward expert AGI.

5 Conclusion

  • Current state-of-the-art MLLMs understand surface image content well, but still lag far behind humans in understanding image implications.The authors frame this as a gap in higher-order theory of mind.Source: §5 · p. 9
  • Including image emotional-polarity information in prompts usually improves model scores, suggesting insufficient emotional understanding contributes to misinterpretation of implied meaning.Source: §5 · p. 9
  • Humans understand neutral and negative emotions better than models, while most MLLMs handle positive emotions better than neutral or negative ones.The authors suggest MLLM training data may be skewed toward positive emotions.Source: §5 · p. 9
  • The authors hope II-Bench will stimulate development of next-generation multimodal foundation models toward expert AGI.Source: §5 · p. 9

Limitations

The authors identify limits involving interpretive consistency, domain coverage, and the extent to which the evaluation metrics capture advanced AI understanding and reasoning.

Limitations

  • Subjective elements in II-Bench can lead to varying interpretations and potentially affect result consistency.Source: Limitations · p. 10
  • II-Bench covers specific domains that represent only a portion of human knowledge.Source: Limitations · p. 10
  • The evaluation metrics may not fully reflect advanced AI systems’ sophisticated understanding and reasoning abilities.Source: Limitations · p. 10

Ethics Statement

The ethics statement describes the authors’ commitments to lawful, fair, and responsible data management, including privacy protections and monitoring for bias. Figure 5 presents statistics about II-Bench image types and domains.

Ethics Statement

  • The authors say II-Bench development follows ethical and legal guidelines, emphasizing fairness, transparency, inclusivity, and responsible data management.Source: Ethics Statement · p. 10
  • The authors report anonymizing personal data and minimizing harmful or biased content, while acknowledging that bias or potentially offensive information may remain.Source: Ethics Statement · p. 10

A Image type and domain statistics

  • Figure 5 presents II-Bench image-type and domain statistics.Source: Figure 5 · p. 14

B Data Annotation Protocol

The data annotation protocol concerns a dataset of questions that explore the metaphorical implications of images.

B Data Annotation Protocol

B.1 Data Collection

The guidelines narrow the benchmark to images with metaphorical implications and specify annotation fields and question construction.

B.2 General Guidelines

  • Images without metaphorical implications are excluded, and annotations are expected to be accurate and consistent.Source: §B.2 · p. 14
  • Each image is labeled by image type, difficulty, domain, emotion, and one or more rhetorical devices.Source: §B.2 · p. 15
  • Each image receives a human explanation and 1–3 fine-grained questions, each with one correct answer and five distractors; all text is in English.Source: §B.2 · pp. 14–15

B.3 Data Quality Assurance

The section describes annotation quality checks and ethical safeguards, then lists prompt variants for none, few-shot, keyword, and chain-of-thought settings.

B.3 Data Quality Assurance

  • Each question was manually validated by at least three annotators, with inconsistencies resolved by consensus across four rounds of data checks.Source: §B.3 · p. 15

B.4 Ethical Considerations

  • Data collection excludes sources that prohibit copying or redistribution, and annotators must avoid collecting personal information.Source: §B.4 · p. 15

C.1 None

  • The none setting asks the model to answer a single-answer multiple-choice question from the provided picture.Source: §C.1 · p. 15

C.2 Few-shot

  • Few-shot prompts provide examples with answers and corresponding pictures before asking the model to answer.Source: §C.2 · p. 15

C.3 Keywords

  • The keyword setting adds one of three cues—emotion, domain, or rhetoric—to the prompt.Source: §C.3 · pp. 16–17

C.4 CoT

  • The CoT prompt asks the model to think through the options step by step.Source: §C.4 · p. 17

D Results on Different Types, Difficulties and Rhetoric

The results section compares MLLMs and humans across image types, difficulty levels, and rhetorical categories.

D Results on Different Types, Difficulties and Rhetoric

D.1 Image Types and Difficulty

  • Table 6 presents MLLM and human results by image type and difficulty level, marking the best and second-best model in each category.Source: §D.1 · p. 17, Table 6 · p. 18

D.2 Rhetoric

  • Table 7 presents MLLM and human results by rhetorical device, marking the best and second-best model in each category.Source: §D.2 · p. 17, Table 7 · p. 18

E Additional Details of Results

The additional results details report accuracy, error, and miss statistics across model settings, including how response-format failures are counted.

E Additional Details of Results

F Other Errors

This section examines GPT-4V’s other errors through an analysis of error types and examples, including 49 error examples and 28 correct examples. Its case studies span society, psychology, art, environment, and Others, presenting correct and erroneous interpretations of visual symbolism and implied meaning.

F Other Errors

  • 11% of errors were Detail Ignorance, in which GPT-4V overlooked crucial image elements that could convey metaphorical meaning.The passage notes that using all image details can help uncover hidden meanings.Source: F Other Errors · p. 21
  • 15% of errors were Surface-Level Interpretation, as GPT-4V focused on superficial image information and missed deeper meanings.For example, it noticed the “Internet” but missed the meme’s emotional value.Source: F Other Errors · p. 21
  • 12% of errors were Reasoning Error: GPT-4V sometimes understood image content but failed at complex logical or mathematical problem-solving.The passage attributes these failures to limited capability with intricate logic and mathematics.Source: F Other Errors · p. 21
  • 4% of errors were Reject to Answer, when GPT-4V judged the image and question insufficient or withheld an answer over potentially harmful content.The safety-driven withholding can frustrate users expecting a response.Source: F Other Errors · p. 21
  • 1% of errors were Answer Extraction Error, where weak instruction-following led GPT-4V to produce an answer in the wrong format for regularized extraction.Source: F Other Errors · p. 21

G Case Study

  • The appendix’s GPT-4V sample analysis covers 49 error examples and 28 correct examples.Source: G Case Study · p. 22

F Other Errors

List of Case Study Figures

Loading 2406.05862v3…