PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, Heng Wang, Hongxia Yang

arXiv:2311.11567v3cs.CV

TL;DR

Existing MLLM benchmarks often use basic reasoning tasks and simple responses, leaving complex reasoning difficult to assess. InfiMM-Eval provides manually curated open-ended questions across three reasoning paradigms and scores both answers and intermediate steps. The paper reports an overall score of 74.44 for GPT-4V, while noting that the evaluation protocol still needs deeper examination of reasoning processes.

  • Problem

    Existing MLLM benchmarks often focus on basic reasoning and simple responses, and their categories and formats can make clear assessment of complex reasoning difficult.

    Source: §1 · p. 2

  • Method

    InfiMM-Eval is a manually curated open-ended visual QA benchmark covering deductive, abductive, and analogical reasoning, with scoring for final answers and intermediate steps.

    Source: §1 · pp. 2–3

  • Results

    GPT-4V attained an overall score of 74.44 and a score of 58.98 on more challenging subsets in the paper’s reported findings.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The benchmark’s rubric gives partial credit for relevant, logical intermediate steps, allowing evaluation to consider reasoning alongside answer correctness.

    Source: §1 · p. 3

    The current overall reasoning score does not provide an in-depth examination of the reasoning process, which the authors say is needed for more interpretable results.

    Source: §6 · p. 11

Abstract

from arXiv · show

Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at https://infimm.github.io/InfiMM-Eval/

1 Introduction

Existing MLLM benchmarks leave open how to clearly assess complex visual reasoning, while multiple-choice or basic tasks may not sufficiently challenge advanced models. InfiMM-Eval addresses this with manually curated questions across three reasoning paradigms and an evaluation protocol that scores both answers and intermediate steps.

Motivation

  • InfiMM-Eval targets a gap in evaluating MLLM reasoning: existing benchmarks may use inconsistent categories or basic and multiple-choice tasks that do not sufficiently challenge advanced models.The paper argues that these formats make it difficult to gain clear insights into MLLMs’ reasoning capabilities.Source: §1 · p. 2

Benchmark contribution

  • The benchmark uses manually curated open-ended visual questions covering deductive, abductive, and analogical reasoning, with sequential reasoning steps for each question.The paper describes it as the first multimodal open-ended QA benchmark to incorporate detailed reasoning steps.Source: §1 · pp. 2–3

Evaluation protocol

  • Responses receive full marks for a correct answer and partial scores based on the relevance and logic of their intermediate reasoning steps.An LLM-based evaluator applies this rubric to open-ended responses with reasoning steps.Source: §1 · p. 3

Evaluation

  • The paper reports ablation studies using InfiMM-Eval to evaluate the reasoning capabilities of representative MLLMs.Source: §1 · p. 3

2 Related work

Related work reviews multimodal model designs and evaluation benchmarks, then frames reasoning through logical categories. It identifies a need for a more unified account of MLLM reasoning capabilities.

2.1 Multi-modal LLMs

  • Multimodal models span approaches including visual-language alignment, instruction tuning, adapters, and multimodal chain-of-thought.Examples include Flamingo, BLIP-2, Otter, and Multimodal-CoT.Source: §2.1 · p. 3

2.2 MLLM evaluation benchmarks

  • Existing evaluation benchmarks cover varied reasoning tasks, but multiple-choice formats can make questions unnatural, provide hints, and leave reasoning processes unmeasured by final-answer-only scoring.The review also notes that open-ended QA requires alternatives to traditional text-generation metrics and costly human evaluation.Source: §2.2 · p. 3

2.3 Reasoning in MLLMs

  • The authors identify the lack of a systematic, unified framework for MLLM reasoning, whose task-specific categories may obscure a holistic view of capabilities.They advocate classification by deductive, abductive, and analogical reasoning, and introduce InfiMM-Eval as an open-ended QA benchmark organized around these paradigms.Source: §2.3 · p. 4
  • Deduction derives conclusions from premises, abduction selects a plausible explanation for observations, and analogy transfers knowledge from known cases to similar situations.Source: §2.3 · p. 4

3 InfiMM-Eval benchmark

InfiMM-Eval is a manually curated benchmark for multi-step MLLM reasoning, with questions classified as deductive, abductive, or analogical. Its collection process applies validation criteria, and the resulting dataset contains 279 questions with varied reasoning complexity and step counts.

3.1 Data collection

  • The benchmark was manually created for multi-step reasoning and organizes questions into deductive, abductive, and analogical categories.The collection process classifies each sample into one of the three reasoning types.Source: §3.1 · pp. 4–5
  • Eight annotators sourced images and created intricate questions and answers intended to require multi-step reasoning rather than immediate visual observation.Questions allowed open-ended answers but were required to follow a single logic path; overly subjective questions were excluded.Source: §3.1 · pp. 4–5
  • After review cycles and quality-control checks, the benchmark collection was refined to 279 samples.Checks covered image relevance, logical consistency, complexity, subjectivity, and question-answer discrepancies; question formats were also diversified.Source: §3.1 · pp. 5–6

3.2 Dataset statistics

  • The dataset contains 279 questions associated with 342 images, including 25 images adopted from MM-Vet.Source: §3.2 · p. 6
  • 212 of 279 questions (about 76%) require three or more reasoning steps; the dataset includes 181 deductive, 49 abductive, and 49 analogical questions, with 108 high- and 171 moderate-complexity questions.It also contains 23 counter-intuitive questions; high-to-moderate complexity is approximately 1:2 for deductive and abductive questions and closer to 1:1 for analogical questions.Source: §3.2 · p. 6
  • The image collection’s visual content categories were assigned by GPT-4V, and a single image may belong to multiple categories.Source: §3.2 · p. 6, Figure 3 · p. 6

4 Experiments

The experiments score MLLMs using both answer correctness and intermediate reasoning, then compare models across reasoning types and complexity levels. GPT-4V leads overall; CoT, in-context examples, and model scale produce different performance patterns across models.

4.1 Evaluation protocol

  • The protocol combines answer correctness with the relevance and logic of intermediate reasoning, assigning full marks for directly correct answers and partial scores for reasoning steps.Question scores range from 0 to 1, and the overall score accounts for Moderate and High reasoning-complexity groups.Source: §4.1 · pp. 6–7

4.2.1 Overall results

  • GPT-4V leads across reasoning domains and complexities with 77.44 overall, while Qwen-VL-Chat leads open-source models at 37.39; open-source models especially lag in analogical reasoning.Instruction-fine-tuned models also outperform solely pretrained counterparts in the reported comparisons.Source: §4.2.1 · pp. 7–8, Table 1 · p. 8

4.2.2 Results with chain-of-thought prompt

  • Adding “Let’s think step by step” improves GPT-4V’s score by 3.7, while open-source models generally show minimal performance differences.The authors hypothesize that model size and instruction-finetuning data quality contribute to this variation.Source: §4.2.2 · p. 8, Table 2 · p. 8

4.2.3 Results with in-context learning

  • Randomly selected in-context examples do not enhance GPT-4V’s performance and may slightly impair it, while Otter and Qwen-VL-Chat show marginal improvements.The examples are drawn from the dataset and added to prompts at inference time.Source: Table 3 · p. 8, §4.2.3 · p. 9

4.2.4 Results with LLMs of varied sizes

  • Increasing Qwen-VL’s language-model size from 7B to 14B raises its overall reasoning score from 35.32 to 37.39.The paper describes LLM dimension as a critical determinant of MLLM reasoning capability.Source: §4.2.4 · p. 10, Table 4 · p. 10
  • GPT-4 prompted with only a question scores close to 0, while CoT raises SOLAR-0-70B’s score from 48.71 to 55.59 using GPT-4V-generated image descriptions.CoT does not produce proportionate gains in the smaller 7B and 13B models.Source: §4.2.4 · p. 10
  • GPT-4 performs best with CoT and GPT-4V-generated image descriptions, and its performance falls with LLaVA-1.5 descriptions.The authors identify OCR and extensive commonsense information in GPT-4V’s descriptions as important to standalone LLM multi-modal reasoning.Source: §4.2.4 · p. 10

5 Conclusion

InfiMM-Eval evaluates complex reasoning in MLLMs using questions, answers, and detailed reasoning steps, and reports GPT-4V as the overall leader.

5 Conclusion

  • GPT-4V attains an overall score of 74.44 and 58.98 on more challenging subsets, while top-performing open-source MLLMs still fall markedly behind.The benchmark uses GPT-4 to assess open-ended answers and intermediate reasoning procedures.Source: §5 · p. 10

6 Limitations

The authors identify limits in the benchmark’s reasoning-category coverage and call for a more in-depth evaluation of reasoning processes.

Expanding reasoning categories

  • The benchmark initially examines deductive, abductive, and analogical reasoning, while human reasoning includes additional forms.Future iterations aim to encompass a broader range of reasoning categories for more comprehensive assessment.Source: §6 · pp. 10–11

Enhancing evaluation protocol

  • The current protocol incorporates intermediate reasoning steps but produces an overall reasoning score; the authors call for deeper examination of the reasoning process.They expect this to provide deeper insight into model capabilities and make results more interpretable and accessible to human understanding.Source: §6 · p. 11

A Counter-intuitive examples

The appendix points readers to additional counter-intuitive examples from InfiMM-Eval.

A Counter-intuitive examples

B Model inference prompts

The appendix lists model-specific inference prompts and describes the chain-of-thought prompt addition.

B Model inference prompts

C Additional ablation study

The section lists additional ablation studies on InfiMM-Eval and includes a table describing evaluation prompt placeholders.

C.1 Multi-Images as input results

The study compares feeding multiple images separately with concatenating them horizontally. Results differ across the three models tested, with GPT-4V showing a substantial drop after concatenation.

C.1 Multi-Images as input results

  • The comparison feeds each image separately or concatenates multiple images horizontally, using a 47-sample subset with 27 moderate- and 20 high-complexity questions.Source: §C.1 · p. 17, Table 6 · p. 17
  • GPT-4V drops substantially after image concatenation, EMU shows no evidence of improvement, and Fuyu-8B performs poorly.The authors suggest EMU's instruction-tuning data may lack multi-image samples; Fuyu-8B is pretrained only and does not follow instructions well.Source: §C.1 · p. 17
  • For models that process images as patches, horizontal concatenation might affect input patches and lead to worse performance.Source: §C.1 · p. 17

Related papers

Loading 2311.11567v3…