PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark

Ziming Cheng, Binrui Xu, Lisheng Gong, Zuhe Song, Tianshuo Zhou, Shiqi Zhong, Siyu Ren, Mingxiang Chen, Xiangchao Meng, Yuxin Zhang, Yanlin Li, Lei Ren, Wei Chen, Zhiyuan Huang, Mingjie Zhan, Xiaojie Wang, Fangxiang Feng

arXiv:2506.04280v1cs.CVcs.AI

TL;DR

Existing benchmarks leave multi-image reasoning underexplored because they focus on single-image reasoning or multi-image understanding without reasoning steps. MMRB provides structured, multi-solution reasoning annotations across 92 tasks and evaluates MLLMs and reward models. Open-source models remain limited relative to commercial models, and multimodal reward models face challenges in multi-image scenarios.

  • Problem

    Existing MLLM benchmarks focus either on single-image visual reasoning or multi-image understanding with answer-only annotations, leaving reasoning over multiple images underexplored.

    Source: §1 · p. 2

  • Method

    MMRB combines multi-image understanding and visual reasoning in a 92-task benchmark with human-refined, multi-solution reasoning paths, plus a subset for multi-image reward-model evaluation.

    Source: §1 · p. 2

  • Results

    Open-source MLLMs average below 50% on outcome and process scores, while commercial models average 65% in outcome and 83% in process accuracy.

    Source: §7.1 · p. 6

  • Takeaways & Limitations

    The evaluations indicate that open-source models have limited multi-image reasoning capabilities and that multimodal reward models remain a research opportunity.

    Source: §8 · p. 6

    Most multimodal reward models lack training data specific to multi-image inputs, a setting in which the paper finds reward-task performance challenging.

    Source: §7.2 · p. 6

Abstract

from arXiv · show

With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the $\textbf{Multimodal Multi-image Reasoning Benchmark (MMRB)}$, the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises $\textbf{92 sub-tasks}$ covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on $\textbf{40 MLLMs}$, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.

1 Introduction

Existing benchmarks separate single-image visual reasoning from multi-image understanding without reasoning-step annotations, leaving multi-image reasoning underexplored. MMRB combines these task types, adds structured multi-solution annotations, and reports findings from evaluations of 40 MLLMs.

1 Introduction

  • MMRB addresses the underexplored evaluation of multi-image reasoning, which prior benchmarks cover either through single-image reasoning or multi-image tasks with answer-only annotations.It combines visual reasoning and multi-image understanding in one benchmark.Source: §1 · p. 2
  • MMRB comprises 92 reasoning-specific sub-tasks across spatial, temporal, and high-level semantic reasoning, with GPT-4o-generated reasoning paths refined by human experts.Annotations retain multiple possible thinking paths and solutions; a subset supports multi-image reward-model evaluation.Source: §1 · p. 2
  • The open-source LLM evaluator matches model CoT answers with annotated reasoning steps at sentence level, avoiding costly GPT-based evaluation while retaining high evaluation quality.The section introduces evaluation of 40 commercial and open-source MLLMs, including reasoning-specific and multimodal reward models.Source: §1 · p. 2
  • Model, data, and inference-time scaling improve multi-image reasoning performance, explicit CoT helps both model types, and multimodal reward models show instability in multi-image reward tasks.CoT gains are larger for non-reasoning models than for reasoning-specific models.Source: §1 · p. 2

2 Related Work

Prior visual reasoning benchmarks largely use single images, while multi-image benchmarks generally omit reasoning-step evaluation. MMRB targets both gaps and adds a multi-image reward benchmark.

2.1 Visual Reasoning Benchmarks

  • Most visual reasoning benchmarks focus on single-image scenarios and often use smaller, domain-specific task sets, limiting insight into general reasoning across multiple images.Table 1 contrasts MMRB's 92 reasoning-specific multi-image tasks with prior benchmarks, which have at most 23 sub-tasks.Source: §2.1 · p. 2

2.2 Multi-image Understanding Benchmarks

  • Prior multi-image benchmarks generally do not evaluate grounding evidence in specific images and deriving answers through step-by-step reasoning; MMRB adds intermediate-step evaluation.MMIU [23] aggregates 52 sub-tasks, while MIBench evaluates multi-image in-context learning.Source: §2.2 · p. 3

2.3 Multimodal Reward Model and Benchmarks

  • MMRB's reward subset pairs incorrect AI-generated answers with human-corrected versions to evaluate multimodal reward models in multi-image scenarios.The paper describes this scenario as not previously considered in research.Source: §2.3 · p. 3

3 Pilot Experiment

The pilot experiment sets criteria for selecting reasoning-specific tasks and estimating per-task sample capacity. The benchmark retains semantically driven tasks and uses a uniform sample size of 50.

3 Pilot Experiment

  • The pilot experiment addresses two design choices for a comprehensive and efficient benchmark: reasoning-specific task selection and sample capacity per task.These pilot results guide construction of the full benchmark.Source: §3 · p. 3

3.1 Reasoning Specific Sub-Task Selection

  • Semantic tasks involve reasoning over abstract concepts, whereas perceptual tasks require interpreting low-level visual features; the construction process retains only semantic tasks.The task candidates were categorized as perceptually and/or semantically driven.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Sample Capacity Estimation

  • Pilot estimates indicated minimum sample sizes of 18 for outcome scores and 10 for process scores; the benchmark instead uses 50 samples per task for robustness.The minima were estimated for stable evaluation results based on a 95% confidence interval.Source: §3.2 · p. 3

4 Multimodal Multi-image Reasoning Benchmark

MMRB combines a large multi-image reasoning dataset with human-reviewed, structured reasoning annotations and a derived reward-model evaluation subset. Its construction proceeds through task selection, reasoning annotation, and manual correction.

4.1 Overview of MMRB

  • MMRB contains 4,750 samples and 68,882 reasoning steps across 92 semantic, spatial, and temporal sub-tasks, averaging 6.17 images and 1.93 distinct solutions per sample.It also corrects 355 source ground truths that could otherwise have caused up to a 14% benchmark deviation.Source: §4.1 · p. 3, Table 2 · p. 3

4.2 Data Construction Pipeline

  • The construction pipeline selects reasoning-specific tasks, prompts GPT-4o for three distinct solution paths with typed reasoning steps, and uses human review to correct reasoning and source answers.Human annotators corrected at least one reasoning step in 1,198 samples; 355 initially incorrect final answers were corrected.Source: §4.2 · p. 4, §4.2.1 · p. 4, §4.2.2 · p. 4

4.3 Subset for Multi-image Reward Model

  • The reward-model subset pairs human-corrected accepted annotations with incorrect rejected annotations, using two test sets with randomized accepted/rejected order.The subset contains 2,313 samples.Source: §4.3 · pp. 4–5

5 Evaluation Metrics

The evaluation combines rule-based final-answer scoring with LLM-based assessment of intermediate reasoning. It reports outcome accuracy alongside process and efficacy scores for chain-of-thought outputs.

5.1 Rule-based Evaluation Metrics

  • A rule-based answer extractor identifies an average of 96% of answers, enabling direct calculation of outcome scores.The protocol accepts direct answers or step-by-step responses in an extractable format.Source: §5.1 · p. 5

5.2 LLM-based Evaluation Metrics

  • The LLM-based evaluator matches generated and annotated reasoning directly at the sentence level rather than first dividing responses into steps.The protocol uses Qwen3-32B for improved cost efficiency and interpretability compared with GPT-4o evaluation.Source: §5.2 · p. 5
  • The evaluation reports process and efficacy scores for models’ chain-of-thought outputs.The detailed scoring formula is provided in Appendix B.Source: §5.2 · p. 5

6 Experimental Setup

The baseline covers commercial and open-source MLLMs, reasoning-specific models, and multimodal reward models, with models lacking multi-image support or usability excluded. The setup also reports compute and annotation costs.

6.1 Baseline Models

  • The baseline evaluates 40 MLLMs, including 10 commercial models, 24 open-source models spanning 0.5B–38B parameters, and 8 multimodal reward models.Nine of the open-source models are designed specifically for reasoning.Source: §6.1 · p. 5
  • Models without multi-image input support or with usability issues are excluded from the baseline.LLaVA-o1 is given as an example of a model without multi-image support.Source: §6.1 · p. 5

6.2 Computational Budget

  • Inference uses 8 NVIDIA A800-80GB GPUs; commercial-model evaluation costs approximately $1,344, and GPT-4o annotations use approximately 25.2 million tokens.Table 4 reports results for 8 multimodal reward models on the MMRB reward subset; an asterisk marks score-output mode.Source: §6.2 · p. 5, Table 4 · p. 6

7 Results and Discussion

Open-source MLLMs remain challenged by MMRB, while commercial models score higher but are not flawless. The results also show benefits from inference-time, prompting, and model scaling, alongside instruction-following and reward-model limitations.

7.1 Overall Quantitative Results

  • Open-source models average below 50% on outcome and process scores, while commercial models average 65% in outcome and 83% in process accuracy.Table 3 reports outcome, process, and efficacy scores for 34 baseline models.Source: §7.1 · p. 6
  • Reasoning-specific API models improve outcome scores by an average 6% over non-reasoning counterparts, while average reasoning-process scores remain roughly the same.Source: §7.1.1 · p. 6
  • CoT prompting improves API-model performance by an average 6.4% for non-reasoning models and 1.2% for reasoning-specific models.The passage reports more moderate gains for reasoning-specific models, while noting that explicit CoT can still improve their performance.Source: §7.1.2 · p. 6
  • Scaling yields average outcome-score gains of 5% for Qwen and 3% for InternVL, plus a 20% process-score gain for InternVL; performance generally improves with model size within families.The passage also notes performance differences across model families, potentially related to training strategies and datasets.Source: §7.1.3 · p. 6
  • For non-reasoning models, CoT efficacy averages 1.3 for open-source models above 8B and 3.3 for API models, versus -1.4 for smaller models.Each 1B increase in model size corresponds to about 0.095 more efficacy points (p = 0.028; R2 = 0.231), as shown in Figure 4.Source: §7.1.4 · p. 6, Figure 4 · p. 6
  • Smaller open-source models often fail to follow answer-format or step-by-step instructions, lowering their process scores; Qwen2.5-Omni also often omits reasoning steps.Source: §7.1.5 · p. 6

7.2 Multimodal Reward Model Evaluation

  • Reward models average 46.5% ranking accuracy on Test Set 1 and 38.8% on Test Set 2; reversing sample order drops GPT-4o-mini from 85.69% to 10.20%.Skywork-VL-Reward and IXC-2.5-Reward output scores rather than comparisons; Gemini-2.0 is the only model described as stable.Source: §7.2 · p. 6
  • Most multimodal reward models lack training data specific to multi-image inputs, a constraint relevant to their performance in multi-image scenarios.Source: §7.2 · p. 6

8 Conclusion

The conclusion presents MMRB as a benchmark for general-purpose multi-image reasoning and summarizes baseline findings on open-source models and multimodal reward models.

8 Conclusion

  • MMRB contains 92 multi-image-only subtasks and over 68,000 reasoning steps, and is presented as the first benchmark for general-purpose multi-image reasoning scenarios.Source: §8 · p. 6
  • Baseline evaluations of 40 API and open-source MLLMs suggest that current open-source models have limited multi-image reasoning capabilities.Source: §8 · p. 6
  • The conclusion identifies multimodal reward models as an area with significant potential for future research.Source: §8 · p. 6

A Minimum Sample Capacity Estimation

The section estimates a minimum sample size per sub-task by measuring accuracy stability across repeated samples. It sets a uniform sample size of 50 after applying a 95% confidence-level and 4% confidence-interval-width target.

A Minimum Sample Capacity Estimation

B LLM-based Evaluation Metrics

The evaluation scores reasoning trajectories by matching annotated steps to answer phrases at sentence level and semantic reasoning-type level. It also includes an efficacy calculation and describes model inference and prompting settings.

B.1 Precision Score

  • A reasoning step counts as correct when an answer phrase matches it at sentence level and conveys the same reasoning-type operation.Trajectory precision is the proportion of correctly recovered reasoning steps.Source: §B.1 · pp. 9–10

B.2 Efficacy Score

  • The process score for a task is defined using three candidate reasoning trajectories, while efficacy follows the calculation used in MME-CoT [15].The supplied passages do not state the efficacy formula.Source: §B.1 · p. 10, §B.2 · p. 10

C Implementation Details

  • Models receive either answer-only or step-by-step prompts, and inference follows model-recommended multi-image settings where available.Open-source inference uses Transformers; commercial models are accessed through official APIs, with one-fifth of data used for expensive reasoning models.Source: C Implementation Details · p. 10

D Error Analysis

The error analysis of OpenAI o1 responses identifies several distinct sources of errors, including reasoning, image grounding, instruction following, and acceptable answer variation.

D Error Analysis

  • About a third of 282 OpenAI o1 response errors involve incorrect reasoning; image grounding or image-message misunderstandings are another major source.About 15% involve failure to follow instructions, while about 18% are considered acceptable by some annotators despite differing from the ground truth.Source: D Error Analysis · p. 10, Figure 7 · p. 10

F Data Sources

E Data Samples

H.1 Reasoning Step Generation by GPT-4o

  • The GPT-4o generation prompt requests three distinct step-by-step reasoning paths in JSON format.Each reasoning step includes a step index and reasoning type.Source: §H.4 · p. 14, Figure 10 · p. 14

H.2 Reasoning Steps Evaluation by Qwen3-32B

  • Figure 11 contains the prompt template for evaluating reasoning steps with Qwen3-32B.The supplied text does not specify the evaluation criteria.Source: §H.2 · p. 10, Figure 11 · p. 15

H.3 Reward Model Evaluation for API Models

H.4 Reward Model Evaluation for Open-source Reward Models

  • The open-source reward-model prompts include llava-critic criteria for object, relationship, and attribute accuracy, plus helpfulness.Figure 16 provides the R1-reward evaluation prompt template.Source: §H.4 · p. 19, Figure 16 · p. 20
Loading 2506.04280v1…