PaperlayerView exact revision on arXiv

Source-linked AI summary

MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence

Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, Jiangmiao Pang

arXiv:2505.23764v3cs.CVcs.CL

TL;DR

Existing benchmarks largely miss the multi-image spatial reasoning required for real-world MLLM deployment. MMSI-Bench addresses this gap with a human-built benchmark and annotated reasoning, and its evaluation exposes a large model–human performance gap while enabling automated failure diagnosis.

  • Problem

    Most existing benchmarks test single-image spatial understanding, despite real-world MLLMs needing to reason across multiple images.

  • Method

    MMSI-Bench uses a human-designed multiple-choice VQA benchmark spanning ten spatial task types and multi-step reasoning, with expert-annotated reasoning processes.

  • Results

    37 MLLMs were evaluated, with GPT-5 achieving 41.9% average accuracy versus more than 97% for humans.

  • Takeaways & Limitations

    The benchmark shows substantial headroom in MLLM multi-image spatial intelligence and supports scalable diagnosis of grounding, scene-reconstruction, transformation, and spatial-logic errors.

  • Takeaways & Limitations

    Manual annotation limits scalability because the benchmark requires substantial human effort; expansion may be needed when performance saturates.

Abstract

from arXiv · show

Spatial intelligence is essential for multimodal large language models (MLLMs) operating in the complex physical world. Existing benchmarks, however, probe only single-image relations and thus fail to assess the multi-image spatial reasoning that real-world deployments demand. We introduce MMSI-Bench, a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision researchers spent more than 300 hours meticulously crafting 1,000 challenging, unambiguous multiple-choice questions from over 120,000 images, each paired with carefully designed distractors and a stepwise reasoning process. We conduct extensive experiments and evaluate 37 open-source and proprietary MLLMs, observing a wide gap: the strongest open-source model attains roughly 30% accuracy and OpenAI's GPT-5 reasoning model reaches 40%, while humans score 97%. These results underscore the challenging nature of MMSI-Bench and the substantial headroom for future research. Leveraging the annotated reasoning processes, we also provide an automated error analysis pipeline that diagnoses four dominant failure modes, including (1) grounding errors, (2) overlap-matching and scene-reconstruction errors, (3) situation-transformation reasoning errors, and (4) spatial-logic errors, offering insights for advancing spatial intelligence. Project page: https://runsenxu.com/projects/MMSI_Bench .

1 INTRODUCTION

MMSI-Bench addresses the lack of rigorous benchmarks for multi-image spatial reasoning, which real-world MLLM deployment requires. Its human-designed evaluation reveals a large gap between current models and humans while providing scalable failure diagnosis.

  • Motivation: Existing benchmarks mostly test single-image, straightforward spatial relations, whereas real-world understanding requires reasoning across images and across non-co-occurring entities.Multi-image reasoning supports tracking object and ego motion and relating entities that never appear together in one frame.
  • Benchmark: MMSI-Bench evaluates multi-image spatial intelligence through ten fundamental task types spanning camera, object, and region positions, attributes, and motions, plus multi-step questions.The benchmark is presented as a multiple-choice VQA benchmark with a long-horizon reasoning split.
  • Results: 37 MLLMs were evaluated, with the strongest open-source model reaching about 30% accuracy, GPT-5 reaching 40%, and humans reaching 97%.The reported results show a pronounced model–human performance gap on MMSI-Bench.
  • Analysis: An automated error-analysis pipeline compares model responses with expert reference reasoning to diagnose grounding, overlap-matching and scene-reconstruction, situation-transformation, and spatial-logic errors.The pipeline is intended to provide actionable insights for future multi-image spatial-intelligence research.

2 RELATED WORK

Related benchmarks have expanded multi-image evaluation beyond single-image reasoning, but they target varied skills and remain challenging for contemporary MLLMs. MMSI-Bench focuses specifically on multi-image spatial intelligence.

  • Multi-image VQA benchmarks: Visual Haystacks, MIBench, ReMI, and MuirBench evaluate multi-image abilities including long-context retrieval, cross-image reasoning, multimodal tasks, robustness, and diversity.These benchmarks differ in the skills and settings they emphasize.
  • Multi-image VQA benchmarks: Contemporary MLLMs still trail humans as the number of images or reasoning complexity increases.This pattern is reported across existing multi-image benchmarks.
  • Positioning: MMSI-Bench differs from general multi-image benchmarks by comprehensively evaluating MLLMs’ multi-image spatial intelligence.Its scope is spatial reasoning rather than a broad collection of multi-image skills.
  • Spatial-intelligence benchmarks: Most spatial-intelligence benchmarks for MLLMs assess single-image spatial understanding rather than multi-image spatial reasoning.MMSI-Bench is positioned as a focused evaluation of the latter capability.

3 MMSI-BENCH

MMSI-Bench defines multi-image spatial reasoning as integrating views or moments to reconstruct scenes and answer complex questions. It operationalizes this idea through a taxonomy, manually constructed questions, annotated reasoning, and quality control.

  • Concept: Multi-image spatial reasoning combines information from multiple images to reconstruct an underlying scene and answer complex spatial questions.The concept reflects spatial inference from different viewpoints or moments in time.
  • Taxonomy: The taxonomy organizes tasks around camera, object, and region, covering their positional relationships, attributes, and potential motion.Camera denotes the scene-capturing agent, object denotes other entities, and region denotes semantic areas such as a kitchen.
  • Dataset: The benchmark contains 1,000 multiple-choice question–answer pairs based on multiple images, each accompanied by an annotated reasoning process.These annotations explain the answer for every question.
  • Construction: The construction pipeline collects real-world images, selects relevant image sets, manually annotates challenging QA and reasoning processes, and applies independent quality control.Reviewers remove ambiguous, incorrect, or single-image- or common-sense-solvable questions, and difficulty is annotated using human answering time.
  • Dataset: Every question requires multiple images, with two images used in all categories except Multi-Step Reasoning.Questions are designed not to be answerable from a single image alone.

4 EVALUATION ON MMSI

MMSI-Bench evaluates 37 MLLMs under standardized conditions, revealing substantial performance gaps and limited benefits from scaling, spatial fine-tuning, and prompting.

  • 4.1 EVALUATION SETUP: 37 MLLMs are evaluated using temperature zero, a maximum output length of 2,048 tokens, exact-match accuracy, and three comparison baselines.The baselines are Random Guessing, Human Performance, and Blind GPT-4o.
  • 4.2 MAIN RESULTS: 41.9% average accuracy is achieved by GPT-5, compared with over 97% human performance, while Blind GPT-4o is close to random guessing.These results indicate that the benchmark requires genuine visual-spatial reasoning and remains difficult for current MLLMs.
  • 4.2 MAIN RESULTS: 30.7% average accuracy is achieved by Qwen2.5-VL-72B, the best open-source model, trailing the top proprietary model.The result indicates a performance gap between leading open-source and proprietary MLLMs.
  • 4.2 MAIN RESULTS: Multi-step reasoning generally performs worse than single-step tasks, while camera-motion comprehension is especially difficult, particularly for open-source models.The authors suggest that limited access to annotated first-person motion data may contribute to the open-source models’ camera-motion difficulty.
  • 4.2 MAIN RESULTS: Qwen2.5-VL-72B exceeds Qwen2.5-VL-32B by 3%, and InternVL3-78B exceeds InternVL3-1B by 1.5%, indicating limited gains from scaling model size.NVILA-15B surpasses most models exceeding 70 billion parameters, suggesting data quality and diversity may be more important bottlenecks than size alone.
  • 4.3 LIMITED EFFECT OF PROMPTING TECHNIQUES ON MMSI-BENCH: Spatial fine-tuning produces only marginal gains over base models, while linguistic and visual prompting yield inconsistent effects across models.CoT modestly improves GPT-4o but degrades other models; visual prompting slightly improves two models and degrades two others.

5 ERROR ANALYSIS

The error analysis identifies four recurring spatial-reasoning failure modes and uses annotated reasoning processes to automate diagnosis at benchmark scale.

  • 5.1 ERROR TYPE CATEGORIZATION: Four error categories are analyzed: grounding, overlap-matching and scene reconstruction, situation transformation, and spatial logic.The analysis begins with manual categorization of GPT-4o reasoning on 100 randomly sampled instances, followed by automated analysis across models.
  • 5.1 ERROR TYPE CATEGORIZATION: Grounding errors occur when the model misidentifies or mislocalizes relevant objects or image details.Such errors prevent spatial reasoning from being anchored to the actual visual content.
  • 5.1 ERROR TYPE CATEGORIZATION: Overlap-matching and scene-reconstruction errors involve failing to match corresponding locations or objects across images and reconstruct their shared scene layout.A representative failure is treating the same tree in two images as different trees.
  • 5.1 ERROR TYPE CATEGORIZATION: Situation-transformation errors arise when models reason relative to the wrong reference object or incorrectly convert relative directions into absolute directions.The affected transformations include left/right and north/south/east/west relations.
  • 5.1 ERROR TYPE CATEGORIZATION: Spatial-logic errors include hallucinated relations, incorrect transitivity, and selecting a moving object instead of a stationary reference for motion reasoning.For example, the model may infer that A is west of C when A is east of B and B is east of C.
  • 5.2 AUTOMATED ERROR ANALYSIS: 78% error-type labeling accuracy with annotated reasoning exceeds 53% using only the correct answer, showing that answer correctness alone is insufficient for evaluating reasoning.The automated method is then applied across all MMSI-Bench questions and representative models.
  • 5.2 AUTOMATED ERROR ANALYSIS: Overlap-matching and scene-reconstruction errors form the largest error proportion across all analyzed models, while reasoning accuracy remains below multiple-choice accuracy.Reasoning accuracy is 37.5% for GPT-4.5, 29.9% for GPT-4o, and 21.5% for Qwen2.5-VL-72B.

6 CONCLUSION

MMSI-Bench evaluates multi-image spatial reasoning in MLLMs and combines comprehensive benchmarking with scalable diagnosis of model failures. The authors position it as a resource for advancing spatially capable multimodal systems.

  • MMSI-Bench assesses MLLMs’ multi-image spatial reasoning through a challenging and comprehensive benchmark.
  • The benchmark evaluates 37 state-of-the-art models and reveals a substantial gap between current MLLMs and human-level spatial intelligence.
  • Expert-annotated reasoning processes support an automated pipeline for scalable and systematic diagnosis of model failures.
  • The benchmark is intended to provide a resource that accelerates progress toward more spatially capable and robust multimodal AI systems.

B.2 LIMITATIONS

MMSI-Bench’s principal limitation is the human effort required for manual annotation, while its current 1,000-sample scale remains useful until model performance saturates. The dataset also carries societal, computational, licensing, and legal-use considerations.

  • Manual annotation improves data quality and content richness but limits benchmark scalability because it requires substantial human effort.
  • MMSI-Bench contains 1,000 samples, while larger-scale expansion may become necessary when model performance begins to saturate.
  • The dataset may have unintended applications involving surveillance or automated decisions that affect privacy or disproportionately affect certain groups.
  • Evaluating modern vision-language models requires significant computational resources, although public release of scripts aims to reduce redundant effort.
  • Use of MMSI-Bench is subject to CC-BY terms, original image licenses, disclosure requirements, and legal responsibilities concerning data subjects.

D COMPARISON WITH OTHER SPATIAL QUESTION-ANSWERING BENCHMARKS

Unlike benchmarks that mainly assess single-image spatial understanding or include limited spatial subsets, MMSI-Bench is human-curated for comprehensive multi-image spatial reasoning. Its question language is also more diverse and structurally complex than comparison benchmarks.

  • Most existing spatial benchmarks focus on single-image understanding or treat multi-image spatial reasoning as a limited component of broader VQA suites.
  • MMSI-Bench covers camera, objects, and regions across ten task types plus one multi-step reasoning category using diverse real-world scenarios.
  • MMSI-Bench is fully human-curated by six 3D-vision researchers without templates and includes human-authored reasoning chains for each question.
  • 1,108 distinct words and 993 unique syntactic structures demonstrate MMSI-Bench’s broad vocabulary and syntactic diversity.The 993 syntactic structures are nearly double VSI-Bench’s and more than triple ERQA’s according to the comparison analysis.
  • The human-curated language complexity raises the bar for evaluating models’ generalization to naturally phrased spatial questions.

E.1 BENCHMARK MODELS.

The evaluation examines diverse proprietary and open-source MLLMs using exact-match accuracy and multiple prompting strategies. It also evaluates reasoning processes with structured prompts and human-annotated references.

  • The benchmark evaluation covers proprietary and open-source MLLM families, diverse model scales, and recent architectural advances.
  • Accuracy is reported as the percentage of exact matches between extracted model answers and ground-truth multiple-choice answers.
  • Models are evaluated with Direct, Linguistic, and Visual prompting strategies.
  • Visual prompts combine image inputs with PATS-drawn prompts, as illustrated in Figures 8–11.
  • Reasoning-process evaluation uses prompts with human-annotated references and four labeled reasoning types corresponding to possible error categories.
  • Human performance is measured from five uninvolved adults who receive questions and images simultaneously and have unlimited answering time.

E.5 CORRELATION WITH DOWNSTREAM EMBODIED AND ACTIVE PERCEPTION TASKS

This section examines whether MMSI-Bench performance transfers to downstream embodied-control and active-perception tasks. Across three EMBODIEDBENCH tasks, model scores show consistently strong positive correlations with MMSI-Bench accuracy.

  • Correlation analysis: MMSI-Bench is compared with EB-Navigation, EB-Manipulation, and EB-Habitat to assess transferability to embodied and active-perception tasks.EB-Navigation measures navigation actions, EB-Manipulation measures object-centric interaction, and EB-Habitat measures active perception and planning.
  • Correlation analysis: Pearson and Spearman coefficients are computed between average MMSI-Bench accuracy and performance on each downstream task across frontier multimodal models.The analysis uses both linear and rank-based correlation measures.
  • Results: Models performing well on MMSI-Bench also tend to perform strongly on downstream navigation, manipulation, and Habitat tasks.The particularly high EB-Navigation correlation is described as closely aligned with competencies needed for embodied navigation.
  • Per-model performance: Table 11 reports per-model scores on MMSI-Bench and the three EMBODIEDBENCH tasks used in the correlation analysis.All table values are reported as percentages.

F.1 DATASET SOURCES

MMSI-Bench draws questions from eight diverse data sources spanning indoor and outdoor scenes, egocentric video, robotics, multi-view stereo, and video object segmentation. Its annotation process uses human-designed, multi-image, multiple-choice questions with explicit quality control and reasoning traces.

  • Dataset sources: The source collection spans indoor RGB-D scenes, autonomous-driving data, egocentric footage, robotic manipulation, multi-view stereo, and video object segmentation.ScanNet, Waymo, Ego4D, AgiBot-World, DTU, nuScenes, and DAVIS 2017 provide these complementary settings.
  • Dataset sources: MMSI-Bench combines eight sources, including Matterport3D, ScanNet, Ego4D, AgiBot-World, DTU, nuScenes, DAVIS 2017, and Waymo.The benchmark contains 463 Matterport3D, 280 ScanNet, 67 Ego4D, 45 AgiBot-World, 72 DTU, 39 nuScenes, 29 DAVIS 2017, and 5 Waymo questions.
  • Annotation process: Annotators select images, write human-readable four-option questions, identify the correct answer, and document the reasoning process.Questions are free-form, require multiple images, and contain exactly one correct option.
  • Quality control: Reviewers approve entries or discard them with specified reasons when quality issues are identified.This quality-control workflow complements the annotation interface and standardized data structure.
  • Annotation process: The taxonomy assigns each question to one of eleven categories covering positions, attributes, motion, and multi-step reasoning among cameras, objects, and regions.The category design aims to cover fundamental spatial reasoning types comprehensively.
  • Annotation process: Multi-Step Reasoning questions contain 3–11 annotated reasoning steps, averaging 5.7 steps per question.Annotators are instructed to make these questions non-direct and explicitly record their stepwise solutions.

G REPRESENTATIVE MMSI-BENCH SAMPLES FROM EACH CATEGORY (FULL VERSION)

This section presents complete MMSI-Bench examples for all eleven categories and illustrates how annotated reasoning exposes model errors. The examples cover positional, motion, attribute, and multi-step spatial reasoning, while error figures distinguish four failure modes.

  • Position categories: Complete examples are provided for six position categories involving camera, object, and region relationships.The categories are Camera-Camera, Camera-Object, Camera-Region, Object-Object, Object-Region, and Region-Region.
  • Motion, attribute, and multi-step categories: The full examples also cover Motion (Camera), Motion (Object), Attribute (Measurement), Attribute (Appearance), and Multi-Step Reasoning.Together with the six position categories, these constitute all eleven benchmark categories.
  • Representative samples: The representative samples include complete reasoning processes designed to make spatial reasoning deficiencies more intuitive.The examples are presented as full category-level versions of questions and rationales simplified in the main text.
  • Error analysis: The error analysis distinguishes grounding, overlap-matching and scene-reconstruction, situation-transformation reasoning, and spatial-logic errors.Figures 25–28 mark these four error types in orange, red, blue, and green, respectively.
Loading 2505.23764v3…