Source-linked AI summary
Video-MSR: Benchmarking Multi-hop Spatial Reasoning Capabilities of MLLMs
Rui Zhu, Xin Shen, Shuchen Wu, Chenxi Miao, Xin Yu, Yang Li, Weikang Li, Deguo Xia, Jizhou Huang
TL;DR
Existing spatial-reasoning benchmarks largely test single-step perception-to-judgment, while multi-hop spatial reasoning in video remains underexplored. Video-MSR evaluates this capability with four task types, and the authors assess 20 MLLMs. They report performance drops on multi-hop tasks and an improvement after instruction-tuning on MSR-9K.
Problem
Existing spatial assessments largely use single-step perception-to-judgment tasks, leaving the capacity of MLLMs to maintain intermediate states and spatial constraints across multi-hop reasoning underexplored.
Source: §1 · p. 1, §2.1 · p. 3Method
Video-MSR evaluates video-based multi-hop spatial reasoning through four task types, using visually grounded question-answer pairs generated and human-verified from video instances.
Source: §1 · p. 2Results
Across 20 evaluated MLLMs, surface-level perception proficiency coexists with performance drops on MSR and multi-step spatial disorientation and hallucination; MSR-9K fine-tuning yields a 7.82% absolute improvement on Video-MSR.
Source: §1 · p. 2Takeaways & Limitations
The authors report that MSR-9K fine-tuning establishes a strong baseline for future research on multi-hop spatial reasoning.
Source: §1 · p. 2The formalization assumes each current spatial state depends on the immediately preceding state, the video, and the current spatial constraint.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
Spatial reasoning has emerged as a critical capability for Multimodal Large Language Models (MLLMs), drawing increasing attention and rapid advancement. However, existing benchmarks primarily focus on single-step perception-to-judgment tasks, leaving scenarios requiring complex visual-spatial logical chains significantly underexplored. To bridge this gap, we introduce Video-MSR, the first benchmark specifically designed to evaluate Multi-hop Spatial Reasoning (MSR) in dynamic video scenarios. Video-MSR systematically probes MSR capabilities through four distinct tasks: Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction. Our benchmark comprises 3,052 high-quality video instances with 4,993 question-answer pairs, constructed via a scalable, visually-grounded pipeline combining advanced model generation with rigorous human verification. Through a comprehensive evaluation of 20 state-of-the-art MLLMs, we uncover significant limitations, revealing that while models demonstrate proficiency in surface-level perception, they exhibit distinct performance drops in MSR tasks, frequently suffering from spatial disorientation and hallucination during multi-step deductions. To mitigate these shortcomings and empower models with stronger MSR capabilities, we further curate MSR-9K, a specialized instruction-tuning dataset, and fine-tune Qwen-VL, achieving a +7.82% absolute improvement on Video-MSR. Our results underscore the efficacy of multi-hop spatial instruction data and establish Video-MSR as a vital foundation for future research. The code and data will be available at https://github.com/ruiz-nju/Video-MSR.
1 Introduction
Video-MSR addresses the gap between MLLMs’ surface-level visual perception and the multi-step spatial reasoning needed to draw conclusions from video. Its evaluation finds weaknesses in multi-hop reasoning, while instruction-tuning Qwen-VL on MSR-9K improves Video-MSR performance.
1 Introduction
- Existing spatial-reasoning evaluations largely use single-step perception-to-judgment tasks, leaving MLLMs’ multi-hop spatial reasoning capacity underexplored.The paper frames maintaining intermediate states, integrating visual cues, and adhering to spatial constraints across long reasoning chains as central MSR demands.Source: §1 · p. 1
- Video-MSR evaluates four reasoning tasks using 3,052 video instances and 4,993 question-answer pairs grounded in video content.The tasks are Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction; the benchmark uses automated generation followed by human verification.Source: §1 · p. 2
- Across 20 evaluated MLLMs, surface-level perception proficiency coexists with performance drops on MSR, including spatial disorientation and hallucination during multi-step reasoning.The authors identify failures to maintain logical consistency during these deductions.Source: §1 · p. 2
- 7.82% absolute improvement on Video-MSR follows from fine-tuning Qwen-VL on the specialized MSR-9K instruction-tuning dataset.The authors present this result as evidence for the efficacy of their data-construction paradigm.Source: §1 · p. 2
2 Multi-hop Spatial Reasoning
The paper defines multi-hop spatial reasoning as constructing a chain of spatial states from video and a query to reach an answer. It also outlines a video-grounded data pipeline and formulates the chain as sequential, constraint-conditioned state transitions.
2 Multi-hop Spatial Reasoning
- Video-MSR data construction combines high-fidelity video collection, template-guided QA generation, text-only blind testing, and manual expert verification.The figure presents this as a pipeline for producing high-quality MSR data while filtering biases and hallucinations.Source: Figure 2 · p. 3
2.1 Problem Definition
- MSR requires an explicit spatial reasoning chain between video and query cues and the final answer, rather than a direct single-hop mapping.The benchmark pipeline generates template-guided question-answer pairs from collected videos and applies text-only blind testing and expert verification.Source: §2.1 · p. 3
- The reasoning chain is represented as spatial states, beginning with a query-derived anchor and continuing through intermediate entities, regions, or physical statuses.The number of reasoning hops is represented by T.Source: §2.1 · p. 3
- Each state transition is modeled as depending on the immediately preceding state, the video evidence, and the current spatial constraint.This Markovian formulation approximates dependence on the full history using the previous state and current constraint.Source: §2.1 · p. 3
- The objective maximizes the joint probability of the reasoning chain and final answer, with success requiring a coherent chain grounded in visual-spatial evidence.The pipeline overview describes automated generation followed by dual-phase verification to support reliable multi-hop reasoning data.Source: §2.1 · p. 3
3 Video-MSR
Video-MSR is a benchmark for evaluating multi-hop spatial reasoning in dynamic video, organized around four tasks and built from visually grounded QA pairs.
3.1 Taxonomy
- CL tests exclusionary filtering in cluttered scenes, while CRR follows landmark-based spatial dependencies and RP requires a feasible, constraint-compliant trajectory.CPD asks models to infer physical consequences from static visual evidence by mentally simulating motion and detecting geometric interference.Source: §3.1 · pp. 3–4
3.2 Benchmark Construction
- Video-MSR combines curated public-dataset videos, template-guided Gemini-3.0-Pro question generation, video-blind automated filtering, and final review by 10 human experts.The blind test discards questions consistently answerable without video context; experts check for hallucinations and logical correctness of ground-truth answers.Source: §3.2 · p. 4
3.3 Data Statistics
- The benchmark contains 4,993 QA pairs across 3,052 video clips and stratifies tasks into easy and hard levels by reasoning steps and video duration.Hard questions require intricate chains and may include distracting visual information; easy questions use fewer visual cues and simpler reasoning.Source: §3.3 · p. 4
4 Experiments
Across Video-MSR’s four tasks, model performance varies substantially by task, and the analyses find that scale and standard reasoning enhancements do not ensure stronger multi-hop spatial reasoning. MSR-9K tuning improves Qwen-VL results, while evaluator scores remain consistent across tested judges.
4.2 Main Results
- Qwen3-VL leads open-source models at 43.92% overall accuracy, while GPT-4o leads proprietary models at 41.87%.The benchmark evaluates performance across four distinct spatial reasoning tasks.Source: §4.2 · p. 5
- Gemini-2.5-Flash leads CRR at 66.33% but underperforms on CL, where Qwen3-VL-8B-Instruct leads at 38.44%.The authors distinguish CRR’s sequential retrieval chain from CL’s multi-step filtering over objects using spatial attributes.Source: §4.2 · pp. 5–6
4.3 Further Analysis
- MSR-9K tuning raises Qwen2.5-VL-7B-Instruct overall accuracy by 7.82% and Qwen3-VL-8B-Instruct by 3.10%.For Qwen2.5-VL-7B-Instruct, the Hard-subset gain is +10.13%, versus +4.79% on Easy; RP gains +48.62%.Source: Table 3 · p. 6, §4.3 · p. 7
- InternVL3.5 shows volatile performance across sizes, and Qwen3-VL-4B leads Easy CL, CRR, and RP, so MSR proficiency is not monotonic with model size.The authors suggest architectural biases or high-quality data distributions may matter beyond scaling general-purpose multimodal models.Source: Figure 5 · p. 6, §4.3 · p. 6
4.2 Main Results
- Gemini-2.5-Flash and Qwen3-VL-32B-Instruct lead RP at 61.21% and 57.93%, while InternVL models score around 10% or lower.LLaVA-NeXT-Video-7B also struggles with open-ended route planning, which the authors largely attribute to its generation limitations.Source: §4.2 · p. 6
- Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct lead CPD at 38.36% and 36.53%, while Gemini-2.5-Flash and GPT-4o score 24.77% and 25.57%.The authors identify a gap between these proprietary systems and leading open-source models on implicit spatial reasoning and mental simulation.Source: §4.2 · p. 6
4.3 Further Analysis
- CoT prompting drops Qwen2.5-VL-7B and Qwen3-VL-8B performance by 1.80% and 4.54%, while Cambrian-S-7B falls over 5% after atomic spatial training.The authors associate CoT losses with hallucinations and reasoning drift that propagate errors; they note that fundamental perception alone does not ensure long-horizon spatial reasoning.Source: §4.3 · pp. 6–7
- Scores for each target model remain highly consistent across Gemini-3.0-Pro, Qwen2.5-VL-7B, and Qwen3-VL-8B evaluators.The evaluation protocol uses direct string matching for multiple-choice CL and CPD, and an LLM judge for open-ended CRR and RP.Source: Figure 6 · p. 7, §4.3 · p. 7
5 Related Work
Prior video-understanding benchmarks cover perception, temporal awareness, and broader reasoning, while spatial benchmarks increasingly test atomic skills and visual perspectives. Complex spatial reasoning under strict constraints remains underexplored.
5.1 Video Understanding of MLLMs
- Video-understanding benchmarks assess foundational perception, temporal awareness, and increasingly complex areas such as video mathematics, long-form comprehension, and cross-domain reasoning.Together, these benchmarks provide systematic assessment of MLLM video-understanding capabilities.Source: §5.1 · p. 7
5.2 Benchmarks for Spatial Reasoning
- Spatial benchmarks cover spatial relationships, atomic skills, and varying visual perspectives, but remain focused on simpler tasks rather than complex reasoning under strict constraints.The text names VSR, CV-Bench, VSI-Bench, and MM-Perspective as examples.Source: §5.2 · p. 7
6 Conclusion
Video-MSR evaluates multi-hop spatial reasoning in video and reveals that MLLMs struggle with complex spatial logic despite strong visual perception. MSR-9K fine-tuning improves performance, while scale, standard Chain-of-Thought prompting, and atomic visual training do not trivially address the gap.
6 Conclusion
- MLLMs can excel at visual perception yet struggle with complex spatial logic, often producing incoherent or hallucinatory reasoning.The analysis also finds MSR proficiency is not strictly correlated with model scale.Source: §6 · p. 7
- Standard Chain-of-Thought prompting and atomic visual training do not trivially address the multi-hop spatial reasoning gap.The conclusion also reports that MSR proficiency is not strictly correlated with model scale.Source: §6 · p. 7
- Fine-tuning on MSR-9K yields substantial performance gains and supports the necessity of logically grounded spatial data.The paper presents Video-MSR as a foundation for future research on stronger MLLM spatial reasoning.Source: §6 · p. 7
Ethical Statement
The videos come from publicly available datasets with permissible licenses, and the authors report privacy protections and content filtering.
Ethical Statement
- Videos are sourced from publicly available datasets with permissible licenses, and the dataset excludes personally identifiable information.The authors report strict filtering for social bias, toxicity, and harmful content.Source: Ethical Statement · p. 8
- Human annotators were recruited with informed consent and compensated fairly.Source: Ethical Statement · p. 8
- The dataset underwent filtering for social bias, toxicity, and harmful content.Source: Ethical Statement · p. 8