Source-linked AI summary
MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence
Jingli Lin, Runsen Xu, Shaohao Zhu, Sihan Yang, Peizhou Cao, Yunlong Ran, Miao Hu, Chenming Zhu, Yiman Xie, Yilin Long, Wenbo Hu, Dahua Lin, Tai Wang, Jiangmiao Pang
TL;DR
Video-based spatial intelligence lacks a comprehensive benchmark covering continuous visual input and the full range of spatial reasoning abilities. MMSI-Video-Bench addresses this gap with a human-annotated, four-level benchmark, but evaluations show models remain far behind humans and exhibit systematic reasoning failures.
Problem
Existing benchmarks do not comprehensively evaluate spatial intelligence over continuous video inputs across holistic task types and diverse scenes.
Method
MMSI-Video-Bench uses a four-level framework and human-designed, reviewed questions grounded in 1,278 clips from 25 datasets and in-house videos.
Results
Models perform near chance on many tasks, and Gemini 3 Pro scores 38.0 versus humans’ 96.4, a gap of nearly 60%.
Takeaways & Limitations
The benchmark provides a rigorous testbed and diagnostic signals for improving video-based spatial reasoning.
Takeaways & Limitations
Models often fail to use supplied 3D spatial cues because the cues are ignored or incorrectly associated with video content and questions.
Abstract
from arXiv · showhide
Spatial understanding over continuous visual input is crucial for MLLMs to evolve into general-purpose assistants in physical environments. Yet there is still no comprehensive benchmark that holistically assesses the progress toward this goal. In this work, we introduce MMSI-Video-Bench, a fully human-annotated benchmark for video-based spatial intelligence in MLLMs. It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips from 25 datasets and in-house videos. Each item is carefully designed and reviewed by 3DV experts with explanatory rationales to ensure precise, unambiguous grounding. Leveraging its diverse data sources and holistic task coverage, MMSI-Video-Bench also supports three domain-oriented sub-benchmarks (Indoor Scene Perception Bench, Robot Bench and Grounding Bench) for targeted capability assessment. We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human--AI gap: many models perform near chance, and the best reasoning model lags humans by nearly 60%. We further find that spatially fine-tuned models still fail to generalize effectively on our benchmark. Fine-grained error analysis exposes systematic failures in geometric reasoning, motion grounding, long-horizon prediction, and cross-video correspondence. We also show that typical frame-sampling strategies transfer poorly to our reasoning-intensive benchmark, and that neither 3D spatial cues nor chain-of-thought prompting yields meaningful gains. We expect our benchmark to establish a solid testbed for advancing video-based spatial intelligence.
1. Introduction
MMSI-Video-Bench addresses gaps in video-based spatial-intelligence evaluation with a holistic, human-designed benchmark spanning perception, planning, prediction, and cross-video reasoning. Its evaluation reveals substantial model–human disparities and systematic failures in spatial reasoning.
- Existing benchmarks often use images rather than videos, while video benchmarks lack holistic question types, diverse human-designed questions, and comprehensive scenes.
- The benchmark evaluates four levels of spatial intelligence: Perception, Planning, Prediction, and Cross-Video Reasoning.
- Eleven 3DV researchers manually designed and reviewed questions from 25 datasets and in-house videos, producing 1,106 questions grounded in 1,278 clips.
- Many models perform near random guessing, while Gemini 3 Pro trails humans by nearly 60%; spatially fine-tuned models also fail to generalize effectively.
- Error analysis identifies geometric-reasoning, fine-grained motion-grounding, prompt–evidence alignment, and cross-video correspondence failures.
2. Related Work
Prior benchmarks assess video understanding or spatial reasoning through narrower task, modality, or scene scopes. MMSI-Video-Bench instead combines diverse real-world videos with fully human-annotated, holistic spatial evaluation.
- Earlier video benchmarks primarily measured global comprehension, while later benchmarks expanded toward temporal dynamics and temporal–event reasoning.
- Table 1 compares spatial benchmarks by diversity, comprehensiveness, challenge, task dimensions, and Human–AI Gap.
- Existing spatial benchmarks vary in task design, modality, scene scope, and targeted abilities, often focusing on single-image reasoning or limited aspects and scene types.
- MMSI-Video-Bench differs by using diverse real-world videos and full human annotation for a more holistic and realistic spatial-intelligence assessment.
3. MMSI-Video-Bench
MMSI-Video-Bench organizes video spatial intelligence around spatial state understanding, higher-level decision-making, and reasoning across discontinuous observations and viewpoints. Its data spans varied real-world scenes and supports fine-grained benchmark coverage.
- Spatial Construction requires inferring fine-grained global layouts and spatial relations from partial, sequential video observations.
- Planning requires goal-directed actions from visual cues, whereas Prediction requires imagining outcomes under hypothetical conditions.
- The benchmark includes examples across relationships, instances, and cameras, with subtype details documented separately.
- Cross-Video Reasoning covers Memory Update across discontinuous observations and Multi-View Integration across viewpoints.
- Its collection spans tabletop recordings, indoor and outdoor scenes, landscapes, sports, and movie footage from 25 public datasets.
- The final benchmark contains 1,106 questions grounded in 1,278 clips, covering five categories and 13 subtypes.
4. Experiments
MMSI-Video-Bench evaluates diverse open-source, proprietary, and spatially fine-tuned MLLMs across spatial reasoning categories and sampling settings. Models remain far below humans, with prediction especially difficult and spatial fine-tuning failing to generalize reliably.
- Main Results: Sufficient-Coverage does not outperform Uniform-50 and can sometimes reduce performance, indicating that additional frames may introduce distracting redundancy.
- Main Results: Proprietary models consistently outperform open-source models, whose strongest scores are 32.7 for QwenVL2.5-72B and 32.4 for QwenVL2.5-32B.
- Main Results: Prediction is the most challenging main category, while Camera–Instance Spatial Relation is the hardest subtype.These tasks require prediction from conditions or physical priors, plus ego-to-scene reasoning and detailed instance grounding.
- Main Results: Gemini 3 Pro leads overall, GPT-5 is strongest on spatial construction and reasoning, and Seed-1.6-Vision excels at motion and multi-segment reasoning.
- Evaluation of Spatially Fine-tuned Models: Spatially fine-tuned models fail to generalize consistently: SpaceQwen changes little, while Spatial-MLLM and VLM3R degrade overall performance.
5. Frame Sampling Study
The frame-sampling study compares frame counts and local consecutive versus whole-video uniform sampling, then tests semantic keyframe selection. These strategies provide limited benefit on MMSI-Video-Bench.
- Frame Sampling Study: Performance is evaluated with 1, 10, and 50 frames using consecutive local-segment sampling or uniform whole-video sampling across three representative models.
- Frame Sampling Study: Adaptive Keyframe Sampling fails to improve MMSI-Video-Bench performance despite gains on LongVideoBench and Video-MME.Semantic similarity alone may not identify the key frames required for the benchmark’s questions.
6. Error Analysis
Error analysis identifies distinct, task-specific failure modes across MMSI-Video-Bench, with geometric reasoning errors most prevalent overall and correspondence failures prominent in cross-video reasoning.
- Geometric Reasoning Error is the most prevalent error type overall, especially in Spatial Construction tasks, where models struggle with simple geometric relations.
- Motion Understanding is limited by detailed grounding failures involving fast, subtle, or long-duration motions.
- Planning and Prediction frequently exhibit Prompt Alignment Error despite models perceiving relevant spatiotemporal context.
- Cross-Video Reasoning is dominated by Latent Logical Inference and Detailed Grounding Errors when matching instances across videos.
- The categorized failures provide diagnostic targets for improving capabilities across benchmark categories.
7. Preliminary Exploration for Model Improvement
The preliminary improvement study tests explicit spatial representations and chain-of-thought prompting against identified errors. Neither approach meaningfully improves performance under the reported setup.
- The exploration frames spatial reasoning improvement around richer training data or explicit spatial representations, while testing only the latter direction preliminarily.
- 3D spatial cues produce gains below 1% for all four evaluated models, indicating no significant improvement under the Uniform-50 setting.The setup reconstructs and renders 50 frames per video through VGGT before evaluation.
- 3D-cue failures arise both from inaccurate reconstruction in complex or dynamic scenes and from models ignoring or misassociating the cues.Rendered cues may provide little useful information or introduce noise, while models may fail to connect them with video content and questions.
- Chain-of-thought prompting does not consistently improve performance because the difficulty lies in inherently challenging reasoning rather than omitted procedural steps.
8. Additional Perspectives of MMSI-Video-Bench
MMSI-Video-Bench supports three domain-oriented subsets for targeted capability assessment. Results show different leaders and bottlenecks across indoor perception, robotics, and grounding tasks.
- Indoor Scene Perception Bench: GPT-5 leads overall indoor scene perception, excelling at instance-centric static scenes, while Gemini 2.5 Flash leads camera-centric static perception.
- Indoor Scene Perception Bench: Weaker indoor-perception models are particularly bottlenecked by Static-CC, which requires reasoning about observer–environment relations.
- Robot Bench: Gemini 3 Pro leads the Robot Bench overall; QwenVL2.5-72B leads Manipulation, while O3 and Gemini 3 Pro lead Navigation.
- Grounding Bench: Gemini 2.5 Flash leads the Grounding Bench overall in both temporal localization and target-object identification.
- The three subset benchmarks enable targeted assessment of specific capabilities and models designed for particular domains or task types.
9. Conclusion
MMSI-Video-Bench is a holistic, human-annotated testbed with domain-oriented subsets for video-based spatial intelligence. Its evaluation exposes substantial model–human gaps, task-specific weaknesses, and limited gains from current improvement strategies.
- MMSI-Video-Bench evaluates perception, understanding, reasoning, and decision-making over spatiotemporal information, alongside three targeted sub-benchmarks.
- Models struggle across task categories beyond spatial construction, and spatially fine-tuned models fail to generalize effectively to the benchmark.
- Error analyses reveal task-specific weaknesses, while neither 3D spatial cues nor chain-of-thought prompting yields meaningful gains.
- The benchmark provides a rigorous testbed and the analyses offer actionable directions for future improvements.
A.2. Data Collection & Preprocessing Details
The benchmark combines diverse video sources with standardized preprocessing and a human-centered annotation and validation workflow.
- The benchmark draws on 25 public video datasets plus additional videos, spanning indoor scans, outdoor environments, egocentric interactions, and exocentric activities.
- Source datasets are summarized by capture type, average duration, and standardized FPS after preprocessing.
- Frame-based datasets are reconstructed into videos using the FPS specified in their original papers, while clips that are too short are filtered out.
- Annotators use a dedicated interface to select question types, videos, temporal boundaries, answer options, correct answers, and reasoning explanations.
- Independent validators inspect randomly assigned annotations and accept or reject them, providing revision reasons and suggestions for rejected samples.
B. Experiment Details
Experiments standardize model inputs and outputs through structured prompts and consistent answer extraction, supported by dedicated interfaces for annotation and task organization.
- The evaluation template supplies timestamped frames, a task description, and the question with options, while output requirements vary by evaluation setting.
- A general-purpose answer extraction function parses model outputs consistently and verifies that correct responses can be detected across models.
- The benchmark provides interfaces for annotation and quality validation, with task categories and subtypes organized for the experiment workflow.