Source-linked AI summary
DSI-Bench: A Benchmark for Dynamic Spatial Intelligence
Ziang Zhang, Zehan Wang, Guanghao Zhang, Weilong Dai, Yan Xia, Ziang Yan, Minjie Hong, Zhou Zhao
TL;DR
Models have not been systematically assessed in realistic dynamic scenes where observers and objects move together, despite the importance of reasoning about their changing spatial relationships. DSI-Bench provides a video question-answer benchmark with decoupled motion patterns and symmetric designs. Evaluations identify motion conflation, semantic bias, and difficulty inferring dynamic relative relationships.
Problem
Current methods are largely limited to settings where either the observer or objects remain static, leaving performance in realistic settings where both move systematically unexplored.
Source: §1 · p. 2Method
DSI-Bench uses nearly 1,000 dynamic videos and over 1,700 manually annotated questions across five decoupled motion patterns, with spatially and temporally symmetric versions.
Source: §1 · p. 2Results
Evaluations of 14 VLMs and expertise models found that models conflate observer and object motion, show semantic biases that can cause hallucinations, and struggle with relative-pose relationships in dynamic videos.
Source: §1 · p. 2Takeaways & Limitations
DSI-Bench is intended as a resource for advancing models’ dynamic spatial perception and reasoning capabilities.
Source: §6 · p. 9Classical geometric constraints are primarily designed for static scenes and can destabilize keypoint tracking and distance estimation when the observer and environment move.
Source: §2.3 · p. 3
Abstract
from arXiv · showhide
Reasoning about dynamic spatial relationships is essential, as both observers and objects often move simultaneously. Although vision-language models (VLMs) and visual expertise models excel in 2D tasks and static scenarios, their ability to fully understand dynamic 3D scenarios remains limited. We introduce Dynamic Spatial Intelligence and propose DSI-Bench, a benchmark with nearly 1,000 dynamic videos and over 1,700 manually annotated questions covering nine decoupled motion patterns of observers and objects. Spatially and temporally symmetric designs reduce biases and enable systematic evaluation of models' reasoning about self-motion and object motion. Our evaluation of 14 VLMs and expert models reveals key limitations: models often conflate observer and object motion, exhibit semantic biases, and fail to accurately infer relative relationships in dynamic scenarios. Our DSI-Bench provides valuable findings and insights about the future development of general and expertise models with dynamic spatial intelligence.
1 INTRODUCTION
Current models have not been systematically assessed in realistic settings where observers and objects move together. DSI-Bench evaluates this dynamic spatial reasoning and finds that models conflate motions, show semantic biases, and struggle with relative relationships.
1 INTRODUCTION
- The paper frames dynamic spatial intelligence as understanding simultaneous changes in one’s own position and other objects’ 3D motion.It motivates this ability with human responses to moving prey and vehicles.Source: §1 · p. 2
- DSI-Bench provides over 1,700 manually annotated questions across five decoupled observer and object motion patterns, with spatially and temporally symmetric versions to mitigate 3D biases.The benchmark uses nearly 1,000 dynamic scene videos collected, cleaned, and clipped from diverse sources.Source: §1 · p. 2
- Across 14 VLMs and expertise models, evaluations found conflated observer and object motion, semantic biases that can cause hallucinations, and inconsistent relative-pose reasoning in dynamic videos.The authors also identify limitations of classical 3D constraints for dynamic spatial perception.Source: §1 · pp. 2–3
- The authors evaluate general and expertise models using symmetric samples to analyze bias and hallucination tendencies in dynamic scenarios.They examine model explanations and visualizations to identify limitations of classical 3D constraints.Source: §1 · p. 3
2 RELATED WORKS
Prior work advances video understanding, spatial benchmarks, and 3D expertise models, but leaves a gap in evaluating motion when observers and objects move together. DSI-Bench addresses this gap with dynamic videos, symmetry-based augmentation, and bias analysis.
2.1 LARGE VISION-LANGUAGE MODEL
- VLMs have advanced video and static-spatial understanding; Qwen2.5-VL Bai et al. (2025) uses timestamp encoding and adaptive frame rates and resolution for long-video understanding.InternVL-3.5 Wang et al. (2025b) uses a visual resolution router to focus on fine-grained video details while maintaining overall performance.Source: §2.1 · p. 3
2.2 BENCHMARKS FOR SPATIAL INTELLIGENCE
- Spatial benchmarks cover diverse tasks and multi-view or spatiotemporal reasoning, while 3DSR-Bench Ma et al. (2024) and Ori-Bench Wang et al. (2024c) examine hallucination and orientation bias.Most evaluations still emphasize static scenes or observers; DSI-Bench instead covers five disentangled motion types and uses symmetry-based augmentation and multi-view bias analysis.Source: §2.2 · p. 3
2.3 3D VISUAL SPATIAL EXPERTISE MODELS
- 3D expertise models combine classical constraints with 2D foundation models; VGGT Wang et al. (2025a) reconstructs 3D scenes, while SpatialTrackerV2 Xiao et al. (2025) reasons jointly about geometry, camera poses, and object motion.Classical geometric constraints are primarily designed for static scenes and can destabilize keypoint tracking and distance estimation when observer and environment both move.Source: §2.3 · p. 3
3 DSI-BENCH
DSI-Bench evaluates perception of observer and object motion in realistic dynamic videos. It defines task categories and builds a benchmark through video selection, annotation, question generation, and symmetry-based augmentation.
3 DSI-BENCH
- DSI-Bench is designed to evaluate models’ perception of observer and object motion in realistic dynamic videos.The section outlines the benchmark overview and its construction process.Source: §3 · p. 4
3.1 OVERVIEW
- Dynamic Spatial Intelligence covers reasoning about changing spatial relationships, including position, distance, and orientation, as both the observer and observed objects move.Tasks examine object–scene, observer–scene, and observer–object relationships; the benchmark contains over 1,700 VQA pairs.Source: §3.1 · p. 4
3.2 BENCHMARK CONSTRUCTION PROCESS
- The benchmark draws over 1,700 question-answer pairs from 943 videos, with translation, rotation, and combined motion represented for observers and objects.Figure 2 shows the task distribution in its left panel and observer and observed-motion distributions in its middle and right panels.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
- Videos are scene-segmented, filtered for irregular observer motion, standardized, and manually annotated before template-based questions are generated and refined by experts.The process includes motion-pattern annotation, relative-distance questions for a subset, and relative-orientation questions using object orientation annotations.Source: §3.2 · p. 5
- Spatiotemporal augmentation creates standard, horizontally flipped, time-reversed, and combined variants, with answer options adjusted to preserve ground-truth labels.Human experts inspect cases where reversal shifts the reference frame and rule-based answer substitutions are insufficient; Figure 3 depicts the construction pipeline.Source: §3.2 · p. 5, Figure 3 · p. 5
4 EVALUATION ON DSI-BENCH
The evaluation tests VLMs and 3D expertise models on DSI-Bench, finding that dynamic scenarios challenge accuracy and robustness. Free-form reasoning and larger model size do not reliably resolve these difficulties.
4.1 EVALUATION SETUP
- The evaluation includes proprietary and open-source VLMs alongside VGGT and SpatialTrackerV2 as 3D expertise models.The VLM set spans model families and scales.Source: §4.1 · pp. 5–6
- VLMs are evaluated with direct-answer and free-form-reasoning protocols; 3D models are tested sample-wise and group-wise across spatio-temporal flips.Group-wise accuracy counts a video group correct only when at least three of its four variants are correct.Source: §4.1 · pp. 6–7, Table 1 · p. 6
4.2 MAIN RESULTS
- Seed-1.6-vision accuracy decreases 2.67% for object dynamics with a moving observer and 11.55% for observer motion under dynamic conditions.Dynamic scenarios consistently challenge models more than equivalent static tasks.Source: Figure 5 · p. 7, §4.2 · p. 7
- Group-wise evaluation makes the static–dynamic performance gap more pronounced and shows reduced robustness in dynamic settings.Across task categories, VLMs drop substantially from sample-wise to group-wise evaluation, while SpatialTrackerV2 degrades minimally.Source: §4.2 · pp. 7–8
- Free-form reasoning yields minor, inconsistent accuracy changes; QwenVL2.5-72B gains 0.17%, while Gemini-2.5-Pro and InternVL3.5-241B score lower than with direct answers.Some models also produce incoherent text until reaching the output-token limit.Source: Table 2 · p. 7, §4.2 · p. 8
- Increasing model size improves sample-wise accuracy but not group-wise robustness, with larger models underperforming their smaller counterparts in group-wise evaluation.The reported sample-wise gains include 2.8% for QwenVL2.5-72B over 32B, 2.69% for InternVL3.5-38B over 8B, and 2.35% for InternVL3.5-241B-A28B over 30B-A3B.Source: §4.2 · p. 8
5 ERROR ANALYSIS
The error analysis identifies forward-selection bias, confusion between rotation and translation, and entangled reasoning about observer and object motion. It also reports instability and relative-distance errors in 3D expertise models.
5.1 BIAS AND HALLUCINATION ON VLM
- VLMs select “forward” far more often than ground-truth annotations contain it, a selection bias linked by the authors to imbalanced motion patterns in visual datasets.The fixed-object examples illustrate how this bias can produce hallucinated forward motion.Source: §5.1 · p. 8, Figure 6 · p. 9
- VLMs mistake in-place observer rotations for translations when using scene entry into the field of view to infer motion direction.3D expertise models using classical geometric constraints for camera-pose estimation perform better on these tasks.Source: §5.1 · pp. 8–9, Figure 7 · p. 9
- VLMs fail to infer observer and object motion independently, sometimes substituting the object's orientation and movement for the observer's or generalizing relative motion to the scene frame.The paper presents these as two patterns of entangled reasoning.Source: §5.1 · p. 9, Figure 8 · p. 9
5.2 INSTABILITY ON 3D EXPERTISE MODEL
- Viewpoint changes and foreground occlusions can cause keypoint loss and bundle-adjustment instability; inaccurate observer–object distance estimates may also compromise object-motion estimates.Despite these issues, 3D experts show the smallest group-wise performance degradation in dynamic scenes.Source: §5.2 · p. 9
6 CONCLUSION
The authors present DSI-Bench as a resource for studying dynamic spatial intelligence and model performance.
6 CONCLUSION
- DSI-Bench combines over 1,700 VQA pairs from nearly 1,000 videos with spatio-temporal flipping, group-wise robustness evaluation, model benchmarking, and statistical and error analyses.The authors hope it will support progress in dynamic spatial perception and reasoning.Source: §6 · p. 9
7 ETHICS STATEMENT
The authors state their commitment to follow the ICLR Code of Ethics in their conference participation.
7 ETHICS STATEMENT
- The authors commit to adhering to the ICLR Code of Ethics in all aspects of their conference participation.Source: §7 · p. 10
8 REPRODUCIBILITY STATEMENT
The paper states that it will release its dataset and evaluation code after the review process to support reproducibility.
8 REPRODUCIBILITY STATEMENT
- The authors state that they will release the dataset and evaluation code after the review process.They present this release as part of ensuring reproducibility.Source: §8 · p. 10
- The planned release is contingent on completion of the review process.Source: §8 · p. 10
- The statement links reproducibility to releasing both data and evaluation code.Source: §8 · p. 10
A.1 FULL RESULTS OF VLM FREE-FORM REASONING
Table 3 reports free-form reasoning results for 12 VLMs and defines sample-wise and group-wise accuracy over augmented videos and video groups.
A.1 FULL RESULTS OF VLM FREE-FORM REASONING
- Table 3 covers free-form reasoning results for 12 VLMs.Source: Table 3 · p. 12
- Sample-wise accuracy treats each augmented video as an independent sample.Source: Table 3 · p. 12
- Group-wise accuracy is the fraction of video groups with at least three correct predictions among variants of the same original video.Source: Table 3 · p. 12
A.2 SYSTEM PROMPT FOR RAWQA AND FFR
The appendix specifies distinct RAWQA and FFR response formats, and describes LLM assistance with paper expression, formatting, and experimental-result visualizations.
A.2 SYSTEM PROMPT FOR RAWQA AND FFR
- RAWQA asks the model to answer a video question by selecting one correct option.Source: §A.2 · p. 12
- RAWQA requires the selected answer to use the prescribed answer-tag format.Source: §A.2 · p. 12
- FFR asks for detailed step-by-step reasoning before the answer.Source: §A.2 · pp. 12–13
- FFR also requires one correct choice from the listed options.Source: §A.2 · pp. 12–13
- The FFR response format separates reasoning and answer using prescribed tags.Source: §A.2 · p. 13
A.4 USE OF LLMS
- LLMs are used to refine expression and assist with paper formatting.Source: §A.4 · p. 14
- For experimental-result summaries, LLMs distill reasoning to help generate more effective visualizations.Source: §A.4 · p. 14