PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models

Shengchao Zhou, Yuxin Chen, Yuying Ge, Wei Huang, Jiehong Lin, Ying Shan, Xiaojuan Qi

arXiv:2512.20557v1cs.CV

TL;DR

VLMs remain limited in reasoning about how object geometry and relationships evolve in 3D over time, with scalable 4D-aware training resources scarce. DSR Suite builds in-the-wild training and evaluation data and introduces GSM to select question-relevant geometric priors. Experiments report improved dynamic spatial reasoning with Qwen2.5-VL-7B while maintaining general video understanding performance.

  • Problem

    VLMs remain limited in dynamic spatial reasoning, while existing dynamic-scene studies have narrow diversity, limited reasoning types, and little training data.

    Source: §1 · p. 2

  • Method

    DSR Suite generates DSR-Train and human-refined DSR-Bench from in-the-wild video geometry, and uses GSM to select question-relevant 4D priors as compact geometry tokens.

    Source: §1 · p. 2

  • Results

    Qwen2.5-VL-7B with GSM achieves state-of-the-art performance on DSR-Bench while maintaining competitive results across general multimodal benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The suite provides resources for studying geometric and temporal reasoning in realistic, dynamic settings.

    Source: §1 · p. 2

    The pipeline uses relative, non-metric 3D structure for qualitative trend-based questions because vision foundation models cannot reliably produce metric-scale structure without scale awareness.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Vision-language models (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.

1. Introduction

VLMs remain limited in dynamic 3D spatial reasoning, while existing data and methods leave gaps in scalable training, comprehensive evaluation, and effective geometric-prior integration. DSR Suite addresses these gaps with in-the-wild training and evaluation resources and a question-guided Geometry Selection Module (GSM).

Motivation

  • Dynamic spatial reasoning concerns how object geometry and relationships evolve in 3D over time, a capability relevant to interactive systems in robotics, autonomy, AR/VR, and embodied intelligence.The paper identifies limited scalable resources, narrow dynamic-scene coverage, and geometric-prior integration that can compromise general-purpose performance as research gaps.Source: §1 · p. 2

Contributions

  • DSR Suite builds DSR-Train and human-refined DSR-Bench from in-the-wild videos annotated with camera poses, local point clouds, masks, orientations, and 3D trajectories.The benchmark evaluates object- and scene-level understanding, multi-object interactions, viewpoint transformations, and fine-grained temporal reasoning.Source: §1 · p. 2
  • GSM uses two stacked Q-Formers to condense question semantics and extract question-relevant 4D reconstruction knowledge into compact geometry tokens.The targeted selection is intended to suppress irrelevant geometric noise and preserve general multimodal understanding.Source: §1 · p. 2
  • Qwen2.5-VL-7B with GSM achieves state-of-the-art performance on DSR-Bench while maintaining competitive results across general multimodal benchmarks.The authors contrast this approach with adding a large set of geometric features, which can improve DSR performance but degrade general-benchmark performance.Source: §1 · p. 2

2. Related Work

Prior spatial-reasoning resources and methods leave gaps in dynamic-scene coverage, training supervision, and integration of geometric knowledge. DSR Suite contributes in-the-wild training and evaluation data and selectively integrates question-relevant 3D knowledge into VLMs.

Spatial Reasoning Data

  • Many existing spatial-reasoning datasets focus on static scenes, while two-image motion benchmarks have short temporal horizons and video benchmarks often use domain-constrained sources.The cited video datasets also tend to have limited question diversity, weak 3D requirements, coarse answers, and little training data.Source: §2 · p. 3
  • DSR-Train and DSR-Bench use in-the-wild videos, diverse questions, randomized target objects and viewpoints, and fine-grained answers for dynamic spatial reasoning.The authors present these resources as addressing limitations in existing training and evaluation data.Source: §2 · p. 3

Spatial Reasoning Models

  • Existing VLM methods inject 3D information through coordinate embeddings or by fusing features from 3D foundation models with vision tokens.The paper notes that task-specific injected features may degrade performance on benchmarks unrelated to spatial reasoning.Source: §2 · p. 3
  • GSM selectively extracts question-relevant 3D knowledge into compact tokens, reducing interference with general multimodal understanding while strengthening dynamic spatial reasoning.This contrasts with direct fusion approaches described as potentially degrading general video understanding.Source: §2 · p. 3

3. Dynamic Spatial Reasoning Data

DSR Suite builds training and evaluation data from in-the-wild videos using extracted 3D cues and question-answer generation. Its benchmark combines diverse scenes and question types with strong 3D demands and fine-grained procedural answers.

3. Dynamic Spatial Reasoning Data

  • The pipeline has three stages: video curation, extraction of object- and scene-level 3D cues, and generation of multiple-choice questions with fine-grained answers.DSR-Bench question-answer pairs receive further human refinement for accuracy.Source: §3 · p. 3

3.1. Video Curation

  • The authors filter clips with negligible motion, retain videos of 20–120 seconds, and obtain 10,000 training videos and 575 evaluation videos.Training candidates are filtered using captions, while evaluation candidates are selected by prompting on video content.Source: §3.1 · p. 3

3.2. Geometric Clue Extraction

  • Because monocular foundation models do not reliably provide metric-scale 3D structure, the pipeline uses relative geometry for qualitative trend-based questions; DSR-Train samples 32 frames per video and DSR-Bench samples at 1 FPS.The authors select a random time sub-interval for each question to accommodate objects entering or leaving the scene.Source: §3.2 · p. 3
  • Scene-level extraction estimates camera poses and local point clouds; object masks are lifted into 3D to form trajectories, with orientations estimated for agent classes and omitted for non-agents.Objects not continuously visible are pruned to support reliability.Source: §3.2 · pp. 3–4

3.3. Data Generation

  • Template questions vary target objects, viewpoints, and viewpoint mobility, transform object centers into the selected reference frame, and use qualitative choices with procedural answers derived from adjacent-frame comparisons.A second, non-template family uses DeepSeek-R1 to generate free-form questions and answers conditioned on trajectories, object identities, and viewpoints; together the families target egocentric and allocentric queries, continuous motion, and multi-object interactions.Source: §3.3 · p. 5

3.4. Benchmark Statistics and Comparisons

  • DSR-Bench uses in-the-wild videos, unlike many video benchmarks drawn from domains such as autonomous driving or human-object interaction, and emphasizes multi-object and multi-viewpoint reasoning.Table 3 compares DSR-Bench with prior spatial-reasoning benchmarks across different aspects.Source: §3.4 · p. 5, Table 3 · p. 6
  • DSR-Bench has strong 3D demand: 34% of questions were judged to require object-level knowledge and 18% to require scene-level knowledge under the paper's checks.The paper classifies its demand as strong when both knowledge types are required.Source: §3.4 · p. 6
  • 78% of DSR-Bench answers were judged to describe the changing procedure of the asked attribute across video, compared with 2% for DynSuperCLEVER, 19% for VLM4D, 22% for STI-Bench, and 18% for OmniSpatial.The authors use this comparison to characterize prior benchmarks as dominated by coarse-grained answers.Source: §3.4 · p. 6

4. Geometric Prior Enhanced VLM for Dynamic Spatial Reasoning

The Geometry Selection Module addresses the trade-off between adding geometric priors and preserving general video understanding. It selects compact, question-relevant geometry tokens and combines them with vision and question tokens for the LLM.

4. Geometric Prior Enhanced VLM for Dynamic Spatial Reasoning

  • GSM selectively retrieves question-relevant geometry rather than exposing the VLM to all 3D tokens, targeting noisy geometric cues that can overwhelm the model and reduce generality.The paper attributes degradation in general video understanding to task-specific overfitting from noisy geometric features.Source: §4 · p. 6
  • Two stacked Q-Formers first condense question semantics into language-conditioned queries, then use those queries to retrieve relevant geometry from 3D tokens.The second Q-Former outputs a fixed-size set of geometry tokens.Source: §4 · p. 7
  • GSM concatenates its geometry tokens with vision and question tokens before passing the combined stream to the LLM.The fixed-size, task-aligned summary avoids exposing the VLM to long, variable-length 3D tokens.Source: §4 · p. 7
  • The paper characterizes GSM as architecture-agnostic, parameter-efficient through fixed N queries, and robust to question length through language condensation.It reports strong DSR gains while maintaining performance on general video and VLM benchmarks.Source: §4 · p. 7

5. Experiments

On DSR-Bench, the proposed model leads the compared VLMs across subtasks and in average performance. Experiments also compare training paradigms, query counts, and DSR-Train sizes, finding trade-offs between spatial reasoning and general video performance.

5.1. Experimental Settings

  • The experiments use Qwen2.5-VL-7B with GSM, trained for one epoch on 50K DSR-Train question-answer pairs while freezing the vision encoder.GSM uses 32 learnable queries; the learning rate is 2×10−7 and the batch size is 32.Source: §5.1 · p. 7

5.2. Comparison with State-of-The-Arts

  • The proposed model achieves the best performance across DSR-Bench subtasks and the highest average, while non-spatial models are only marginally above random guess and static-scene spatial models still fall short.The comparison includes proprietary, video-understanding, general-purpose, and spatial-reasoning VLMs.Source: §5.2 · pp. 7–8

5.3. Ablation Studies

  • Training on DSR-Train improves performance on DSR-Bench and other dynamic spatial reasoning benchmarks; GSM improves spatial benchmarks over SFT and is competitive with Addition while retaining general video understanding.The comparison uses VLM4D and STI-Bench for spatial reasoning and Video-MME for general video understanding.Source: §5.3 · p. 8, Table 5 · p. 8
  • More GSM queries improve dynamic spatial reasoning but harm general video understanding and lower average performance, making query count a trade-off.The tested query counts were 8, 16, 32, and 64.Source: §5.3 · p. 8, Table 6 · p. 8
  • Increasing DSR-Train size from 5K to 50K question-answer pairs improves DSR-Bench performance.The reported training-set sizes are 5K, 10K, 20K, and 50K.Source: §5.3 · p. 8, Figure 5 · p. 8

6. Conclusion

The conclusion presents DSR Suite as a unified framework for dynamic spatial reasoning in VLMs, combining training data, a human-refined benchmark, and GSM. It reports superior DSR-Bench results for Qwen2.5-VL-7B with GSM without regressions on general benchmarks.

6. Conclusion

  • DSR Suite combines an automated pipeline for DSR-Train with a human-refined DSR-Bench for evaluating dynamic spatial reasoning on in-the-wild videos.The data emphasize viewpoint transformations, multi-object interactions, and fine-grained procedural answers.Source: §6 · p. 8
  • GSM selectively integrates geometric priors through two stacked Q-Formers to avoid noisy geometric overload and preserve general video understanding capability..Source: §6 · p. 8
  • Trained on DSR-Train, Qwen2.5-VL-7B with GSM achieves superior DSR-Bench results without regressions on general benchmarks..Source: §6 · p. 8

Supplementary Material

The supplementary material adds implementation details, further experiments and analysis, and examples of question-answer pairs from DSR-Train and DSR-Bench.

Supplementary Material

7. Implementation Details

The implementation details specify evaluation controls and describe how objects, prompts, questions, and answers are represented or generated. The QA pipeline combines template-based and non-template-based questions with procedural answer construction.

7.1. Training Data Details

  • DSR-Train contains 50K question-answer pairs, and sampled training subsets preserve its question-type distribution.That distribution is described as similar to DSR-Bench’s.Source: §7.1 · p. 11, Figure 6 · p. 11

7.2. Evaluation Details

  • All VLMs receive 32 uniformly sampled frames and must output a final choice directly; timestamps are appended for models without native absolute-time encoding, and Video-MME subtitles are excluded.These controls are used to standardize evaluation inputs and responses.Source: §7.2 · p. 11

7.3. QA Generation Details

  • Objects are identified by category and bounding-box coordinates tied to a timestamp or the queried interval’s first or last frame.For viewpoint questions, coordinates refer to the specified timestamp; for direction prediction, initial or final boxes mark the interval endpoints.Source: §7.3 · p. 11
  • Question generation mixes predefined templates with DeepSeek-R1-generated non-template questions, while answers derive from adjacent-frame attribute comparisons and randomized-length choice sequences.The final second is hidden for direction-prediction questions because those frames determine the movement direction.Source: §7.3 · pp. 11–12, Table 7 · p. 13, Table 8 · p. 13

8. Complementary Experiments

Complementary experiments analyze model comparisons, question-type mixtures, base-model transfer, combined static and dynamic training, and downstream agent tasks. Together, they examine the scope of DSR-Train and GSM across these settings.

8.1. Additional Result Analysis

  • VLM-3R and VG-LLM, despite training only for static spatial reasoning, outperform proprietary models in the reported comparison; VideoRefer’s marginal gain over LLaVA-Video-7B suggests bounding-box references do not explain poor DSR-Bench performance.The authors interpret the results as indicating a need for 3D-relevant data and insufficient current dynamic-spatial-reasoning capability.Source: §8.1 · p. 14

8.2. Effect of Training Question Distribution

  • Both template-based and non-template-based questions contribute to strong overall performance, with template-based questions recommended as the majority.The experiments hold the total training set at 20K question-answer pairs across four distribution settings.Source: §8.2 · p. 14, Table 9 · p. 14

8.3. Ablation on Different Base Model

  • On Qwen3-VL-8B-Instruct, DSR-Train improves dynamic spatial reasoning, GSM further improves over SFT, and GSM retains broader video-understanding ability unlike Addition.The comparison covers DSR-Bench, STI-Bench, VLM4D, and Video-MME.Source: §8.3 · p. 14, Table 10 · p. 14

8.4. Performance with Data Mixture

  • Combining static and dynamic spatial-reasoning data yields the best simultaneous performance on DSR-Bench and VSI-Bench while preserving general understanding; static-only training does not enable dynamic reasoning.The static data comprise 800K question-answer pairs from VLM-3R and Cambrian-S.Source: §8.4 · pp. 14–15, Table 11 · p. 15

8.5. Extension to Agent Tasks

  • The DSR-Train-fine-tuned model shows an advantage on MineDojo agent tasks involving dynamic objects, evaluated over three seeds and 200 episodes per task.MineRL data are used for training because MineDojo provides videos and task instructions without intermediate action labels.Source: §8.5 · p. 15, Table 12 · p. 15

9. Visualization

The visualization section presents video and question-answer examples from DSR-Train and DSR-Bench, illustrating the resources’ sources and evaluation aspects.

9. Visualization

  • The examples illustrate in-the-wild video sources and evaluation of multi-object interaction, viewpoint transformation, and fine-grained temporal reasoning.Source: §9 · p. 15
  • Arrows indicate movement from the camera’s viewpoint, which can differ when the viewpoint changes.Source: §9 · p. 15
  • Figures 10–12 show video and question-answer examples from DSR-Train and DSR-Bench.Source: §9 · p. 15, Figure 10 · p. 16, Figure 11 · p. 17
Loading 2512.20557v1…