PaperlayerView exact revision on arXiv

Source-linked AI summary

Make Geometry Matter for Spatial Reasoning

Shihua Zhang, Qiuhong Shen, Shizun Wang, Tianbo Pan, Xinchao Wang

arXiv:2603.26639v1cs.CVcs.AI

TL;DR

VLMs remain limited on spatial reasoning, and naive geometry-token fusion often leaves geometric cues underutilized as models rely on 2D visual cues. GeoSR combines geometry-unleashing masking with geometry-guided fusion to encourage effective geometry use, achieving consistent improvements over prior methods on static and dynamic benchmarks. The paper concludes that geometry can become actionable and controllable through these designs, while dataset quality remains a boundary on further progress.

  • Problem

    VLMs show strong general visual understanding but remain brittle on spatial reasoning, while naive geometry-token injection can be ignored or even harm performance.

  • Method

    GeoSR masks selected 2D vision tokens during training and adaptively routes geometry tokens through gated fusion where geometric evidence is needed.

  • Results

    GeoSR achieves consistent improvements over prior methods on both static and dynamic spatial reasoning benchmarks, especially in dynamic settings.

  • Takeaways & Limitations

    GeoSR makes geometry tokens more actionable and controllable for spatial reasoning instead of allowing them to be ignored or indiscriminately fused.

  • Takeaways & Limitations

    Current automatically or semi-automatically constructed datasets can contain geometrically ambiguous questions and annotations misaligned with visual evidence.

Abstract

from arXiv · show

Empowered by large-scale training, vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited. Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs. Nevertheless, we observe that naive token fusion followed by standard fine-tuning in this line of work often leaves such geometric cues underutilized for spatial reasoning, as VLMs tend to rely heavily on 2D visual cues. In this paper, we propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens. GeoSR introduces two key components: (1) Geometry-Unleashing Masking, which strategically masks portions of 2D vision tokens during training to weaken non-geometric shortcuts and force the model to consult geometry tokens for spatial reasoning; and (2) Geometry-Guided Fusion, a gated routing mechanism that adaptively amplifies geometry token contributions in regions where geometric evidence is critical. Together, these designs unleash the potential of geometry tokens for spatial reasoning tasks. Extensive experiments on both static and dynamic spatial reasoning benchmarks demonstrate that GeoSR consistently outperforms prior methods and establishes new state-of-the-art performance by effectively leveraging geometric information. The project page is available at https://suhzhang.github.io/GeoSR/.

1 Introduction

Spatial reasoning remains difficult for VLMs despite strong general image and video understanding, and naive geometry-token fusion often leaves geometric cues unused. GeoSR addresses this through masking and adaptive fusion, producing consistent improvements across static and dynamic benchmarks.

  • Motivation: VLMs perform strongly on general image and video understanding but remain brittle on spatial questions involving viewpoint changes, motion continuity, and quantitative spatiotemporal judgments.Spatial reasoning covers where objects are, how they relate in 3D space, and how relations evolve over time.
  • Motivation: Naive geometry-token fusion followed by standard fine-tuning can yield limited gains in static scenes and negative effects in dynamic scenes.These models may rely on appearance-driven shortcuts in 2D visual tokens instead of treating geometry as actionable evidence.
  • GeoSR: GeoSR uses Geometry-Unleashing Masking to weaken 2D appearance shortcuts during training and encourage consultation of geometry tokens.The framework strategically masks portions of 2D vision tokens so geometric information becomes necessary for spatial reasoning.
  • GeoSR: GeoSR uses Geometry-Guided Fusion, a gated routing mechanism that increases geometry-token contributions where geometric evidence is needed.This avoids treating geometry as uniformly helpful across all tokens and frames.
  • Results: GeoSR is validated on static and dynamic spatial reasoning benchmarks, where it delivers consistent improvements over prior methods.The contributions emphasize making geometry tokens actionable and controllable rather than easily ignored or indiscriminately fused.

2 Related Work

Spatial reasoning research extends VLMs toward measurable spatial and temporal relations, but existing approaches vary in their use of temporal modeling, spatial supervision, geometric priors, and reconstruction.

  • Video understanding: Video-capable VLMs commonly represent clips as sampled frames while improving temporal modeling and long-context handling for video understanding.These methods support tasks including video captioning and VideoQA, but spatial reasoning remains a distinct challenge.
  • Spatial reasoning with VLMs: Spatial reasoning concerns measurable relations such as distance, direction, and motion changes over time, especially under viewpoint variation and dynamic evolution.Benchmarks include static settings with viewpoint and visibility changes and dynamic settings with evolving relations.
  • Limitations of prior approaches: Explicit reconstruction pipelines can limit scalability in monocular in-the-wild videos, while indiscriminate geometry fusion may be ineffective or harmful without proper control.The related work contrasts these limitations with the paper’s focus on making geometry useful through controlled integration.

3 Method

GeoSR extends geometry-aware VLMs with masking and gated fusion so geometric cues become actionable for static and dynamic spatial reasoning. Its masking suppresses 2D shortcuts, while query-aware selection and learned routing control where geometry contributes.

  • 3.1 Preliminary: Geometry-Aware Framework: Geometry-aware VLMs add pretrained 3D geometry tokens to 2D vision tokens before the VLM generates an answer.The geometry branch extracts F_G, while the vision and prompt branches produce F_V and F_P; a fusion module forms F.
  • 3.2 Geometry-Unleashing Masking: For dynamic reasoning, bottleneck tokens attend to prompt tokens, then query geometry tokens to produce compact evidence and relevance scores for TopK masking.The attention-derived relevance score selects critical geometry locations, with interpolation used when geometry and vision resolutions differ.
  • 3.1 Preliminary: Geometry-Aware Framework: Naive fusion and standard fine-tuning can leave geometry tokens underutilized because the VLM continues relying on 2D visual cues.This motivates training designs that make geometric evidence actively useful rather than optional.
  • 3.2 Geometry-Unleashing Masking: Geometry-Unleashing Masking masks vision tokens during training to weaken appearance shortcuts and encourage spatial reasoning from geometry tokens.Static settings use random masks; dynamic settings identify question-relevant geometry and mask corresponding 2D vision positions.
  • 3.3 Geometry-Guided Fusion: Geometry-Guided Fusion uses a learned token-and-channel-wise gate α to adaptively mix masked vision features with geometry features.The gate enables geometry to dominate at locations where geometric evidence is useful instead of uniformly mixing or concatenating the streams.

4 Experiments

GeoSR is evaluated on static and dynamic spatial reasoning benchmarks, with ablations testing its masking and fusion components. Results show that adaptive geometry use is especially valuable for dynamic reasoning, while naive injection can be ineffective or harmful.

  • Benchmark Setup: GeoSR is evaluated on static and dynamic spatial reasoning benchmarks, including VSI-Bench and DSR-Bench.VSI-Bench contains over 5k QA pairs from 288 real videos, while DSR-Bench contains 1484 QA pairs from 575 in-the-wild videos.
  • Dynamic Spatial Reasoning: GeoSR achieves the highest score on each DSR-Bench subtask type and rivals the best-performing model.The result is reported across absolute, relative, and non-template dynamic reasoning categories.
  • Ablation Studies: Replacing Geometry-Guided Fusion with naive fusion reduces performance on both static and dynamic benchmarks.Removing both fusion mechanisms while retaining masking further degrades results, indicating that masking alone does not fully exploit geometry.
  • Ablation Studies: Removing Geometry-Unleashing Masking while retaining Geometry-Guided Fusion also causes a noticeable performance drop.The ablation indicates that adaptive routing alone does not prevent geometry underutilization without shortcut suppression.
  • Ablation Studies: On the dynamic benchmark, the no-geometry model outperforms naive geometry fusion, whereas the static benchmark shows only a moderate drop without geometry.The authors attribute the stronger dynamic benefit to motion and occlusion making appearance cues less stable.
  • Hyperparameters and Overhead: γ = 0.8 and β = 0.5 provide a balance between weakening appearance shortcuts, retaining context, and maintaining stability.Small values leave shortcuts largely intact, while overly large β can remove too much context and hurt stability.

5 Conclusion

The paper identifies geometry underutilization as a failure mode of naive fusion and standard fine-tuning in geometry-aware VLMs. GeoSR addresses it through masking and gated fusion, achieving consistent improvements, especially in dynamic settings.

  • Conclusion: Naive geometry-token fusion followed by standard fine-tuning can underutilize geometry, producing limited gains or harming spatial reasoning.The paper argues that informative geometry priors are not automatically treated as actionable evidence by the VLM backbone.
  • Conclusion: Geometry-Unleashing Masking weakens vision-driven shortcuts, while Geometry-Guided Fusion adaptively routes geometry evidence through a fine-grained gate.Together, the components encourage effective and reasonable geometry usage.
  • Conclusion: Experiments on static and dynamic benchmarks show consistent improvements over prior baselines, especially in dynamic settings.The reported conclusion is that GeoSR makes geometry matter for spatial reasoning.

A Qualitative Results

Qualitative results compare GeoSR with strong baselines on static and dynamic spatial reasoning examples. The examples are annotated with video IDs and accompanied by supplementary videos.

  • Qualitative Results: Figures 4 and 5 present qualitative GeoSR results on static and dynamic spatial reasoning benchmarks alongside VG-LLM and GSM.Each example includes a video ID, and the corresponding videos are provided in the supplementary video file.

B More Analysis

The analysis compares QFormer-style architectures across dynamic and static reasoning. QFormer helps aggregate temporal dynamics in the dynamic setting but does not improve static performance.

  • Architecture Analysis: The dynamic baseline uses a QFormer-style architecture, whereas the static baseline does not.The design is motivated by dynamic reasoning's need to aggregate motion patterns and temporal relations across frames.
  • Architecture Analysis: Introducing QFormer does not improve GeoSR performance on static spatial reasoning.The authors conclude that static reasoning relies less on explicit global temporal aggregation and more on grounding spatial judgments in geometric cues.

C Limitations and Future Work

GeoSR improves model-side exploitation of geometric cues, but progress is also constrained by ambiguity and annotation misalignment in current spatial-reasoning datasets. The paper therefore identifies geometry-aware question construction and annotation consistency as priorities for future work.

  • Limitations: Current datasets can contain geometrically ambiguous questions and annotations that do not perfectly align with visual evidence.The paper illustrates this issue with examples involving speed comparison and relative spatial relations in videos.
  • Limitations: Dataset-quality issues may affect the performance of all methods, not only GeoSR.
  • Future Work: Future work should improve geometry-aware question construction and annotation consistency alongside continued model-design advances.
Loading 2603.26639v1…