PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Geometry: Active Geometry Integration for Spatial Reasoning

Haoyuan Li, Qihang Cao, Tao Tang, Kun Xiang, Zihan Guo, Jianhua Han, JiaWang Bian, Hang Xu, Xiaodan Liang

arXiv:2602.06037v5cs.CV

TL;DR

Existing geometry integration strategies expose geometric information passively, despite spatial cues being task-dependent and selectively relevant. GeoThinker enables active retrieval through Spatial-Grounded Fusion and Importance Gating, achieving state-of-the-art spatial reasoning and transfer across downstream scenarios, while remaining sensitive to initial geometric encoding accuracy.

  • Problem

    Existing approaches commonly expose geometry as a uniform stream, overlooking task-dependent and spatially selective cues.

  • Method

    GeoThinker uses Spatial-Grounded Fusion with frame-strict cross-attention and Importance Gating to actively integrate task-relevant geometry at selected VLM layers.

  • Results

    GeoThinker achieves consistent gains across spatial benchmarks, reaches 72.6 on VSI-Bench, remains robust under debiased and long-video evaluation, and transfers to embodied referring and autonomous driving.

  • Takeaways & Limitations

    Active geometry integration is presented as a promising path toward stronger spatial reasoning and a more structured understanding of the 3D world.

  • Takeaways & Limitations

    GeoThinker is sensitive to the accuracy of initial geometric encodings, so encoder information loss can propagate through fusion modules.

Abstract

from arXiv · show

Recent progress in spatial reasoning with Multimodal Large Language Models (MLLMs) increasingly leverages geometric priors from 3D encoders. However, most existing integration strategies remain passive: geometry is exposed as a global stream and fused in an indiscriminate manner, which often induces semantic-geometry misalignment and redundant signals. We propose GeoThinker, a framework that shifts the paradigm from passive fusion to active perception. Instead of feature mixing, GeoThinker enables the model to selectively retrieve geometric evidence conditioned on its internal reasoning demands. GeoThinker achieves this through Spatial-Grounded Fusion applied at carefully selected VLM layers, where semantic visual priors selectively query and integrate task-relevant geometry via frame-strict cross-attention, further calibrated by Importance Gating that biases per-frame attention toward task-relevant structures. Comprehensive evaluation results show that GeoThinker sets a new state-of-the-art in spatial intelligence, achieving a peak score of 72.6 on the VSI-Bench. Furthermore, GeoThinker demonstrates robust generalization and significantly improved spatial perception across complex downstream scenarios, including embodied referring and autonomous driving. Our results indicate that the ability to actively integrate spatial structures is essential for next-generation spatial intelligence. Code can be found at https://github.com/Li-Hao-yuan/GeoThinker.

1. Introduction

GeoThinker addresses the limitations of passive geometry fusion by enabling MLLMs to actively retrieve task-relevant geometric cues according to internal reasoning demands. It reports strong spatial reasoning, robustness, and downstream transfer across multiple settings.

  • Geometry encoders provide fine-grained spatial priors that help MLLMs move beyond 2D semantic perception toward structured 3D understanding.
  • Passive integration exposes geometry uniformly, overlooking task-dependent and spatially selective cues and causing semantic-geometry misalignment and redundant noise.
  • GeoThinker enables active perception by retrieving and integrating geometric evidence conditioned on the MLLM’s internal reasoning needs.
  • Spatial-Grounded Fusion uses semantic visual priors to query task-relevant geometry through frame-strict cross-attention, while selected VLM layers and Importance Gating focus integration on relevant structures.
  • 72.6 on VSI-Bench marks GeoThinker’s peak score and a new state of the art, alongside robust debiased evaluation and transfer to embodied referring and autonomous driving.
  • GeoThinker is presented as robust across debiased and long-video settings and as effective across downstream spatial scenarios.

2. Related Work

Related work incorporates spatial intelligence through general MLLM advances, explicit geometry priors, scaling, reasoning-centric training, and passive fusion architectures. GeoThinker is positioned as an active alternative that selectively integrates task-relevant geometry.

  • Recent MLLM research pursues spatial intelligence through general multimodal understanding, explicit 3D cues, data scaling, and reasoning-centric training.
  • GeoThinker introduces Spatial-Grounded Fusion to integrate task-relevant geometric cues according to internal semantic demands through frame-strict cross-attention.
  • Geometry-prior methods commonly use passive input-level fusion, dedicated geometric and semantic experts, or feature distillation and alignment.
  • Some prior architectures require large-scale multi-task training and additional objectives, motivating more efficient geometry integration mechanisms.

3. Method

GeoThinker integrates 3D geometry into MLLM reasoning through Spatial-Grounded Fusion, replacing indiscriminate fusion with frame-constrained, importance-weighted retrieval of task-relevant cues. The method aligns semantic and geometric tokens, regulates geometry flow, and applies fusion selectively across backbone layers.

  • Spatial-Grounded Fusion: GeoThinker introduces Spatial-Grounded Fusion to integrate task-relevant geometric cues conditioned on internal semantic demands via frame-strict cross-attention.The framework shifts from passive feature addition toward active geometry integration.
  • Architecture: The architecture combines 2D semantic visual features with VGGT geometry features, resampling geometry to the backbone’s token grid for spatial alignment.The geometry encoder models inter-frame dependencies, while interpolation reconciles differing patch resolutions.
  • Frame-wise Constraints: Frame-wise constraints restrict each semantic query to geometric cues from the same frame, preserving spatial correspondence and reducing cross-image interference.This design is intended to maintain generalization for multi-view and video inputs.
  • Importance Gating: Importance Gating predicts region-level importance from image hidden states and adds a bias to cross-attention logits, emphasizing relevant geometry while suppressing redundant cues.The gating mechanism uses a lightweight MLP and an additive attention bias.
  • Global Scaling: A global learnable scalar α, initialized to 0, controls the overall intensity of geometric injection before the SGF output is added to the residual stream.This scaling mechanism balances semantic processing with geometric querying.
  • Layer Selection: Fusion layers are selected with offsets, excluded early visual layers, and a final-layer buffer so geometry supports reasoning without disrupting visual processing or response generation.The constraints vary fusion timing by task setting and preserve instruction-following priors.

4. Experiments

GeoThinker is evaluated across spatial benchmarks and downstream embodied and autonomous-driving tasks, showing strong benchmark performance, robustness, and transfer. The experiments also examine general-video mixtures, long-context evaluation, and component behavior.

  • Spatial reasoning benchmarks: GeoThinker consistently outperforms VG-LLM on VSI-Bench, reaching average scores of 48.9 at 3B and 50.5 at 7B.The comparison keeps the backbone and encoders fixed while replacing the model design.
  • Spatial reasoning benchmarks: GeoThinker reaches peak AVG. scores of 60.43 with Qwen2.5VL-7B and 62.23 with Qwen3VL-8B across diverse spatial-temporal tasks.The models outperform specialized general and spatial models as well as leading proprietary models.
  • Robustness and generalization: Adding general video data improves VideoMME by +5.7 and MVBench by +26.3 while improving VSI-Bench by +0.6.This result contrasts with the reported VSI-Bench reduction for Cambrian-S-7B after adding 3M general-video samples.
  • Robustness and generalization: GeoThinker reaches 68.1 on VSI-Debiased with 128 frames, surpassing Cambrian-S-7B at 59.9 despite training with at most 8/32 frames.The evaluation tests zero-shot extrapolation to longer inference contexts.
  • Downstream transfer: GeoThinker improves RefSpatial-Bench by +1.00% on location, +1.00% on placement, and +3.89% on unseen relations, yielding a +1.66 gain in average accuracy.The larger unseen-split gain indicates stronger compositional generalization within the reported evaluation.
  • Downstream transfer: GeoThinker improves autonomous-driving DAC by +3.7% and TTC by +2.2%, while ablations raise the average score from 47.45 to 48.93.The ablation adds frame-wise constraints and importance gating after spatial-grounded fusion.

5. Conclusion

The conclusion presents GeoThinker as an active geometry-integration framework that replaces uniformly exposed geometry with selective, task-conditioned retrieval. Spatial-Grounded Fusion and Importance Gating support gains across benchmarks, debiased and long-video evaluation, and downstream tasks.

  • Conclusion: GeoThinker shifts geometry integration from passive fusion to active perception through task-conditioned retrieval of geometric cues.The framework is motivated by semantic–geometry misalignment and redundant noise from uniformly exposed geometry.
  • Conclusion: Spatial-Grounded Fusion uses frame-strict cross-attention, while Importance Gating concentrates integration on task-relevant regions.The design uses semantic visual priors to query geometric evidence within each frame.
  • Conclusion: GeoThinker shows consistent gains across spatial benchmarks, robust performance under debiased and long-video evaluation, and transfer to RoboRefer and ReCogDrive.These results are presented as evidence supporting active geometry integration for spatial intelligence.
  • Implementation scope: The framework applies SGF across VLM layers, with the fusion ratio ρ controlling the proportion of integrated LLM layers.The reported settings use ρ=0.5 out of domain and ρ=0.75 in domain, while consistently skipping the final 25% in-domain layers.
  • Implementation scope: Importance Gate intensity is controlled by ϵ, using ϵ=1e-6 out of domain and ϵ=0.1 in domain.The smaller value produces more concentrated spatial-texture control, whereas the larger value provides smoother control for mixed video domains.
  • Implementation scope: The efficiency evaluation compares Total FLOPs and Inference Latency against native QwenVL and VG-LLM using 32 sampled frames per scene.This appendix analysis evaluates computational cost on the VSI-Bench test set.

C.1. Analysis of Total FLOPs

The efficiency analysis finds that SGF adds little computational cost, while spatial compression substantially reduces the cost of 32-frame inputs. Inference latency is mainly affected by sequential VGGT processing rather than SGF itself.

  • FLOPs: Spatial compression in the 32-frame setting significantly reduces total FLOPs, producing lower FLOPs than the original baseline on larger Qwen backbones.This is reported for Qwen-2.5VL-7B and Qwen3-VL-8B.
  • FLOPs: The number of visual tokens, rather than the fusion architecture itself, is the dominant factor influencing total FLOPs.The conclusion follows the comparison between dense 8-frame and spatially compressed 32-frame settings.
  • Inference latency: In the 8-frame setting, GeoThinker has latency nearly identical to VG-LLM, and in the 32-frame setting it is consistently faster.The speed advantage is attributed to effective spatial compression.
  • Inference latency: VGGT-based models are significantly slower than native QwenVL because the 2D encoder and VGGT operate sequentially.VGGT feature processing before the LLM dominates time consumption.
  • Inference latency: Spatial compression provides a better trade-off between temporal context and inference speed for long-video understanding than dense sampling.The conclusion concerns the 32-frame version of the method.

D.1. Performance Analysis

The performance analysis shows that moderate fusion ratios outperform full-layer integration across model scales. A ratio of ρ=0.5 provides the strongest reported balance, while late-layer integration can disrupt language decoding.

  • Fusion-ratio ablation: ρ=1.0 causes a catastrophic drop to an average score of 0.41, whereas moderate ratios from 0.25 to 0.75 improve over baseline.The ablation is conducted with the Qwen2.5VL-3B backbone on VSI-Bench.
  • Fusion-ratio ablation: ρ=0.5 reaches the best Qwen2.5VL-3B-8frame average score of 48.93.Ratios of ρ=0.25 and ρ=0.75 also remain effective.
  • Fusion-ratio ablation: ρ=0.5 reaches the highest Qwen2.5VL-7B average score of 50.50, exceeding ρ=0.25 at 49.21.The result is consistent with the 3B backbone comparison.
  • Layer sensitivity: Integrating SGF into all LLM layers can disrupt [EOS] prediction because late-stage layers are specialized for linguistic structure.The authors hypothesize that intermediate layers better internalize task-relevant geometric textures.
  • Layer sensitivity: The 3B and 7B models differ in layer depth, with 36 LLM layers in the 3B model and 28 in the 7B model.Consequently, the same fusion ratio covers different absolute numbers of layers across model scales.
  • Layer sensitivity: The analysis concludes that ρ=0.5 provides enough depth for spatial relationships without compromising instruction following and termination capabilities.This conclusion is reported across the evaluated model scales.
  • Additional evaluation: GeoThinker Qwen3-VL-8B-32frame ranks 6th overall on the EASI Leaderboard with an average score of 55.0.The result is reported for the in-domain evaluation setting.

E.1. Data Efficiency

GeoThinker demonstrates strong data efficiency, outperforming larger-data training approaches while improving substantially over its original backbone. It also generalizes beyond its training frame counts, though several specialized benchmarks remain below top proprietary models.

  • Data Efficiency: GeoThinker Qwen3-VL-8B-32frame outperforms SenseNova-SI-1.1-QwenVL3-8B by 2.8 points, scoring 55.0 versus 52.2.GeoThinker uses 1.8M training samples, compared with 8M for the SenseNova variant.
  • Data Efficiency: GeoThinker’s spatial gains are attributed to Spatial-Grounded Fusion and its training strategy extracting effective representations from limited data.The passage presents this as an interpretation of the observed data-efficiency gap.
  • Backbone Comparison: GeoThinker improves over Qwen3-VL-8B-Instruct by 7.7 points, reaching 55.0 versus 47.3.The improvement is especially large on VSI and MindCube: 72.6 versus 57.9, and 83.0 versus 29.4, respectively.
  • Remaining Gaps: GeoThinker remains below top-tier proprietary models on MMSI, BLINK, and 3DSRBench.The paper identifies these benchmarks as specific areas for improvement.
  • Remaining Gaps: The paper proposes broader visual perception and refined 3D structure-from-motion capabilities as future directions.These directions are intended to complement the model’s current spatial-grounded features.

G. Additional comparisons on VSTI-Bench

On VSTI-Bench and PhysGame, GeoThinker performs strongly in spatial and physical understanding, including relative-position reasoning and quantitative camera-displacement estimation. Its performance remains below human level in complex metric depth estimation, and adding general video data slightly reduces PhysGame accuracy.

  • VSTI-Bench: 67.4 average score places GeoThinker first on VSTI-Bench, 29.2 points above GPT-4o and 8.6 points above VLM-3R-7B.The cited comparison reports GPT-4o and VLM-3R-7B scores of 38.2 and 58.8, respectively.
  • Spatial Relations: 93.6% accuracy on Object-Object Relative Position approaches Human Level at 97.5% and exceeds GPT-4o at 58.1%.Similar trends are reported for camera movement direction and relative distance tasks.
  • Metric Estimation: 45.8 on camera displacement nearly doubles GPT-4o’s 23.4 and approaches human performance at 46.8.The result addresses quantitative geometric reasoning rather than only qualitative spatial perception.
  • Remaining Gap: GeoThinker scores 67.4 versus Human Level at 77.0, with the remaining gap particularly evident in absolute distance estimation.The paper identifies this as an area requiring further research toward human-like metric depth precision.
  • PhysGame: 56.9% average accuracy on PhysGame surpasses PhysVLM-SFT at 56.7% and GPT-4o-0806 at 56.1%.This result is reported without the additional video mixture.
  • PhysGame: Adding a 430k-sample general video mixture lowers PhysGame average accuracy from 56.9% to 55.7%.The evaluation uses only 8 frames per video scene, while performance remains competitive across all 12 categories.

I. Additional comparisons on general benchmark

Additional evaluations show that GeoThinker retains competitive general-benchmark performance and transfers to dynamic scenes without task-specific fine-tuning. Visualizations indicate selective grounding of landmarks and spatial referents across limited-view and cluttered environments.

  • General Benchmarks: GeoThinker maintains highly competitive performance against Cambrian-S-7B on AI2D, MMB, and BLINK without significant degradation.The results are presented as evidence of generalized spatial understanding rather than overfitting to spatial datasets.
  • Dynamic Scenes: GeoThinker achieves superior zero-shot performance on DSR against VLM-3R and VG-LLM.The comparison is aligned for model scale and training data, and the evaluation targets dynamic scene understanding.
  • MindCube Visualization: On MindCube, importance scores concentrate on semantic landmarks and surrounding spatial context during multi-step movement queries.Examples include a pink plush toy and an electric fan across discrete, non-overlapping viewpoints.
  • VSI-Bench Visualization: On VSI-Bench, GeoThinker attends to queried spatial referents such as a towel, trash bin, backpack, and computer mouse in cluttered scenes.The visualization concerns continuous frame streams and fine-grained object localization.
  • VSI-Bench Visualization: The VSI-Bench visualization indicates that sharply localized attention helps filter task-irrelevant noise around partially obscured objects.The office example centers on identifying a backpack among similar desk items to resolve relative positioning.

K.3. Robustness to image resolution

GeoThinker remains focused on central objects and semantic features even after severe image-resolution degradation. This supports robustness to lost fine-grained information and is associated with selective geometric integration across frames.

  • Resolution Robustness: GeoThinker continues identifying and attending to core objects when resolution is reduced to 6.25% of the original, or 28×28 pixels.The model maintains focus on central-object texture and key semantic features across degradation levels.
  • Resolution Robustness: The method captures essential semantic information rather than relying solely on high-frequency visual details.The paper links this behavior to robustness across image-resolution variations.
  • Mechanistic Interpretation: Selective VGGT-feature integration provides a consistent spatial anchor for each frame and helps additional frames act as complementary geometric evidence.The paper associates this mechanism with scaling to longer video contexts without treating added frames as redundant noise or out-of-distribution inputs.

L.2. Limitation

GeoThinker’s primary limitation is its sensitivity to the accuracy of initial geometric encodings, because encoder-level information loss can propagate through fusion modules. The paper also evaluates robustness across substantially reduced image resolutions.

  • Failure pathway: Information loss at the encoder level can propagate through GeoThinker’s fusion modules.
  • Robustness setting: The robustness experiment evaluates performance across input quality ranging from original resolution down to 6.25%.Importance heatmaps and masked visualizations are used in this evaluation.
Loading 2602.06037v5…