Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views

Zhangquan Chen; Manyuan Zhang; Xinlei Yu; Xufang Luo; Mingze Sun; Zihao Pan; Xiang An; Yan Feng; Peng Pei; Xunliang Cai; Ruqi Huang

Source-linked AI summary

Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views

Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xufang Luo, Mingze Sun, Zihao Pan, Xiang An, Yan Feng, Peng Pei, Xunliang Cai, Ruqi Huang

arXiv:2510.18632v4Oct 21, 2025cs.CVcs.AI

TL;DR

VLMs struggle to infer 3D spatial relationships from limited views because existing text- and 2D-based reasoning has limited geometric representation. 3DThinker introduces intrinsic 3D mentaling through supervised latent alignment followed by outcome-based trajectory optimization, and it outperforms previous methods across multiple benchmarks. The approach avoids dense 3D annotations and external geometry priors, while extremely sparse views remain a failure boundary.

Problem
VLMs have limited ability to extract 3D geometry and imagine complete scenes from limited views, while existing reasoning methods rely mainly on pure text or 2D visual cues.
Method
3DThinker trains VLMs in two stages by aligning generated 3D latents with a 3D foundation model, then optimizing full reasoning trajectories using outcome signals.
Results
3DThinker consistently outperforms strong baselines across multiple benchmarks and generalizes across different base VLMs.
Takeaways & Limitations
3DThinker integrates recoverable 3D representations into multimodal reasoning without dense geometric annotations or external priors during inference.
Takeaways & Limitations
Extremely sparse views can provide insufficient spatial constraints, causing inaccurate 3D reconstruction and erroneous spatial reasoning.

Abstract

from arXiv · show

Though recent advances in vision-language models (VLMs) have achieved remarkable progress across a wide range of multimodal tasks, understanding 3D spatial relationships from limited views remains a significant challenge. Previous reasoning methods typically rely on pure text (e.g., topological cognitive maps) or on 2D visual cues. However, their limited representational capacity hinders performance in specific tasks that require 3D spatial imagination. To address this limitation, we propose 3DThinker, a framework that can effectively exploits the rich geometric information embedded within images while reasoning, like humans do. Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training. Specifically, our training consists of two stages. First, we perform supervised training to align the 3D latent generated by VLM while reasoning with that of a 3D foundation model (e.g., VGGT). Then, we optimize the entire reasoning trajectory solely based on outcome signals, thereby refining the underlying 3D mentaling. Extensive experiments across multiple benchmarks show that 3DThinker consistently outperforms strong baselines and offers a new perspective toward unifying 3D representations into multimodal reasoning. Our code is available at https://github.com/zhangquanchen/3DThinker.

1. Introduction

Limited-view spatial reasoning requires models to recover 3D geometry and imagine complete scenes, capabilities that current text- and 2D-based methods lack. 3DThinker addresses this gap by integrating 3D mental representations into VLM reasoning through annotation-light, two-stage training.

1. Introduction: Limited-view spatial reasoning is essential for machine interaction with the real 3D world but remains difficult for VLMs.The challenge arises because limited views carry frame-dependent spatial semantics and do not directly reveal the full scene.
1. Introduction: Existing approaches rely on pure text, 2D cues, cognitive maps, or external models, limiting spatial representation or requiring additional annotations and tools.MindCube uses bird’s-eye-view annotations, while Ego3D invokes external models for referring expressions and depth estimation.
1. Introduction: 3DThinker enables VLMs to generate geometric representations during reasoning without densely labeled 3D data or external geometry encoders.The framework aligns generated 3D latents with a 3D foundation model rather than constructing raw 3D training data.
1. Introduction: The framework trains in two stages: supervised 3D-latent alignment followed by outcome-driven optimization of the full reasoning trajectory.The first stage uses constructed Chain-of-Thought data and 3D foundation-model features; the second preserves latent alignment while optimizing outcomes.
1. Introduction: 3DThinker consistently outperforms strong baselines across multiple benchmarks and generalizes across different base VLMs.The reported contributions also include recovery of 3D representations from latent reasoning through a projector.

2. Related Work

Prior multimodal reasoning methods operate mainly in textual, visual, or latent spaces, while spatial methods often add views, depth, point clouds, or cognitive maps. 3DThinker instead uses latent tokens as a bridge for intrinsic 3D mental reasoning without densely annotated spatial inputs.

2. Related Work: Prior reasoning methods are organized into pure-text, visual, and latent approaches, each incorporating different intermediate representations.Pure-text methods use textual rationales, visual methods generate or invoke visual evidence, and latent methods use hidden representations.
2. Related Work: Pure-text reasoning can limit performance when visual evidence cannot be adequately described in language, motivating visual and latent alternatives.Visual approaches include intrinsic multi-hop cues, external image tools, and continuous visual reasoning.
2. Related Work: The framework combines supervised 3D-latent alignment with outcome-based trajectory optimization to integrate 3D representations into multimodal reasoning.Figure 2 summarizes the transition from alignment with 3D foundation-model features to outcome-driven refinement.
2. Related Work: 3DThinker treats latent tokens as a bridge for mental-level 3D reasoning, aligning multimodal reasoning more closely with human cognition.Its 3D mentaling perspective differs from methods focused primarily on textual or 2D reasoning spaces.
2. Spatial Understanding: Spatial-understanding methods often enhance inputs with multiview, depth, or point-cloud information, or construct cognitive maps requiring complex collection and annotation.These methods provide additional spatial information rather than learning 3D mental representations directly within the reasoning process.

3. Methodology

3DThinker integrates compact 3D latent tokens into VLM reasoning, aligns them with VGGT geometry features, and refines entire trajectories using outcome-based reinforcement learning.

3.1. Data Generation: 3DThinker generates interleaved textual reasoning and 3D placeholders, whose latent representations encode imagined 3D scenes.The training data are synthesized from multi-view images, questions, and responses using prompted step-by-step reasoning.
3.2. Supervision for 3D Grounded Reasoning: Supervised training combines 3D feature alignment with cross-entropy losses so inserted 3D tokens remain compatible with surrounding textual reasoning.The total objective weights the 3D and textual losses with λ_3D and λ_text.
3.2. Supervision for 3D Grounded Reasoning: The projector maps VLM-generated 3D latents, together with image features, into a VGGT-compatible feature space for geometric alignment.The aligned projected features are compared with VGGT geometry features using a Frobenius loss.
3.3. Reinforced Spatial Mentaling: Reinforcement learning optimizes complete sampled trajectories with group-relative policy optimization while using outcome-driven rewards.The policy objective uses likelihood-ratio clipping, group-normalized advantages, and KL regularization against a reference policy.
3.3. Reinforced Spatial Mentaling: The reward combines 3D latent alignment, output-format compliance, and final-answer correctness, distributing outcome-based supervision across trajectory tokens.The 3D reward is the cosine similarity between projected latent features and VGGT features; format and answer rewards provide additional task signals.

4. Experiments

Across limited-view spatial benchmarks, 3DThinker consistently improves generalist VLM performance through staged training, while ablations identify design choices that preserve 3D reasoning and interpretability.

4.1. Benchmarking Generalist VLMs: 51.8%–108.8% gains on MindCube-Tiny and 18.1%–36.9% gains on Ego3D-Bench show consistent improvements across generalist VLM settings.For Qwen2.5-VL-3B, stage 1 reaches 62.7 from 33.2 on MindCube-Tiny, while stages 1 and 2 reach 50.8 from 39.1 on Ego3D-Bench.
4.2. Comparisons with Baselines: Across six spatial benchmarks, 3DThinker surpasses strong spatial baselines and improves over SpatialLadder-3B by 21.8% and VILASR-7B by 33.7% after stage 2.The method is reported to improve consistently across tasks, unlike task-specific overfitting and single-view weaknesses observed in some baselines.
4.3. Training Strategies: 75.2 vs. 70.7 shows a 6.4% improvement over cognitive-map-based SFT-then-RL, while outcome-based RL further enhances dynamic spatial capability.The supervised stage primarily targets static spatial understanding; whole-trajectory optimization in stage 2 adds gains.
4.4. Visualization: Projected 3D latents can reconstruct rough point clouds whose clearer regions typically correlate with prompt-relevant objects.During inference, hidden states from 3D special tokens are projected into VGGT feature space and processed by DPT.
4.5. Ablation Study: A latent size of about 12 achieves optimal performance, whereas larger sizes can trigger repetitive latent outputs that fail to produce final answers.Smaller sizes limit representational capacity, while larger sizes compromise natural expressive ability.
4.5. Ablation Study: 75.2 vs. 42.0 shows that placing 3D tokens inside natural-language reasoning disrupts coherence, while isolated beginning or end placement performs better.The preferred projector maps VLM hidden states into VGGT space, yielding 75.2 vs. 74.1 and recoverable 3D representations.
4.5. Ablation Study: Removing 3D alignment reduces performance to 68.3 and removing final-answer reward to 64.2, confirming both constraints are important in stage 2.Formatting reward has minimal impact, while final-answer reward provides the sole ground-truth supervision signal.

5. Conclusion

3DThinker enables VLMs to perform 3D spatial mentaling by using geometric information from images during reasoning without dense annotations or external priors.

3DThinker uses a two-stage scheme that distills geometric features from a pretrained 3D model, then optimizes reasoning trajectories using outcome signals while maintaining 3D alignment.
Across multiple benchmarks, 3DThinker outperforms previous methods, supporting 3D mentaling as a foundation for multimodal spatial reasoning.

7. Dataset

The evaluation uses benchmarks spanning limited-view, ego-centric, single-image, multi-view, video, and alternative-frame spatial reasoning.

MindCube-Tiny contains 1,050 examples with rotation, around, and among camera movements for limited-view spatial understanding.
Ego3D-Bench contains over 8,600 question–answer pairs covering distance, localization, motion, and travel-time reasoning in ego-centric and object-centric settings.
Additional benchmarks assess spatial understanding across real-world videos, single- and multi-view tasks, fundamental 2D/3D reasoning, and positional, attribute, and motion questions.
SPAR-Bench covers 20 spatial tasks across single-view, multi-view, and video settings with 7,207 manually verified question–answer pairs.
ViewSpatial-Bench evaluates egocentric and allocentric localization across five task types using over 5,700 question–answer pairs from more than 1,000 3D scenes.

8. Additional Results

Additional experiments show strong benchmark performance, preserved general VLM abilities, benefits from 3D alignment, and comparable feature-alignment alternatives.

Experiments extend the method to Qwen3-VL and compare it across benchmarks using stage 1 alone or stages 1 and 2.
3DThinker improves POPE accuracy by 2.91%, reaching 88.4 versus 85.9 for the base model, while remaining comparable on MME.
The two feature-alignment strategies achieve comparable performance at 74.1 and 75.2, while projecting VLM latents enables recovery of 3D representations.
Removing the 3D alignment loss reduces Qwen2.5-VL-3B performance from 62.7 to 54.1, compared with 53.4 for CoT SFT.

9. Training Curve

Training curves show slower convergence for supervised stage 1 and much faster stabilization for reinforcement-learning stage 2.

Stage 1 loss converges after approximately 20k steps at around 0.15, whereas stage 2 reward stabilizes after roughly 500 steps at about 2.7.
The curves track stage-1 total and 3D losses alongside stage-2 total, answer, and format rewards.
The training prompt supplies a question and ground-truth answer for chain-of-thought generation.

10. Details for Dataset Generation

The dataset-generation process uses GPT-4.1 to produce chain-of-thought reasoning from multi-view images, questions, and ground-truth responses. Generated reasoning is filtered against a required output format illustrated by the system prompts.

10. Details for Dataset Generation: GPT-4.1 generates chain-of-thought reasoning from multi-view image sets, questions, and ground-truth responses.The generated reasoning is denoted o.
10. Details for Dataset Generation: Accepted reasoning must begin with <output 3D> and follow the <think>...</think> <answer>...</answer> format.Fig. 8 and Fig. 9 show the prompts designed for generating this data.

11. Explanation of the Claim

The method’s annotation-free claim concerns the absence of manual geometric annotations for 3D mentaling, while its teacher-free evaluation shows an 8.93% improvement without the teacher model.

11. Explanation of the Claim: “Annotation-free” means that 3D mentaling does not use manual geometric annotations such as ground-truth point clouds.GPT-4.1 generates 3D placeholders, while VGGT provides the associated geometric features.
11. Explanation of the Claim: GPT-4.1 is used to generate 3D placeholders rather than manual geometric annotations for the 3D features.The passage distinguishes placeholder generation from the provision of actual geometric features by VGGT.
11. Explanation of the Claim: 68.3 versus 62.7 yields an 8.93% improvement without r3D, indicating robustness without the teacher model.The comparison is reported even in the absence of r3D.

12. Cost

The reported cost reduction comes from rapid near-optimal convergence and accelerated reinforcement-learning rollouts. For Qwen2.5-VL-3B on one H200 GPU, supervised and reinforcement training take 21.84 and 12.85 hours, respectively.

12. Cost: 21.84 hours for supervised training and 12.85 hours for reinforcement training are reported for Qwen2.5-VL-3B on one H200 GPU.The configuration uses a batch size of 1.
12. Cost: Near-optimal performance is typically reached within 500 training steps, contributing to the cost reduction.This convergence behavior is referenced alongside Fig. 7.
12. Cost: VLLM acceleration substantially improves reinforcement-learning rollout efficiency, further reducing cost.

13. More Visualization

Additional MindCube-Tiny visualizations expose 3DThinker’s reasoning outputs and reconstructed point clouds. A representative failure shows that extremely sparse views can produce structurally inconsistent reconstructions and incorrect spatial reasoning.

13. More Visualization: Reconstructed 3D point clouds visualize what the VLM is “thinking” during spatial reasoning on MindCube-Tiny.The visualizations include reasoning outputs and 3D imagery from 3DThinker-S1+S2Qwen2.5-3B.
13. More Visualization: The additional figures present reasoning processes, ground-truth answers, and reconstructed point-cloud imagery.Figures 10–16 provide further visualizations of the method.
13. More Visualization: Extremely sparse views can yield structural inconsistencies in the reconstructed point cloud and erroneous spatial reasoning.In the failure case, a wall is placed on the wrong side of a frontal window view, causing an incorrect rear-object position inference.