PaperlayerView exact revision on arXiv

Source-linked AI summary

Think3D: Thinking with Space for Spatial Reasoning

Zaibin Zhang, Yuhan Wu, Lianjie Jia, Yifan Wang, Zhongbo Zhang, Yijiang Li, Binghao Ran, Fuxi Zhang, Zhuohan Sun, Zhenfei Yin, Lijun Wang, Huchuan Lu

arXiv:2601.13029v3cs.CV

TL;DR

VLMs remain limited by passive, 2D-centric processing for genuine 3D spatial reasoning. Think3D adds interactive 3D manipulation and exploration, while Think3D-RL learns exploration strategies for smaller models; these methods improve benchmark performance across closed-source and open-weight settings.

  • Problem

    VLMs excel at 2D visual understanding but remain constrained on genuine 3D spatial reasoning because they use a passive, 2D-centric paradigm.

  • Method

    Think3D equips VLM agents to iteratively explore reconstructed point clouds with 3D manipulation tools, while Think3D-RL learns viewpoint and action-selection policies through reinforcement learning.

  • Results

    +7.8% average gain on BLINK Multi-view and MindCube and +4.7% on VSI-Bench were achieved for closed-source models; Qwen3-VL-4B’s benefit rose from +0.7% to +10.7% after Think3D-RL.

  • Takeaways & Limitations

    Think3D establishes tool-augmented active exploration as a promising route to genuine spatial reasoning, with RL enabling smaller VLMs to approach larger models’ exploration behavior and performance.

Abstract

from arXiv · show

While contemporary Vision-Language Models (VLMs) excel at 2D visual understanding, they remain constrained by a passive, 2D-centric paradigm that severely limits genuine 3D spatial reasoning. To bridge this gap, we introduce Think3D, a novel framework that equips VLM agents with interactive, 3D chain-of-thought reasoning capabilities. By integrating a suite of 3D manipulation tools, Think3D transforms passive perception into active spatial exploration, closely mirroring human geometric reasoning. We demonstrate that Think3D acts as a highly effective zero-shot plug-in for state-of-the-art closed-source models (e.g., GPT-4.1, Gemini 2.5 Pro), yielding absolute performance gains of +7.8% on BLINK Multi-view and MindCube, and +4.7% on VSI-Bench. Furthermore, to optimize tool-use in smaller open-weight models, we propose Think3D-RL, a reinforcement learning paradigm designed to autonomously learn spatial exploration strategies. When applied to Qwen3-VL-4B, Think3D-RL amplifies the performance gain from a marginal +0.7% to a substantial +10.7%. Notably, this RL formulation induces an exploration policy that qualitatively aligns with the sophisticated behavior of much larger models, entirely circumventing the need for costly operation-trajectory annotations. Ultimately, Think3D establishes tool-augmented active exploration as an effective paradigm for unlocking human-like 3D reasoning in multimodal agents. Code, models, and data are available at https://github.com/zhangzaibin/spagent.

1 Introduction

Think3D reframes spatial reasoning as active exploration in reconstructed 3D space rather than passive 2D perception. Its interactive toolkit and reinforcement-learning variant improve spatial reasoning across benchmarks, especially for larger closed-source models and RL-trained smaller models.

  • Current VLMs remain strong 2D analyzers but lag humans on tasks requiring genuine 3D reasoning, including multi-view understanding and route planning.
  • Think3D lets VLM agents manipulate reconstructed point clouds through camera-based actions and iterative spatial reasoning chains.
  • The agent uses camera poses as reference anchors, alternates global and local views, and iteratively refines its understanding of the reconstructed scene.
  • +7.8% average gain on BLINK Multi-view and MindCube and +4.7% on VSI-Bench were obtained for closed-source models, while Qwen3-VL-4B’s tool-use benefit rose from +0.7% to +10.7% after RL.
  • Think3D frames spatial reasoning as active 3D exploration instead of conventional passive 2D perception.
  • Think3D-RL formulates viewpoint acquisition and action selection as reinforcement learning, using final task rewards to learn exploration policies.

2 Related Work

Related work improves spatial reasoning through stronger VLMs, spatial data and cues, tool augmentation, reinforcement learning, and 3D reconstruction. Think3D builds on these directions by combining tool-use reasoning with reconstructed 3D representations.

  • VLMs for Spatial Reasoning: Recent VLM research improves spatial reasoning using 3D reconstruction, depth cues, large-scale 3D spatial VQA data, spatial prompting, mental simulation, visual chain-of-thought, and explicit grounding.
  • Tool-Augmented VLMs: Tool calling lets VLMs use external tools for high-resolution image analysis, long-video understanding, and other complex tasks.
  • Tool-Augmented VLMs: Reinforcement learning has become a central paradigm for tool-use and reasoning policies, with DeepEyes specifically inspiring the paper’s think-with-images design.
  • 3D Reconstruction: Computer-vision research has advanced 3D reconstruction from 2D images through transformer-based architectures, including methods that infer camera poses and metric-scale reconstructions.

3 Think3D for Spatial Reasoning

Think3D gives VLMs an interactive 3D reasoning loop by reconstructing scenes, manipulating viewpoints, rendering novel views, and accumulating observations. Think3D-RL trains exploration policies with trajectory-level rewards to improve multi-step spatial exploration.

  • The toolkit provides 3D reconstruction, transformation, and novel-view rendering for flexible spatial exploration.Reconstruction estimates a point cloud and camera poses from multi-view images, while transformation and rendering produce new observations.
  • At each step, the agent selects an anchor camera, global or ego-centric mode, and azimuth and elevation rotations to determine the next view.Global mode transforms the scene around its centroid; ego mode rotates a virtual camera around the anchor camera center.
  • Rendered views are appended to the interaction history, allowing the VLM to condition later actions and answers on accumulated observations.The policy receives the query, original multi-view images, and prior history, then emits a response and optional tool call.
  • Think3D equips VLMs to explore and reason in 3D through iterative observe → manipulate → reflect interactions.The agent accumulates rendered observations into an explicit 3D-aware reasoning trace.
  • Think3D-RL optimizes multi-step exploration with trajectory-level rewards based on answer correctness and formatting, using GRPO with observation-token gradient masking.The reward is assigned only at the end of each trajectory and jointly reinforces preceding viewpoint decisions.

4 Experiment

The experiments evaluate Think3D on challenging spatial-reasoning benchmarks and compare closed-source, specialized, and reinforcement-learning baselines. Results show benchmark improvements and systematic exploration behavior after RL training.

  • Benchmarks: Think3D is evaluated on BLINK Multi-view, MindCube, and VSI-Bench, covering multi-view geometry, camera-motion reasoning, and video-based spatial reasoning.BLINK and MindCube are included in Table 2, while VSI-Bench is reported separately.
  • Baselines: The comparison includes leading closed-source VLMs, spatial-reasoning specialists, and Qwen3-VL-4B trained with standard GRPO.The baseline set includes GLM-4.5V, Doubao-1.5, GPT-4.1, Gemini-2.5-Pro, RoboBrain, Spatial-MLLM, VLM-3R, and REVPT.
  • Main results: 11.57% and 4.00% relative gains are achieved by GPT-4.1 and Gemini-2.5-Pro, respectively, on the multi-view reasoning benchmark in a zero-shot setting.These gains are reported for Think3D applied to proprietary models without training.
  • Main results: 0.8% to 12.05% is the increase in Think3D gain for the RL-finetuned model on VSI-Bench.The reported comparison contrasts the RL-finetuned model’s gain before and after reinforcement learning.
  • Experimental settings: Think3D uses up to three exploration iterations on BLINK Multi-view and the MindCube subset, while Qwen3-VL-4BT3RL is trained with Think3D-RL.The table caption also distinguishes Qwen3-VL-4BGRPO, trained with standard GRPO.
  • Exploration behavior: After RL training, Think3D autonomously selects viewpoints and switches between global and ego-centric views while exploring angles more systematically than the untuned baseline.Figure 3 provides a qualitative example of the learned exploration behavior.

5 Ablation Study

The ablations show that Think3D benefits from anchored, multi-view exploration, while reinforcement learning improves both viewpoint selection and the use of additional exploration turns. Its gains persist when the 3D reconstructor is replaced.

  • 5.1 Ablation of Components: Removing the anchor camera pose causes a mild performance drop, while adding anchor selection and ego-view configuration greatly improves performance.Raw 3D input alone is insufficient; active exploration of multiple viewpoints helps process point clouds more effectively.
  • 5.3 Ablation of Reinforcement Learning Dynamics: After about 50 training steps, increased spatial tool usage produces more point-cloud views and steadily improves accuracy.Earlier turn reduction raises reward but decreases accuracy by limiting access to 3D viewpoints.
  • 5.3–5.4 Reinforcement Learning Dynamics: RL fine-tuning shifts Qwen3-VL-4B toward the informative viewpoint patterns of stronger models, including more frequent top-down views for global structure.Strong models concentrate on oblique and top-down perspectives, while exploration strategies vary across task types.
  • 5.5 Ablation of Exploration Rounds: After RL training, Qwen3-VL-4B gains accuracy as exploration turns increase, unlike models without RL training.The learned policy improves the returns from additional visual evidence and supports more effective use of Think3D.
  • 5.7 Robustness of the 3D Reconstruction Tool: Replacing Pi3 with VGGT preserves clear gains over the no-tool baseline on BLINK and MindCube, retaining most of Pi3’s improvement.The result indicates that Think3D can use off-the-shelf reconstructors with different accuracy profiles.

6 Conclusion

Think3D enables VLM agents to reason through iterative interaction with reconstructed 3D point clouds rather than passive 2D perception. Its RL-enhanced variant learns efficient exploration and shows gains across spatial benchmarks.

  • Conclusion: Think3D lets VLM agents actively reason in 3D by iteratively exploring reconstructed point clouds with a 3D manipulation toolkit.The framework targets deeper and more consistent spatial understanding than passive 2D perception.
  • Conclusion: The RL-enhanced variant enables smaller VLMs to approach the behavior and performance of large proprietary models.Experiments on BLINK, MindCube, and VSI-Bench-Tiny show strong gains and cross-benchmark generalization.

A Prompts and Implementation Details

The implementation uses modular prompts to coordinate iterative 3D tool use, viewpoint selection, and reasoning across training and evaluation. RL training pre-generates point clouds and stages prompts by iteration.

  • Training-free Workflow: The training-free workflow separates system, tool, and continual prompts to stabilize multi-round tool-augmented reasoning.These prompts define roles, 3D tool usage, viewpoint exploration, and context updates across reasoning rounds.
  • RL Training Prompt: RL training pre-generates point clouds to avoid running Pi3 during every rollout step and reduce per-sample processing overhead.Stage-specific continual prompts address the weaker instruction-following and prompt-utilization capabilities of smaller open-source models.
  • 3D Tool: The 3D tool reconstructs point clouds and visualizations from custom camera angles for motion and spatial reasoning tasks.Azimuth controls left-right rotation, elevation controls up-down rotation, and rotations use the input-camera coordinate frame.
  • 3D Tool: The default (0°, 0°) viewpoint corresponds to the first input image and is excluded from tool use.The prompt explicitly identifies this viewpoint as cam1.
  • Self-Refine: The self-refine experiment repeatedly critiques and refines an answer for a specified number of iterations before returning the final answer.The workflow initializes an answer, generates a critique, and produces a refined answer at each iteration.

B Further Experiment Analysis

Further analyses show that GPT-4.1 adapts viewpoint type and tool-call frequency to task demands. Fine-grained tasks favor ego views, while route planning favors global views and fewer tool calls.

  • Ego View Usage: Fine-grained tasks such as MindCube and Object Direction rely more on ego views, whereas Route Planning favors global views for broader context.The analysis links viewpoint usage to whether a task requires local detail or global spatial structure.
  • Tool Calling Iteration: GPT-4.1 uses tools less frequently for route planning but often performs multiple tool-call rounds for other tasks.These additional calls provide richer spatial information when required by the task.

C Think3D-RL Training And Evaluation Setting

The paper reports the parameters used for Think3D-RL training and evaluation, averaging each setting over three runs for the main results and ablations.

  • Three runs were used for every training and evaluation setting, with average performance reported for the main results and ablation studies.

D Angle choose

The angle-selection evaluation compares random, fixed heuristic, and RL-based exploration strategies using the Qwen3-VL-4B backbone. Results indicate that improvements primarily arise from learning an effective spatial policy.

  • Random sampling selects two views from eight candidate angles, while Heuristic uses fixed views (−45, 30) and (45, 30).
  • RLrandom performs similarly to Random when using the RL-trained backbone, separating backbone training from exploration-policy effects.
  • The learned RL policy significantly improves performance over random and heuristic exploration, indicating gains mainly come from spatial policy learning.
  • All exploration methods run three rounds in the comparison.

E Results on More Models

Additional experiments broaden the comparison of vision-language models on BLINK Multi-view and a MindCube subset, with results reported in Table 7.

  • Additional evaluation results compare more vision-language models on the BLINK Multi-view and MindCube subset benchmarks.
  • Table 7 reports the additional BLINK Multi-view and MindCube subset results as percentages.

F Interaction Visualization

The visualizations illustrate how Think3D agents use reconstructed scenes and viewpoint changes to reason about camera motion and object relations. Examples span MindCube, BLINK, and VSI-Bench interactions.

  • MindCube: A 180-degree turn reverses the facing direction, so the original left-right relation changes when determining what is to the right.
  • MindCube: The agent uses top-down reconstruction to verify the spatial arrangement of cabinets, windows, pictures, and cameras before answering directional questions.
  • Visualization examples: The examples include MindCube and BLINK figures as well as VSI-Bench visualizations.
  • BLINK: The agent requests a more horizontal viewpoint at azimuth=90° and elevation=0° to clarify the camera-motion arc.
  • BLINK: The camera moves right, or counter-clockwise, from cam1 to cam2 around the mug, based on reconstructed camera positions and movement path.
  • BLINK and VSI-Bench: The interaction examples include questions about camera direction, object proximity, and spatial relations among room objects.
Loading 2601.13029v3…