PaperlayerView exact revision on arXiv

Source-linked AI summary

Think3D: Thinking with Space for Spatial Reasoning

Zaibin Zhang, Yuhan Wu, Lianjie Jia, Yifan Wang, Zhongbo Zhang, Yijiang Li, Binghao Ran, Fuxi Zhang, Zhuohan Sun, Zhenfei Yin, Lijun Wang, Huchuan Lu

arXiv:2601.13029v3cs.CV

TL;DR

VLMs remain limited on tasks requiring genuine 3D reasoning, and existing 2D tool use provides only shallow spatial cues. Think3D enables interactive exploration of reconstructed 3D point clouds, yielding consistent benchmark gains, including +7.8% on BLINK Multi-view and MindCube and +4.7% on VSI-Bench.

  • Problem

    VLMs lag on tasks requiring genuine 3D reasoning, while existing 2D tool-based approaches provide shallow spatial cues rather than deeper multi-view or geometric reasoning.

    Source: §1 · p. 2

  • Method

    Think3D lets VLMs iteratively manipulate reconstructed point clouds and inspect novel views, while Think3D-RL uses reinforcement learning to develop spatial exploration strategies.

    Source: §1 · p. 4, §3.1 · p. 6

  • Results

    Think3D yields an average +7.8% gain on BLINK Multi-view and MindCube and an additional +4.7% improvement on VSI-Bench, with consistent improvements across all tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Think3D suggests explicit 3D interaction is a promising route to genuine spatial reasoning in VLMs.

    Source: §6 · p. 15

Abstract

from arXiv · show

While contemporary Vision-Language Models (VLMs) excel at 2D visual understanding, they remain constrained by a passive, 2D-centric paradigm that severely limits genuine 3D spatial reasoning. To bridge this gap, we introduce Think3D, a novel framework that equips VLM agents with interactive, 3D chain-of-thought reasoning capabilities. By integrating a suite of 3D manipulation tools, Think3D transforms passive perception into active spatial exploration, closely mirroring human geometric reasoning. We demonstrate that Think3D acts as a highly effective zero-shot plug-in for state-of-the-art closed-source models (e.g., GPT-4.1, Gemini 2.5 Pro), yielding absolute performance gains of +7.8% on BLINK Multi-view and MindCube, and +4.7% on VSI-Bench. Furthermore, to optimize tool-use in smaller open-weight models, we propose Think3D-RL, a reinforcement learning paradigm designed to autonomously learn spatial exploration strategies. When applied to Qwen3-VL-4B, Think3D-RL amplifies the performance gain from a marginal +0.7% to a substantial +10.7%. Notably, this RL formulation induces an exploration policy that qualitatively aligns with the sophisticated behavior of much larger models, entirely circumventing the need for costly operation-trajectory annotations. Ultimately, Think3D establishes tool-augmented active exploration as an effective paradigm for unlocking human-like 3D reasoning in multimodal agents. Code, models, and data are available at https://github.com/zhangzaibin/spagent.

1 Introduction

Think3D frames spatial reasoning as active exploration of reconstructed 3D scenes, enabling VLMs to manipulate point clouds and iteratively reason across viewpoints. Think3D-RL learns exploration policies from task rewards, while the framework improves performance across spatial benchmarks.

1 Introduction

  • Current VLMs remain 2D analyzers whose performance drops on spatial tasks, while image-based tools provide shallow cues but not deeper multi-view or geometric reasoning.The introduction contrasts these limitations with human manipulation of consistent 3D representations.Source: §1 · p. 2
  • Think3D-RL trains models to discover effective viewpoint and action strategies using final task rewards, without supervision over how they manipulate the 3D scene.Multi-round exploration reinforces trajectories that improve downstream performance, yielding more consistent and informative viewpoint choices.Source: §1 · pp. 3–4
  • +7.8% average gains on BLINK Multi-view and MindCube and +4.7% on VSI-Bench for closed-source models; Think3D-RL raises Qwen3-VL-4B tool-use gains from +0.7% to +10.7%.Closed-source models use Think3D training-free; Qwen3-VL-4B receives learned exploration through Think3D-RL.Source: §1 · p. 3
  • Think3D enables VLM agents to explore reconstructed point clouds through camera-based reference actions and iterative spatial reasoning rather than passive 2D perception.Estimated camera poses anchor consistent rotations and directions; agents switch between global and local views to capture coarse and fine cues.Source: §1 · p. 3

2 Related Work · 3 Think3D for Spatial Reasoning

The related work spans VLM spatial reasoning, tool use, and 3D reconstruction; Think3D combines active exploration of reconstructed 3D point clouds with iterative spatial reasoning.

2.1 VLMs for Spatial Reasoning

  • VLM spatial-reasoning methods use 3D reconstruction, depth, spatial VQA, prompting, simulation, visual chain-of-thought, RL, and explicit grounding [17] [78] [25] [11] [16] [60] [61] [66].Robotics and navigation work extends VLM capabilities toward embodied interaction, 3D grounding, mapping, route planning, and decision making [51] [1] [52] [85] [80].Source: §2.1 · p. 4
  • Unlike [71]'s beam-search exploration of video-generated space, Think3D lets VLMs plan tool-based exploration on reconstructed point clouds, avoiding video-generation hallucinations.Other code-driven methods use 3D model outputs with an emphasis on task decomposition rather than direct spatial reasoning [10].Source: §2.1 · p. 4

2.2 VLM tool calling

  • Tool calling extends VLMs through prompting or code generation [63] [73] [81], while DeepEyes [84] inspires Think3D with “think with images” reasoning without external tools.Other work trains VLMs for specific toolsets, and reinforcement learning is used to optimize tool-use and reasoning policies [20] [50] [9] [15] [45] [67] [88].Source: §2.2 · pp. 4–5

2.3 3D Reconstruction

  • 3D reconstruction advances include pose-free multi-view DUSt3R, metric-scale MASt3R, and VGGT, which infers scene attributes from multiple views in one pass.CUT3R, MapAnything [23], and Pi3 support continual reconstruction, multi-task metric geometry, and permutation-equivariant visual geometry, respectively.Source: §2.3 · p. 5

3 Think3D for Spatial Reasoning

  • Think3D’s pipeline repeatedly cycles through observe, manipulate, and reflect: the VLM changes camera pose and rendering parameters, then uses each rendered image in its next reasoning step.The cycle is shown in Figure 2.Source: Figure 2 · p. 5

3.1 Framework Overview

  • Given multi-view images and a query, Think3D can autonomously reconstruct a point cloud and camera poses, then accumulate novel-view observations into an explicit 3D chain of thought.The loop allows at most K=3 iterations by default, one reconstruction call, and K rendering calls, with early termination via STOP when evidence is sufficient.Source: §3.1 · p. 6

3.2 3D Manipulation Toolkit

  • Think3D’s toolkit provides 3D reconstruction, 3D transformation, and novel-view rendering; reconstruction estimates a point cloud and camera poses from multi-view images using Pi3.Depth and confidence predictions are fused across views to produce a clean colored point cloud.Source: §3.2 · p. 6

3D Transformation

Think3D lets VLM agents transform 3D scenes or virtual cameras to render new viewpoints, then iteratively observe, manipulate, and reflect. Think3D-RL trains exploration through trajectory-level rewards and masked-token GRPO optimization.

3D Transformation

  • The agent selects a camera, rotation angles, and global or ego mode to explore the reconstructed point cloud.Global mode rotates the scene around its centroid while keeping the camera fixed; ego mode rotates a virtual camera while keeping its center fixed.Source: 3D Transformation · p. 7

Novel View Rendering

  • Global rendering uses the transformed point cloud, while ego rendering restricts points to a forward-facing wide-field-of-view cone.A lightweight point-based renderer synthesizes the view using the mode-specific point cloud and camera.Source: Novel View Rendering · p. 7

3.3 VLM-based Spatial Reasoning Agent

  • The VLM iteratively observes rendered views, manipulates the scene, and reflects, maintaining an explicit 3D-aware reasoning trace.Each tool call selects an anchor camera, view mode, and azimuth/elevation angles; the resulting view and action are appended to the history.Source: §3.3 · pp. 7–8

3.4 Think3D-RL for Multi-Step Exploration

  • Because exploration depends on choosing informative viewpoints and when to explore, Think3D-RL optimizes the exploration policy with reinforcement learning.Source: §3.4 · p. 8

Trajectory Formulation & Training-time Sampling.

  • An episode is a trajectory of VLM states and outputs ending in an answer, with the agent determining the number of exploration steps.Training precomputes each sample’s point cloud and renders selected views from it, retaining an interactive loop while making training and evaluation efficient.Source: Trajectory Formulation & Training-time… · p. 8

Trajectory-level reward.

  • Trajectory-level rewards combine answer correctness against the multiple-choice option with a small formatting bonus, reinforcing preceding viewpoint decisions.Source: Trajectory-level reward · p. 9

Optimization.

  • Think3D-RL uses GRPO [43] with a token-wise mask that stops gradients on rendered-observation tokens and optimizes model-generated action and answer tokens.The masking follows [84].Source: Optimization · p. 9

4 Experiment

Think3D is evaluated on BLINK (Multi-view), MindCube, and VSI-Bench, with reported gains for proprietary and RL-fine-tuned models. The experiments also describe benchmark sampling, comparison models, and Think3D-RL training settings.

Setting and Dataset

  • Think3D-RL uses GRPO with eight rollouts per step and trains on 977 MindCube samples without test-set overlap, fully fine-tuning the language model while freezing the vision encoder.Training runs for one epoch on eight H200 GPUs; inference deploys the Pi3 tool on an RTX 3090 GPU.Source: Setting and Dataset · p. 9

Benchmarks

  • Evaluation covers BLINK (Multi-view), MindCube, and VSI-Bench, using all BLINK multi-view data, 120 MindCube questions, and seven sampled frames per VSI-Bench-tiny video.MindCube questions span rotation, around, and among; VSI-Bench-tiny covers route planning, relative direction, appearance order, and relative distance, with the same sample sets used for all models.Source: Benchmarks · p. 9

Baseline Models

  • Comparisons include leading closed-source VLMs, spatial-reasoning-specialized models, REVPT, and Qwen3-VL-4BGRPO trained with accuracy and format rewards.Closed-source models include GLM-4.5V, Doubao-1.5, GPT-4.1, and Gemini-2.5-Pro; specialized models include RoboBrain, Spatial-MLLM, and VLM-3R.Source: Baseline Models · p. 10

4.2 Main Results

  • 11.57% and 4.00% relative gains with Think3D on multi-view for GPT-4.1 and Gemini-2.5-Pro contrast with Qwen3-VL-4B’s 0.61% gain, rising to 9.32% for Qwen3-VL-4BT3RL.The authors attribute the stronger RL-finetuned result to improved viewpoint selection and spatial exploration.Source: §4.2 · p. 10, Table 2 · p. 11
  • On VSI-Bench, Think3D improves GPT-4.1 by 2.96% and Gemini-2.5-Pro by 6.45%; Qwen3-VL-4BT3RL’s gain rises from 0.8% to 12.05%.The authors state that these results support improvement on video-based spatial reasoning.Source: §4.2 · pp. 10–11, Table 1 · p. 10

5 Ablation Study

The ablations show that guided viewpoint exploration improves 3D reasoning, while RL helps smaller models explore more effectively. Think3D also retains gains when its 3D reconstructor is replaced with VGGT.

5.1 Ablation of Components

  • Using 3D reconstruction without an anchor pose mildly hurts GPT-4.1 versus the no-tool baseline, while anchor-camera selection and ego-view configuration greatly improve performance.The findings suggest that raw 3D input alone is insufficient and that active exploration across viewpoints matters.Source: §5.1 · p. 12

5.2 Ablation of Space Exploration Strategy

  • GPT-4.1’s viewpoint strategy varies by task: route planning and appearance-order tasks favor top-down views, while MindCube and orientation estimation favor rotational views.The analysis spans multi-view reasoning, route planning, and object-orientation estimation across models with different base capabilities.Source: §5.2 · p. 12, Figure 4 · p. 13

5.3 Ablation of Reinforcement Learning Dynamics

  • After an initial turn reduction that lowers accuracy during the first 50 training steps, RL training increases spatial-tool use and accuracy improves steadily.The early turn reduction means fewer spatial-tool calls and fewer 3D viewpoints; later, the model uses tools to render point-cloud images more often.Source: §5.3 · p. 12, Figure 5 · p. 13

5.4 Ablation on What the Model Learns through RL

  • After RL, Qwen3-VL-4B’s viewpoint patterns move closer to GPT-4.1 and Gemini-2.5-Pro, including more top-down views for capturing global structure.The paper interprets this convergence as evidence that RL improves informed 3D exploration.Source: §5.4 · p. 13, Figure 4 · p. 13

5.5 Ablation of Exploration Rounds

  • Without RL, more interaction turns yield no clear performance gain; after RL, Qwen3-VL-4B accuracy rises steadily with turns.The result indicates improved returns from additional visual evidence and more effective use of Think3D.Source: §5.5 · p. 14, Figure 6 · p. 14

5.7 Robustness of the 3D Reconstruction Tool

  • Replacing Pi3 with VGGT preserves clear gains over the no-tool baseline on BLINK and MindCube and retains most of Pi3’s improvement.The comparison keeps prompting, tool-calling, and reasoning budget unchanged, indicating that Think3D can work with different off-the-shelf 3D reconstructors.Source: §5.7 · p. 14, Table 4 · p. 15

6 Conclusion

Think3D enables VLM agents to reason actively in 3D by iteratively exploring reconstructed point clouds with a manipulation toolkit. Its RL-enhanced variant learns efficient exploration for smaller VLMs, and experiments show gains and cross-benchmark generalization.

6 Conclusion

  • Think3D uses iterative exploration of reconstructed point clouds with a 3D manipulation toolkit to support deeper, more consistent spatial understanding.Source: §6 · p. 15
  • The RL-enhanced Qwen-4B-VLT3RL learns efficient exploration, enabling smaller VLMs to approach large proprietary models’ behavior and performance.Source: §6 · p. 15
  • Experiments on BLINK, MindCube, and VSI-Bench-Tiny show strong gains and cross-benchmark generalization.Source: §6 · p. 15

A Prompts and Implementation Details

Think3D uses modular prompts to guide reliable, efficient multi-round 3D exploration, while its RL setup precomputes spatial inputs and adapts instructions to training stage. The appendix also describes prompts for tool-free evaluation and self-refinement.

Training-free Workflow Prompt

  • Think3D’s training-free workflow separates system, tool, and continual prompts to improve format reliability, viewpoint efficiency, and multi-round reasoning stability.The tool prompt treats the input as the default (0°, 0°) view and recommends complementary viewpoints; the continual prompt tracks task progress and unresolved uncertainty. The design follows.Source: A Prompts and Implementation Details · p. 16

RL Training Prompt

  • Pre-generating point clouds for training scenes removes Pi3 inference from RL rollouts, reducing per-sample overhead and enabling more efficient policy optimization.Stage-aware continual prompts for initial, intermediate, and final training rounds are intended to improve prompt utilization and training stability in smaller open-source models.Source: A Prompts and Implementation Details · p. 16

Evaluation and Self-Refine Prompts

  • Without tools, evaluation uses standard chain-of-thought reasoning; the self-refine experiment iteratively critiques and refines answers, assessing correctness, completeness, reasoning, and confidence.The self-refine procedure repeats critique and refinement for a specified number of iterations before returning the final answer.Source: A Prompts and Implementation Details · pp. 18, 23

B Further Experiment Analysis

GPT-4.1 adapts its spatial exploration to task demands: it relies more on ego-view for fine-grained tasks and uses fewer tool calls for route planning than for other tasks.

Ego view analysis

Tool calling iteration analysis

C Think3D-RL Training And Evaluation Setting

Think3D-RL reports average performance across three runs for each experimental setting to ensure fair comparison.

C Think3D-RL Training And Evaluation Setting

  • Each RL training and evaluation setting, including main results and ablations, was run three times, with average performance reported for fair comparison.Tab. 5 provides the parameters used for training and evaluation.Source: C Think3D-RL Training And Evaluation… · p. 25

D Angle choose

In the angle-selection comparison, the learned RL exploration policy significantly improves performance, whereas RLrandom performs similarly to Random. This indicates that gains mainly come from spatial policy learning.

D Angle choose

  • Random samples two views from eight candidate angles, while Heuristic uses fixed views (−45, 30) and (45, 30), both with the original Qwen3-VL-4B.Source: D Angle choose · p. 25
  • RLrandom applies random exploration to the RL-trained backbone, and all methods run three rounds.Source: D Angle choose · p. 25
  • The learned RL exploration policy significantly improves performance, while RLrandom performs similarly to Random, indicating gains mainly come from spatial policy learning.The comparison is reported in Table 6, which presents exploration strategies.Source: D Angle choose · p. 25, Table 6 · p. 26

E Results on More Models

This section broadens the comparison of vision-language models on spatial reasoning benchmarks with additional evaluation results.

F Interaction Visualization

The interaction visualizations show agents using reconstructed 3D views to reason about orientation, camera movement, and object positions. In the BLINK example, the reconstructed camera positions support a rightward, counter-clockwise movement judgment.

F Interaction Visualization

  • For a MindCube turn-around question, the agent relates right after a 180° turn to left before the turn and requests a top-down view to verify object positions.The requested Pi3 view uses azimuth 0° and elevation 60° to inspect the spatial arrangement.Source: F Interaction Visualization · p. 27
  • The reconstructed BLINK scene shows the second camera slightly right of the first, supporting a rightward, counter-clockwise camera-movement judgment.The reasoning also cites the mug handle and background shifts as consistent with that direction.Source: F Interaction Visualization · p. 29
  • For the VSI-Bench pillow-distance question, the agent requests a top-down Pi3x view to compare the heater, plant, chair, and ceiling light with the pillow.The passage describes the tool call but does not provide the resulting closest-object answer.Source: F Interaction Visualization · p. 31
  • For a VSI-Bench trash-bin location question, the agent identifies the window, clock, and bin, then requests a bird’s-eye view to confirm their spatial relationships.The supplied passages do not show the answer after this view request.Source: F Interaction Visualization · p. 32

Related papers

Loading 2601.13029v3…