PaperlayerView exact revision on arXiv

Source-linked AI summary

SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

Siyi Chen, Mikaela Angelina Uy, Chan Hee Song, Faisal Ladhak, Adithyavairavan Murali, Qing Qu, Stan Birchfield, Valts Blukis, Jonathan Tremblay

arXiv:2512.04069v2cs.CVcs.RO

TL;DR

VLMs need more precise spatial reasoning for embodied applications, but coordinating many tools creates a difficult exploration problem. DIRL trains SpaceTools through staged supervision and interactive multi-tool RL, achieving state-of-the-art spatial benchmarks and robotic manipulation performance.

  • Problem

    Precise spatial reasoning remains challenging for VLMs, while reinforcement learning with multiple heterogeneous tools faces a combinatorial search space.

  • Method

    DIRL combines single-tool interactive-RL demonstrations and frontier-model full-tool traces for teaching, then refines multi-tool coordination through interactive RL exploration.

  • Results

    SpaceTools achieves state-of-the-art performance across spatial reasoning benchmarks and completes robot pick-and-place tasks with an 86% success rate.

  • Takeaways & Limitations

    VLMs can acquire complex spatial reasoning through learned tool coordination without architectural modification or large-scale data-driven fine-tuning.

  • Takeaways & Limitations

    DIRL assumes efficient tool invocation, while heterogeneous tool serving can introduce latency and memory bottlenecks; mock robot tools are used during training.

Abstract

from arXiv · show

Vision Language Models (VLMs) demonstrate strong qualitative visual understanding, but struggle with metrically precise spatial reasoning required for embodied applications. The agentic paradigm promises that VLMs can use a wide variety of tools that could augment these capabilities, such as depth estimators, segmentation models, and pose estimators. Yet it remains an open challenge how to realize this vision without solely relying on handcrafted prompting strategies or enforcing fixed, predefined tool pipelines that limit VLMs' ability to discover optimal tool-use patterns. Reinforcement Learning could overcome this gap, but has so far been limited to reasoning with a single visual tool due to the large search space in multi-tool reasoning. We introduce Double Interactive Reinforcement Learning (DIRL), a two-phase training framework where VLMs learn to coordinate multiple tools through interactive exploration and feedback. In the teaching phase, we combine demonstrations from a single tool specialist trained via interactive RL with traces from a frontier model using all tools. In the exploration phase, the model further refines multi-tool coordination through continued RL. Our model, SpaceTools, with tool-augmented spatial reasoning ability, achieves state-of-the-art performance on spatial understanding benchmarks (RoboSpatial-Home, BLINK, BOP-ASK) and demonstrates reliable real-world manipulation using a 7-DOF robot as a tool. DIRL provides substantial improvements over the vanilla SFT (+12% on RoboSpatial) and RL (+16% on RoboSpatial) baselines. Project page: https://spacetools.github.io/.

1. Introduction

Spatial reasoning is important for embodied VLM applications, yet precise geometric and 3D understanding remains challenging. SpaceTools addresses this challenge through interactive multi-tool learning and achieves strong benchmark and robotic-control results.

  • Spatial reasoning supports geometric questions about relative positions, spatial configurations, and physical affordances needed by embodied systems.Robotic applications additionally require perception to translate into decision-making and physical action.
  • Tool use provides precise measurements and intermediate geometric representations while combining heterogeneous computer-vision and robotics modules.This offers an alternative to task-specific fine-tuning that requires large-scale annotations and extensive data engineering.
  • DIRL trains VLMs to coordinate multiple tools through a teaching phase followed by an exploration phase.The framework combines single-tool interactive-RL demonstrations with frontier-model traces before continued multi-tool RL.
  • Toolshed hosts compute-intensive vision tools as rapid on-demand services for interactive training.The platform includes tools such as SAM2, Depth Pro, RoboRefer, and GraspGen.
  • SpaceTools achieves state-of-the-art performance across multiple spatial reasoning benchmarks and completes robot pick-and-place tasks with an 86% success rate.The evaluations cover RoboSpatial-Home, BLINK, RefSpatial, CVBench, and BOP-ASK, with a real robot used as a tool.

2. Related Work

Prior work studies VLM spatial reasoning, tool augmentation, and reinforcement learning, but multi-tool spatial reasoning remains difficult because heterogeneous tools create a combinatorial exploration space. DIRL addresses this with progressive training phases.

  • Spatial reasoning research targets geometric relationships, but real-world robotics additionally demands multi-step reasoning, precise geometry, and strong 3D awareness.These requirements connect visual understanding to physical-world perception and interaction.
  • Tool-augmented reasoning enriches models with information from external modules, including search engines, calculators, code executors, and vision tools.The related-work landscape spans both language-model and vision-language-model applications.
  • With 10+ heterogeneous tools, the action space grows combinatorially, causing naive reinforcement-learning exploration to fail.Prior spatial-reasoning RL has largely used a single lightweight tool such as cropping.
  • DIRL decomposes multi-tool learning into progressive, tractable phases to enable effective coordination strategies.This directly targets the weak optimization signals caused by applying interactive RL to all tools simultaneously.

3. Problem Formulation

The paper formulates spatial reasoning as sequential decision-making in which a VLM interacts over multiple turns with vision and robotic tools before producing a final answer.

  • A VLM policy interacts with external tools to answer image-text queries or robotic manipulation tasks.Interaction ends with a final answer or after at most T_max steps.
  • At each turn, the policy conditions on the full dialogue history and generates either a tool-interaction response or the next conversational action.Tool outputs and the response are appended to the history for subsequent decisions.
  • The workflow uses structured tags for reasoning, tool calls, and final answers while learning multi-turn interaction with vision and robotic tools.Reasoning uses <think>, tool calls use <tool_call>, and final responses use <answer>.

4. Double Interactive Reinforcement Learning

DIRL combines teacher-guided supervision with interactive multi-tool reinforcement learning, supported by Toolshed and task-specific rewards. The design initializes tool use before refining coordination through exploration.

  • Exploration phase: The exploration phase resumes interactive RL with all tools, using the SFT-initialized policy to refine tool chaining and avoid collapse in the large action space.Interactive feedback supplies additional refinement of multi-tool coordination.
  • Toolshed: The training system assumes efficient tool invocation, while high-throughput interactive learning remains constrained by tool-serving latency and memory bottlenecks.The work mitigates robot-in-the-loop costs during training with mock robot tools.
  • Toolshed: Toolshed serves heterogeneous compute-intensive tools through isolated instances, decoupled execution, and asynchronous parallel workers.These design choices allow tool resources to scale independently from training resources.
  • Rewards: Rewards are normalized and task-specific, measuring final-answer correctness or geometric consistency for multiple-choice, boxes, pointing, pose, and grasp tasks.Examples include binary accuracy, MIoU, NNDC, projected-corner IoU, and NNCE.

5. Experiments

SpaceTools is evaluated on diverse spatial reasoning benchmarks and real-robot manipulation, using interactive multi-tool trajectories and alternating perception, reasoning, and action. It achieves strong benchmark performance, while ablations indicate that both teacher initialization and subsequent interactive RL materially contribute.

  • 5. Experiments: The teaching dataset contains 8k multi-turn tool-use trajectories, combining 6k universal-teacher traces with 2k demonstrations from a pointing-tool IRL teacher.Stage-2 interactive RL reuses the same spatial reasoning image-question pairs after SFT initialization.
  • 5.3. Experiments on Real Robot Manipulation: SpaceTools completes a multi-step flashlight pick-and-place task by alternating language reasoning, vision tools for perception, and robot tools for action.The robot arm is treated as an action tool within a closed perception–action loop.
  • 5.1. Spatial Reasoning Evaluation: The evaluation spans spatial VQA, pointing, 2D and 3D relations, relative depth, placement, pose estimation, and grasp estimation across multiple benchmarks.Grasp evaluation uses normalized MACE and success rate, with success defined by MACE > 40.
  • 5.2. Spatial Reasoning Results: +7.5% on RoboSpatial, +24.4% on pose estimation, and +8.3% on grasp prediction versus proprietary baselines, while SpaceTools-3B gains +12% and +16% on RoboSpatial over tool-free SFT and RL.These results cover comparisons with Gemini-ER 1.5, Claude Sonnet 4.5, and GPT-5, respectively.
  • 5.2. Spatial Reasoning Results: SpaceTools dynamically selects, orders, composes, and recovers from vision tools, using pointing for relations, depth for relative depth, and multiple tools for pose or grasp.It can fall back to self-estimation or switch pointing tools when a tool fails or detections are uncertain.
  • 5.4. Ablation Study: Removing either teacher degrades performance, while removing Stage-2 IRL affects RoboSpatial, RefSpatial, and pose tasks; Tool SFT and Tool NIRL trail DIRL by +13.4 and +14.4 mean improvement.The IRL-trained teacher is especially important for fine spatial grounding, and the universal teacher for multi-tool pose composition.

6. Discussion & Conclusion

The discussion examines tool augmentation and generalization, finding clear gains for precise spatial grounding but mixed effects on higher-level tasks. The paper concludes that DIRL and Toolshed support state-of-the-art spatial reasoning and robot interaction through learned tool coordination.

  • 6. Discussion & Conclusion: GPT-5 with Toolshed improves RefSpatial from 23.1 to 36.1 and pose estimation from 9.0 to 15.0, while RoboSpatial and BLINK show mixed trends from tool overuse and difficulty interpreting nuanced outputs.The comparison is reported for proprietary models evaluated with and without Toolshed.
  • 6. Discussion & Conclusion: IRL-trained single-tool models transfer beyond their in-domain benchmark, reaching 72.3% on RoboSpatial and 34.3% on RefSpatial after training only on RoboSpatial.Other fine-tuning approaches score zero on RefSpatial in this comparison.
  • 6. Discussion & Conclusion: DIRL trains tool-augmented VLMs through progressive interactive learning, while Toolshed deploys diverse tools for scalable online interaction during training.The framework supports teacher demonstrations, self-exploration, and robot use as a tool.
  • 6. Discussion & Conclusion: SpaceTools achieves state-of-the-art performance across multiple spatial reasoning benchmarks and exhibits out-of-distribution generalization, including robot-tool use.The conclusion attributes these capabilities to learned tool coordination rather than architectural modification or large-scale data-driven fine-tuning.

A. Limitations and Future Directions

The paper identifies application, methodology, and infrastructure boundaries for tool-augmented spatial reasoning, while outlining directions for broader tasks, richer tool outputs, improved robustness, and more scalable execution.

  • Application scope: Current applications focus on short- or medium-horizon tasks, such as spatial question answering and grasp-and-place manipulation, leaving longer multi-stage tasks for future work.The authors also propose richer robotic simulations, interactive games, and physics-rich virtual worlds.
  • Methodology: Toolshed primarily exposes structured text or variables, so reasoning over visual tool outputs remains an open methodological direction.Future work also includes better perception, verification, and recovery from tool errors or inaccuracies.
  • Infrastructure: Serving heterogeneous tools in parallel can create latency and memory bottlenecks, especially for high-resolution vision tools and robot-in-the-loop execution.Training mitigates robot latency by using mock robot tools.
  • Infrastructure: Toolshed provides a scalable backbone for interactive tool use, but scheduling, caching, batching, asynchronous execution, and lighter-weight tools could improve scaling.The infrastructure uses decoupled execution, asynchronous processing, and isolated resources for tool instances.

B.1. Example Toolshed Integration with Proprietary Models

Toolshed connects proprietary VLMs such as Claude to modular vision and robotic tools through interactive communication, with APIs that expose perception, geometry, and manipulation operations.

  • Integration: Toolshed enables Claude to communicate interactively with users and external tools through an integrated web-based setup.The system is designed to attach external tools to proprietary models through simple interfaces.
  • Interfaces: Pythonic modular APIs reduce integration effort by requiring tool definitions while automating execution and reinforcement-learning integration.Toolshed supports both visual and variable-based outputs for downstream model use.

C. Additional Method Details

The method uses GRPO to optimize asynchronous multi-turn tool-use rollouts, while reward design combines task accuracy with optional structural constraints and specialized pointing rewards.

  • Group Relative Policy Optimization: GRPO increases the probability of high-reward responses while using clipping and KL regularization to stabilize optimization against a stage-1 reference policy.Rollouts are generated asynchronously and converted into group-relative advantages before policy updates.
  • Alternative Reward Design: Pointing rewards are explored because pointing often initiates spatial reasoning or subsequent tool use, especially for tasks requiring explicit numerical estimation.The alternatives include normalized signed distance to the convex hull and normalized area change, with clipping against binary accuracy.
  • Alternative Reward Design: The reward formulation combines task-specific accuracy with a weighted format score that checks the required ordering and uniqueness of tool-call and answer tags.The format score was explored but not used in final training; its weight λ is often set to 0.3 in prior work.

D.1. More Training and Compute Details

Training uses prompt-based teaching data, synthetic robot examples with mock tools, answer rebalancing, and newer infrastructure choices to address data, calibration, and stability concerns.

  • Teaching data: Synthetic robot data from HOPE and mock robot tools supplies the robotic portion of teaching data, while interactive learning focuses exclusively on vision-tool spatial reasoning.This separation avoids the latency of robot-in-the-loop training and data collection during teaching-data construction.
  • Answer balancing: Rebalancing RoboSpatial VQA answers mitigates its over 75% “no” distribution and improves answer calibration across tasks.The imbalance otherwise biases predictions toward “no” during supervised and interactive reinforcement learning.
  • Training stability: A small KL coefficient encourages exploration but can cause an initial reward drop and training instability in older Verl versions.The instability is alleviated by Toolshed integration with newer Verl agent-loop functionality.

D.2. MACE Metric for Grasp Affordances

This section defines grasp affordance evaluation and describes robot experiments, infrastructure, and interactive tool-learning results. It also reports evidence that interactive tool use improves transferable spatial reasoning and supports efficient tool expansion.

  • MACE Metric: A grasp estimator predicts five normalized image points defining the grasp center, finger bases, and finger tips, from which four finger-direction vectors are derived.These representations support grasp-affordance evaluation with the MACE metric.
  • Robot Evaluation: Robot experiments use a Kinova Jaco arm, CuRobo motion planning, and a ZED2 RGB-D camera exposed through capture, grasp, and placement tools.Pick tasks are scored by success rate, while Pick & Place tasks receive one point for each correct pick and place operation.
  • Robot Results: SpaceTools outperforms Claude and GPT-5 on relational-pick and pick-and-place tasks, while all models fail the soft-toy task because the pointing tool cannot distinguish soft from rigid toys.The authors attribute the stronger spatial-relation performance likely to interactive training with the tool.
  • Toolshed Infrastructure: Toolshed improves execution latency and throughput for multi-tool pipelines and reduces latency under contention by scaling tool instances.The reported advantages are especially relevant for tools with large inputs or outputs, such as SAM and depth estimation.
  • Interactive RL Results: 72.3% overall accuracy on RoboSpatial was achieved by IRL with Tools, which also reached 34.3% on unseen RefSpatial tasks while other fine-tuning strategies scored zero.The result is reported for a constrained setup using the RoboSpatial dataset and pointing tools.
  • Scaling to New Tools: DIRL can add a new tool with at most 40% of full-training compute because the foundational pointing-tool stage, accounting for roughly 60%, is reusable.This supports partial rather than complete retraining when expanding the tool set.

E.4. Other Ablations

These ablations examine direct multi-tool RL, reward and prompt choices, and data composition. They show that diverse balanced data can matter more than simply increasing dataset size, while direct all-tool RL remains difficult.

  • Direct Multi-Tool IRL: Direct IRL over all tasks and tools creates a large search space and is difficult to learn effectively.The appendix provides qualitative evidence supporting this limitation.
  • Reward and Prompt Design: NNDC without an additional format reward produces the most stable and reliable learning behavior in the pointing-task reward and prompt ablation.The comparison uses a subset of 1k vacant-space localization questions with the Molmo pointing tool.
  • New-Tool Adaptation: The efficient new-tool adaptation recipe combines SFT with final IRL on grasp data and requires about 10% of full training compute while achieving comparable performance.Grasp prediction is used as the representative case study.
  • Dataset Composition: Increasing dataset size beyond a moderate scale yields limited gains, suggesting that data diversity and label balance contribute more to IRL effectiveness than raw quantity.The analysis varies RoboSpatial data types including configuration, compatibility, grounding, and vacant-space localization.

E.5. Additional Demonstrations

The demonstrations illustrate successful tool-augmented reasoning in grasping, relative depth, and cluttered robot manipulation, alongside failures caused by object detection and imprecise physical placement.

  • Success Cases: SpaceTools can identify target objects and complete manipulation steps in cluttered, visually complex real-world scenes.The examples include successful grasp estimation and relative-depth reasoning supported by intermediate tool outputs.
  • Failure Cases: Grasp estimation has the lowest accuracy because cluttered scenes make target detection and accurate collision-free pose prediction difficult.The authors identify complex-environment object detection as a bottleneck for grasp and pose estimation.
  • Implications: The examples motivate strengthening individual tool components and incorporating robot feedback to improve tool coordination and point selection.The proposed direction targets tighter coupling between geometric predictions and physical feasibility.
  • Failure Cases: A robot placement failure occurs when the model correctly finds vacant space but selects a point near the boundary, causing placement on the boundary.This connects 2D point selection with physical feasibility during manipulation.
  • Failure Analysis: The reported failure analysis covers 30 of 60 grasp-benchmark trials and 4 of 27 robot-manipulation trials, with intertwined traces preventing unique attribution in every case.Failures are categorized by their most direct cause.
Loading 2512.04069v2…