PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

Yang Zhou, Zixuan Huang, Sunzhu Li, Zhuo Yang, Chen Zhang, Shunian Chen, Caijun Yan, Jianyao Xu, Shunyu Liu, Weijie Fu, Peiliang Li, Xiaozhi Chen, Yuxiang Cai

arXiv:2607.27703v2cs.AI

TL;DR

VLMs can reason at the task level but often lack the fine-grained spatial perception that specialist models provide, while specialists do not decide how to compose those capabilities. SpatialCLI connects specialists as tools, learns tool use, and internalizes successful tool-assisted reasoning; it reports strong tool-enabled and tool-free performance, but remains limited by tool coverage and reliability.

  • Problem

    General VLMs lack fine-grained spatial perception, while specialist models cannot readily translate their evidence into task-level decisions.

  • Method

    SpatialCLI uses Call, Learn, and Internalize stages to augment VLM perception, train tool-use policies, and internalize specialist capabilities from successful trajectories.

  • Results

    SpatialCLI improves both tool-enabled and tool-free reasoning across embodied and spatial benchmarks.

  • Takeaways & Limitations

    External specialist capabilities can augment inference and be transferred into the VLM without sacrificing tool use.

  • Takeaways & Limitations

    SpatialCLI remains limited by specialist-tool coverage and reliability, structured perceptual outputs, and perception-centric tasks.

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning. However, a fundamental capability mismatch remains: general VLMs can reason about the overall task but often miss the visual details that determine success, while specialist vision models can capture those details but cannot translate them into task-level decisions. In this work, we propose SpatialCLI, a framework that teaches VLMs to reason with spatial tools and progressively internalize the specialist perceptual capabilities they provide. SpatialCLI proceeds in three stages: (1) Call exposes specialist vision models as spatial tools to augment the VLM's perception; (2) Learn uses Cold-Start SFT and agentic RL to improve tool use; and (3) Internalize verbalizes successful tool-use trajectories to internalize specialist perceptual capabilities. We further introduce SpatialCLI-Bench, a 516-example benchmark for compositional perception across localization, segmentation, depth, and pose. On MindCube, SpatialCLI raises Qwen3-VL-8B-Instruct from 29.3% to 84.6% with tools, surpassing GPT-5.6 Sol with tools (72.1%), while retaining 73.8% without tools after internalization.

1 Introduction

SpatialCLI addresses the mismatch between VLMs’ task-level reasoning and specialist models’ fine-grained perception by enabling tool use and capability internalization. It introduces a three-stage framework and a compositional benchmark for evaluating these abilities.

  • Motivation: Composing localization, segmentation, depth, and pose is increasingly necessary for open-ended embodied tasks.For example, finding the farthest teddy bear requires segmentation, depth comparison, and localization.
  • Capability mismatch: General VLMs handle instructions and multi-step reasoning but remain unreliable for precise localization, boundaries, metric depth, and pose.
  • SpatialCLI: SpatialCLI exposes specialist models as spatial tools, learns tool-use policies, and internalizes their perceptual capabilities through successful trajectories.Its stages are Call, Learn, and Internalize, using fine-grained evidence, Cold-Start SFT, agentic RL, and trajectory-derived supervision.
  • Benchmark: 516 examples form SpatialCLI-Bench, which evaluates compositional reasoning across localization, segmentation, depth, and pose.The benchmark is designed to address evaluations that predominantly test spatial capabilities in isolation.
  • Results: 91.3% with tools and 72.7% without tools are achieved by SpatialCLI-8B on SpatialCLI-Bench.The reported results indicate gains in both tool-enabled and tool-free reasoning.

2 Related Work

Prior work develops spatial capabilities, tool-augmented agents, and embodied planning, but existing evaluations and systems leave compositional coordination insufficiently assessed. SpatialCLI is positioned as a framework for calling, learning, and internalizing specialist perceptual abilities.

  • Learning Spatial Capabilities in Vision-Language Models: Embodied VLMs and VLAs acquire grounded visual-language knowledge, yet multimodal models still struggle with spatial relations, viewpoint transformation, and 3D understanding.
  • Learning Spatial Capabilities in Vision-Language Models: Prior spatial-capability methods train with spatial question answering, depth cues, 3D annotations, or task-specific supervision.
  • Positioning: SpatialCLI aims to turn external specialist perceptual capabilities into abilities that VLMs can call, learn, and internalize.
  • Tool-Augmented Agents and Spatial Reasoning: Tool-augmented agents can plan and invoke tools, while hierarchical robotics agents use VLMs as planners and VLAs as executors.
  • Evaluation gap: Existing benchmarks predominantly evaluate spatial abilities in isolation, motivating compositional evaluation across multiple complementary capabilities.

3 Method

SpatialCLI combines specialist spatial tools with VLM planning, staged tool-use training, and trajectory-guided internalization. Its method supports both external tool interaction and direct reasoning from internalized perceptual supervision, alongside a benchmark for compositional evaluation.

  • Inference-Time Tool Augmentation: SpatialCLI equips a VLM with specialist tools while retaining VLM task understanding, tool selection, evidence composition, and reasoning.The four tools are Locate, Segment, Depth, and Pose.
  • Inference-Time Tool Augmentation: Each interaction step lets the VLM reason, either answer or call a tool, then continue with the returned result and updated history.The framework also exposes the remaining tool-call budget and retains reasoning across turns.
  • Agentic Fine-Tuning: Cold-Start SFT establishes valid tool interactions before agentic RL improves planning, argument generation, result use, and termination through task feedback.This staged approach addresses the hybrid action space and common failures of direct multi-tool RL from the initial model.
  • Trajectory-Guided Capability Internalization: Successful RL trajectories are progressively verbalized into evidence-grounded reasoning chains that preserve cross-turn dependencies without unsupported perceptual content.Turn-wise consolidation precedes global verbalization, which removes redundancy and organizes evidence-to-answer dependencies.
  • Trajectory-Guided Capability Internalization: Dual-View Capability Internalization trains direct-answer and tool-use views to internalize specialist perception while preserving tool-use behavior.
  • SpatialCLI-Bench Construction: SpatialCLI-Bench contains 516 English six-choice examples covering compositional reasoning across localization, segmentation, depth, and pose.

4 Experiments

Experiments evaluate SpatialCLI across multiple models, benchmarks, training dynamics, scaling behavior, and ablations. Results show gains with runtime tools and after internalization, while structured returns and dual-view supervision are especially effective.

  • Experimental setup: SpatialCLI is evaluated on SpatialCLI-Bench, MindCube, MMSI, DA-2K, and BOPASK with and without tools across three base VLMs.Baselines include general-purpose models and agentic spatial methods.
  • Overall performance: 21.0 points versus 6.1 points are the average benchmark gains from SpatialCLI Tools for Qwen3-VL-8B-Instruct and GPT-5.6 Sol, respectively.Gains are generally larger for less capable base models, and SpatialCLI Tools outperform competing frameworks in the controlled Qwen3-VL-8B-Instruct comparison.
  • Overall performance: 72.7 without tools and 91.3 with tools show that training improves native spatial reasoning while preserving practical tool use.For SpatialCLI-8B, the corresponding initial-model score without tools is 35.3.
  • Analysis: 3.36 to 6.74 executed calls per trajectory occur without SFT, whereas the SFT-initialized strategy remains near 2.56 and maintains more stable training.RL without tools reaches 52.7 direct-answer performance, below the 72.7 achieved through capability internalization.
  • Analysis: 40.1% to 74.0% is the rise in the without-tools score as internalization data increases, while four-capability macro-average CII rises from 45.6 to 61.6.The aligned trends indicate continued transfer of specialist perceptual capabilities.
  • Ablation studies: 20.1 points is the SpatialCLI-Bench gain from adding structured coordinates and polygons over visual-only returns.Structured-only returns perform comparably to structured-plus-visual returns, while extra rendered images provide no consistent score benefit.
  • Ablation studies: 72.7 without tools and 91.3 with tools are reached by Full Dual-View, which outperforms alternative internalization supervision variants with shorter outputs.The ablation supports progressive evidence consolidation and complementary internalization and tool-use views.

5 Conclusion

SpatialCLI connects specialist spatial tools with VLM reasoning through Call–Learn–Internalize, improving both tool-enabled and tool-free reasoning. Its scope remains limited by specialist-tool coverage and reliability.

  • Conclusion: SpatialCLI connects specialist vision models as runtime tools, learns agentic tool use, and converts successful trajectories into dual-view supervision for capability internalization.The framework registers spatial tools, trains tool use, and jointly optimizes internalized direct reasoning with retained agentic behavior.
  • Conclusion: SpatialCLI consistently improves both w/ Tools and w/o Tools reasoning, transferring external specialist capabilities into the VLM without sacrificing tool use.
  • Conclusion: SpatialCLI remains restricted to structured perceptual outputs and perception-centric tasks because specialist-tool coverage and reliability remain limiting factors.Future work targets multimodal outputs and joint perception-action tools under VLM planning.

B.1 Detailed Training Settings

Training uses three initial VLMs and a shared 37,000-task pool, with Cold-Start SFT, agentic RL, and capability internalization configured across the SpatialCLI pipeline.

  • Initial Models: Qwen3.6-27B, Qwen3.6-35B-A3B, and Qwen3-VL-8B-Instruct are each trained through Cold-Start SFT, agentic RL, and trajectory-guided capability internalization.
  • Training Data: 37,000 training tasks are shared across stages, combining embodied QA, grounding, multi-view reasoning, trajectory planning, object rearrangement, and diverse spatial supervision.The pool contains 5,000 Vlaser, 10,000 MindCube-Train, 10,000 BOPASK-Trajectory, 10,000 BOPASK-Object-Rearrangement, and 2,000 RefSpatial tasks.
  • Agentic RL: Agentic RL samples groups of complete tool interactions and assigns each a final-answer outcome reward before GRPO optimization.Multiple-choice tasks use deterministic exact-match rewards, while BOPASK tasks use continuous point-set scores.
  • Training Configurations: Cold-Start SFT and capability internalization use full-parameter fine-tuning with λ = 0.5 for the Tool-Use View loss.Training uses two epochs, batch size 64, context length 32768, cosine scheduling, and random seed 42.

B.2 Detailed Evaluation Settings

Evaluation covers compositional spatial reasoning, embodied benchmarks, and multiple inference configurations, with standardized tool access, generation settings, and repeated runs.

  • Evaluation Benchmarks: Evaluation spans SpatialCLI-Bench, MindCube, MMSI subsets, DA-2K, and BOPASK subsets, including Motion-Cam, Pos-Cam-Cam, Trajectory, and Object-Rearrangement.
  • Evaluation Configurations: Under w/ Tools inference, all models receive the same four spatial tools; under w/o Tools inference, no tools or tool prompt are provided.Malformed or missing final answers are treated as incorrect, and samples without valid predicted points receive zero.
  • Evaluation Configurations: Performance evaluations allow at most ten tool calls per sample and report means over three runs without fixed evaluation seeds.Unless benchmark-specific settings differ, decoding uses temperature 1.0, top-p 0.95, top-k 20, and maximum length 40,960 tokens.
  • Infrastructure: Experiments run on four-node training allocations using nodes with 16 accelerators, approximately 96 GB device memory each, and 1.8 TiB system memory.

C.2 Data Provenance and Composition

SpatialCLI-Bench combines 516 human-validated visual questions from eight sources, with balanced coverage of grounding, depth, and pose compositions across one- and two-image examples.

  • Data Provenance: SpatialCLI-Bench contains 516 examples drawn from eight visual sources, with 243 single-image and 273 two-image examples.The benchmark retains 516 of 720 generated candidates after independent human agreement, for a 71.67% retention rate.
  • Capability Composition: Grounding appears in 414 examples (80.23%), depth in 403 (78.10%), and pose in 402 (77.91%), providing nearly balanced marginal coverage.All two-image examples include pose through cross-view camera-motion or pose evidence.
  • Oracle Plans: The benchmark’s 516 private oracle plans contain 1,964 tool steps, averaging 3.81 steps per example.Query_segment, query_depth, query_pose, and query_locate occur 862, 471, 404, and 227 times, respectively.
  • Construction Pipeline: Construction uses frontier-model entity inventories, specialist annotations for localization, segmentation, depth, and pose, and evidence-grounded question generation with distractor validation.

D Spatial Tool Interfaces and Implementation

SpatialCLI exposes four specialist-backed spatial tools through normalized structured interfaces, allowing the agent to obtain localization, segmentation, depth, and pose evidence. The interfaces constrain queries and serialize outputs for downstream spatial reasoning.

  • Shared representation: All spatial coordinates use integers in the normalized [0, 999] image space with the origin at the upper-left corner.Points, boxes, and polygons follow the paper’s JSON grounding convention.
  • Tool interfaces: SpatialCLI provides query_locate, query_segment, query_depth, and query_pose interfaces backed by specialist models for localization, segmentation, depth, and pose.The service layer hides the underlying specialist-model identities from the agent.
  • Tool interfaces: The agent receives structured tool outputs through model-level interfaces and does not choose the underlying specialist vision models.This abstraction separates tool selection from specialist-backend selection.
  • Locate: Locate returns boxes and centers for every matching visible instance, supporting explicit positions, counts, and comparisons after locating the base category.Relations such as closest or farthest should be resolved after retrieving all instances, potentially using query_depth.
  • Segment: Segment returns retained instances as boxes, points, and polygonal connected components after confidence filtering, mask suppression, and contour processing.Polygons use at most 16 vertices when feasible and target rasterized mask IoU of 0.97.
  • Depth and Pose: Depth queries selected image points and returns estimated camera-axis distance in meters, while Pose returns object orientation or cross-view camera motion.Depth values are monocular Z-depth estimates, and Pose supports both object and camera-motion modes.

E.1 Evaluation Data and Capability Coverage

The CII validation suite contains 1,000 held-out examples covering five balanced query types across the four spatial-tool capability areas.

  • Capability coverage: The CII validation suite contains 1,000 held-out examples, with 200 examples each for Locate, Segment, Depth, object orientation, and camera motion.Object orientation and camera motion are the two modes of the Pose interface.

E.2 Reference Construction and Evaluation Protocol

Evaluation tests whether models can reproduce specialist spatial-tool outputs from images without executing external tools. References are independently human-verified, and capability-specific scores are aggregated into a four-capability macro-average.

  • Protocol: At evaluation time, the model receives only the image and an instruction to produce the requested structured tool format; no spatial tool is registered or executed.The protocol therefore evaluates reproduction of spatial-tool outputs without external tool execution.
  • Reference construction: Two experts retain an example only when both judge its request unambiguous and its specialist-tool reference valid and visually consistent.Candidates failing either approval are discarded before evaluation.
  • Locate and Segment: Locate and Segment compare predicted and reference instance sets using IoU-maximizing one-to-one Hungarian matching, with unmatched items contributing zero.Empty predicted and reference sets receive similarity one.
  • Depth: Depth matches predictions to reference points and converts absolute relative error into bounded similarity, without median or scale alignment.Missing, duplicated, or unrequested points receive zero similarity.
  • Pose: Pose scores object orientation and camera motion separately using discrete direction representations, then macro-averages the two modes.The object-orientation score evaluates the canonical facing_direction_camera field.
  • Aggregation: Capability similarities lie in [0, 1], while capability-level CII and CIImacro lie in [0, 100] with higher values indicating stronger internalization.Exact matches receive similarity one.

F.1 Comparison with Direct Fine-Tuning

Compared with direct fine-tuning, SpatialCLI separates inference-time perceptual support, tool-use learning, and capability internalization. Tool-use training raises performance with tools, while internalization transfers gains to tool-free evaluation.

  • Inference-time augmentation: Inference-time tool augmentation improves the initial model’s performance with tools, compensating for its fine-grained perceptual limitations.The same initial checkpoint is evaluated with and without enabling spatial tools.
  • Agentic fine-tuning: 86.4 with tools follows SFT tool-use training, and 91.3 follows RL with tools plus SFT on SpatialCLI-Bench.RL with tools without SFT eventually reaches 90.5 but uses substantially longer and more tool-intensive trajectories.
  • Capability internalization: 72.7 without tools follows capability internalization from 40.1, while preserving the 91.3 score with tools.SpatialCLI-8B is obtained by dual-view capability internalization after RL with tools plus SFT.
  • Comparison: Direct variants trained without explicit tool-interaction supervision improve tool-free scores but do not acquire a comparably strong tool-use policy.This contrast isolates complementary roles for the framework’s training stages.

F.2 Tool-Set Ablation

The ablations show that spatial-tool choice is benchmark-dependent, while jointly exposing all four tools improves average performance. Internalization must also balance tool-free capability gains against retention and stability of tool use.

  • Tool-set effects: 35.7 to 56.7 macro-average: exposing Locate, Segment, Depth, and Pose jointly outperforms single-tool variants overall.No single tool is uniformly beneficial across benchmarks.
  • Tool-set effects: 91.6 on DA-2K: Depth alone matches All Tools, while Pose leads several other benchmarks and Locate or Segment specialize on distinct BOPASK tasks.Pose performs best on SpatialCLI-Bench, MindCube, and both MMSI subsets; Locate and Segment are strongest on BOPASK-Trajectory and BOPASK-Object-Rearrangement, respectively.
  • Tool-use behavior: 99.3% of initial-model samples invoke tools, averaging 2.00 calls, whereas Internalization View Only averages 20.70 calls overall, indicating unstable repetitive use.The unconstrained diagnostic uses a 40,960-token maximum and omits the ten-call cap.
  • Loss-weight trade-off: 89.2 versus 91.3 w/ Tools performance: lowering λ from 0.5 to 0.2 weakens tool-policy retention, while increasing λ to 1.5 substantially lowers CII and w/o Tools performance.The loss is L_internal + λL_agentic, and λ = 1.0 maintains w/ Tools performance at 91.4 with slightly lower internalization metrics.

H.1 Case 1: Two-Image Camera Rotation and Depth

The first case contrasts an initial visual guess with tool-grounded spatial reasoning. Depth reverses the guessed closer object, while Pose supplies the camera rotation needed for the final option.

  • Final decision: The direct answer selects C from apparent size and image position, but measured depths overturn that initial visual guess.The tool-augmented agent uses a ground–measure–cross-check–verify sequence.
  • Depth verification: 0.928 m versus 0.776 m: Depth identifies the blue receptacle as closer than the beige receptacle.The blue receptacle is on the right in the first image.
  • Camera motion: Rightward view rotation: Pose indicates that the camera rotated right from image 1 to image 2.The blue bin shifts from the right side toward the center or left of the view.
  • Final decision: Option D combines the verified facts: the blue receptacle is closer, located on the right, and associated with rightward camera rotation.The tool-augmented trajectory grounds objects, measures depth, and cross-checks camera motion before selecting the answer.

H.2 Case 2: Two-Image Camera Translation and Instance Segmentation

The second case shows how tool composition resolves both instance-level ambiguity and camera translation. Segmentation identifies the relevant sandal, while Pose corrects the direct answer’s motion interpretation.

  • Camera translation: The direct answer approximately identifies the target sandal and coordinate but selects C after interpreting scale change as backward camera motion.The tool-augmented agent obtains coarse location and camera motion, then refines the instance-level evidence.
  • Instance segmentation: 856 versus 747: the sandal extending farther right has maximum y-coordinate 856, based on instance-level segmentation.Locate merges the pair into one box, so Segment recovers two instances for coordinate comparison.
  • Camera translation: Forward-left: Pose returns the camera translation direction, replacing the direct answer’s backward-left interpretation.The trajectory combines coarse grounding, granularity checking, instance refinement, and joint decision-making.
  • Instance segmentation: The target is the front/right sandal because its bounding box reaches farther right than the back/left sandal’s box.The reported boxes are [143, 508, 765, 856] and [159, 492, 715, 747].

H.3 Case 3: SpatialCLI-8B after Capability Internalization

This case contrasts an incorrect tool-free Qwen3-VL-8B-Instruct trace with SpatialCLI-8B's tool-assisted and internalized reasoning on the same spatial task. Tool queries provide precise localization and depth evidence, while the tool-free model reaches the same final conclusion after internalization.

  • Case comparison: The Qwen3-VL-8B-Instruct baseline incorrectly identifies the yellow ball as closer and selects F after reasoning from cross-image scale changes.It correctly identifies the ball as farther right and camera translation as forward-left, but confuses the first-image depth relation.
  • Capability internalization: After capability internalization, SpatialCLI-8B estimates depths of 0.362 m and 0.615 m without tools and reaches the same forward-left conclusion.The paper presents this agreement between direct answering and runtime tool verification as a qualitative result.
  • Tool-assisted reasoning: SpatialCLI-8B with tools localizes the bottle and ball at horizontal centers 329 and 644, respectively, establishing that the ball is farther right.The tool responses report bottle point [329, 417] and ball point [644, 348].
  • Tool-assisted reasoning: Depth queries report 0.351 m for the bottle and 0.637 m for the ball, indicating that the bottle is closer to the camera.The case explicitly uses the smaller depth value as the closer-distance criterion.
  • Final decision: The tool-assisted trace verifies the yellow ball's rightward position, the bottle's closer depth, and forward-left camera translation, supporting option E.Each clause is checked using horizontal centers, depth values, and the pose result.
Loading 2607.27703v2…