PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning

Haoyuan Li, Zhengdong Hu, Jun Wang, Hehe Fan, Yi Yang

arXiv:2606.07436v2cs.CV

TL;DR

Tool-using MLLM agents often apply uniform strategies to heterogeneous 3D scenes, where tasks require different evidence. Skill-3D learns reusable scene-aware skills from tool trajectories; across benchmarks it improves performance, including a VSI-Bench average increase from 42.9 to 64.5 over w/o Tools across four closed-source agents.

  • Problem

    Tool-augmented MLLM agents use relatively uniform strategies across heterogeneous scenes, while task-relevant evidence and tool workflows vary by scene.

    Source: §1 · p. 1

  • Method

    Skill-3D records scene-task tool trajectories in Scene Memory, distills successful trajectories from similar scenes into reusable skills, retains failures as lessons, and retrieves updated skills to guide subsequent tool use.

    Source: §1 · pp. 1–2

  • Results

    Across VSI-Bench, BLINK, CV-3D, and MMSI-Bench, Skill-3D consistently improves performance; on VSI-Bench, four closed-source agents average 42.9 to 64.5, a 50.3% relative gain over w/o Tools.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Results suggest gains stem from retrieving scene-aware skills that specify task-relevant evidence and tool workflows, rather than from tool availability alone.

    Source: §4.2 · p. 6

    Evaluation focuses on indoor 3D spatial reasoning; transfer to outdoor scenes, embodied navigation, or real-time robotics may require new tool interfaces, scene signatures, and safety constraints.

    Source: Limitations · p. 9

Abstract

from arXiv · show

This paper explores agentic 3D spatial understanding, i.e., MLLM agents performing 3D reasoning through tool use. Existing methods often misuse tools and exhibit biased tool preferences under 3D scenarios, leaving the agentic paradigm with only marginal gains over non-agentic strategies. We reveal that 3D spatial reasoning tasks are heterogeneous across scenes, while these agents apply a uniform tool-use strategy to all scenes rather than selecting tools according to the specific scene and task. To address this, we propose Skill-3D, a framework that learns self-evolving scene-aware skills. Specifically, Skill-3D identifies the task scene and records the agent's tool-use trajectory into a Scene Memory, where successful trajectories from similar scenes are aggregated and distilled into a reusable scene-aware skill, with failed ones attached to the skill as lessons. During training, once a similar scene recurs, the corresponding skill is injected to guide the agent, producing new trajectories whose successes and failures further refine the skill, forming a loop in which the memory and the skill library co-evolve. Experiments show that Skill-3D substantially improves tool utilization in 3D spatial reasoning (from 39% to 78% on VSI-Bench), driving the agent toward correct and sufficient tool use. For instance, it improves Gemini-3-Flash by 67% on MMSI-Bench. Furthermore, we conduct agentic post-training over skill-guided trajectories, which boosts Qwen3-VL-8B by 60% on VSI-Bench.

1 Introduction

Skill-3D addresses scene-dependent tool-use needs by building reusable scene-aware skills from stored trajectories and evolving them as agents encounter similar scenes. It also trains policies to internalize skill-guided tool use.

1 Introduction

  • Scene heterogeneity makes required evidence and tool workflows vary, but existing methods favor uniform tool strategies that can mismatch the task’s needs.For absolute object-to-object distance, depth evidence is needed, whereas detection and 3D reconstruction mainly provide relative spatial relationships.Source: §1 · p. 1
  • Skill-3D stores scene-specific tool trajectories and outcomes in Scene Memory, distills successful paths from similar scenes into reusable skills, and retains failures as lessons.Retrieved skills guide subsequent tool use, and new successes and failures refine those skills, allowing Scene Memory and the Skill Library to co-evolve.Source: §1 · pp. 1–2
  • On multiple 3D spatial reasoning benchmarks, Skill-3D consistently outperforms strong MLLM baselines across representative 3D reasoning dimensions, using scene-aware skills to guide tool-use workflows such as detection, depth, and reconstruction.Source: Figure 1 · p. 2, §1 · p. 2
  • Skills update with new trajectories to broaden coverage across variations within a scene, while Scene Memory and the Skill Library evolve without being predefined upfront.This is intended to avoid narrow coverage, such as learning kitchen depth-estimation but not living-room depth-estimation.Source: §1 · p. 2
  • Skill-guided post-training uses supervised fine-tuning followed by GRPO with a composite reward for answer correctness, tool-use quality, and structured output.The training teaches skill retrieval, tool invocation, and evidence accumulation so the policy can internalize scene-aware tool-use behavior (DeepSeek-AI et al., 2025; Shao et al., 2024b).Source: §1 · p. 2

2 Related Work

Related work spans MLLM spatial reasoning, tool-augmented agents, and reusable skills; Skill-3D focuses on skills that encode perception-grounded tool workflows for 3D spatial reasoning.

2.1 MLLMs for Spatial Reasoning

  • MLLM spatial reasoning draws on stronger backbones and benchmarks, fine-grained grounding and 3D cues, prompting and reasoning techniques, and extensions to embodied and robotic settings.Source: §2.1 · pp. 2–3

2.2 MLLM Agents

  • Tool-augmented MLLM agents use external modules and learned tool-use policies across visual domains, while 3D methods include reconstruction-based reasoning loops that may use uniform workflows across heterogeneous scenes.Source: §2.2 · p. 3

2.3 Agent Skills

  • Memory-based agents store trajectories, while skill-based methods distill reusable behavioral primitives; Skill-3D targets 3D skills encoding perception-grounded workflows over objects, geometry, and multi-view evidence.Source: §2.3 · p. 3

3 Method

Skill-3D records scene-task rollouts in a shared Scene Memory, evolves reusable skills from successful and failed cases, retrieves skills to guide inference, and uses skill-guided trajectories to post-train compact agents.

3.1 Scene-Aware Skill Extraction

  • Skill-3D turns successful rollouts into reusable workflows and uses diagnosed failures as lessons, fallback rules, or failure-aware skills while validating updates to keep its library compact.Successful workflows encode trigger conditions, required evidence, tool order, key arguments, and evidence-to-answer mapping; static skills remain fixed while dynamic skills evolve.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Skill-Guided Inference

  • Skill-3D identifies scene-task context, retrieves and selects a compact set of relevant skills, then guides iterative tool use toward required evidence and grounded answers.Retrieval ranks skills by semantic alignment, historical success, failure lessons, and estimated tool cost; selection avoids redundant workflows and can generate fallback rules.Source: §3.2 · pp. 4–5

3.3 Skill-Guided Agentic Post-Training

  • With the Skill Library frozen, Skill-3D post-trains compact agents on skill-guided trajectories using SFT followed by GRPO, rewarding answer correctness, format compliance, and efficient evidence-gathering tool use.The tool reward requires benchmark-specified evidence determined independently of the model-selected skill, preventing easier skill choices from inflating reward.Source: Figure 2 · p. 4, §3.3 · p. 5

4 Experiments

Skill-3D improves performance across closed- and open-source agents and increases effective tool use; ablations indicate that task-aligned tool selection, a frozen Skill Library, and SFT initialization matter for its use and training.

4.1 Experimental Setup

  • The evaluation uses four benchmarks, disjoint training and test samples, and both closed- and open-source agents under four tool-use settings.The training split samples 30% of questions per category; implementation uses Qwen3-VL-4B/8B as base models and GPT-5.4 as teacher, with a shared Scene Memory and Skill Library frozen during evaluation and post-training.Source: §4.1 · pp. 5–6

4.2 Main Results

  • Averaged across four closed-source agents, Skill-3D raises the VSI-Bench average from 42.9 to 64.5, a 50.3% relative gain over w/o Tools.The paper reports consistent improvements on BLINK, CV-3D, and MMSI-Bench and attributes the gains to retrieving scene-aware skills that specify task-relevant evidence and tool workflows.Source: Table 1 · p. 6, §4.2 · p. 6
  • Skill-3D-4B achieves a 59.7% relative gain and Skill-3D-8B a 60.3% relative gain over w/o Tools on VSI-Bench.The paper reports gains across spatial reasoning categories and says the stronger 8B results indicate that larger base models better exploit retrieved skills and tool evidence.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Ablation Study

  • ETU rises from 39.2% to 78.7% on VSI-Bench, 36.4% to 79.2% on BLINK, 31.8% to 87.5% on CV-3D, and 30.5% to 80.3% on MMSI-Bench.ETU measures the fraction of tool calls returning valid evidence that agents actually use; the gains indicate more effective tool use, not simply more calls.Source: Figure 3 · p. 7, §4.3 · p. 7
  • On two VSI-Bench task groups, Skill-3D yields a more task-aligned tool-use distribution than GPT-5.4 and Think3D, which show tool-selection biases.For depth-, distance-, and size-related tasks, Think3D heavily relies on Pi3 while GPT-5.4 mostly calls GroundingDINO; similar biases appear in spatial-relation and direction tasks.Source: Figure 4 · p. 7, §4.3 · p. 7
  • Offline GRPO with an SFT cold start achieves the most stable and highest reward trajectory among the tested skill-updating and initialization variants.Online library updating introduces non-stationarity, while removing the SFT warm-up leads to early degradation and slower convergence.Source: Figure 5 · p. 8, §4.3 · p. 8

5 Conclusion

Skill-3D evolves reusable, scene-aware skills from tool-use trajectories to improve agentic 3D spatial reasoning. Skill-guided post-training transfers this behavior to compact agents.

5 Conclusion

  • Skill-3D builds Scene Memory and evolves a Skill Library by distilling successful tool-use trajectories into reusable skills and retaining failures as lessons.This addresses uniform tool-use strategies across heterogeneous 3D scenes, which can cause biased tool preferences and insufficient evidence acquisition.Source: §5 · p. 8
  • At inference, retrieved skills guide tool planning, evidence collection, and answer grounding.Source: §5 · p. 8
  • Skill-guided agentic post-training transfers this behavior into compact agents, while experiments show improved accuracy and effective tool usage.Source: §5 · p. 8

Limitations · A LLM Usage Claim · B More Experimental Results

The evaluation focuses on indoor 3D spatial reasoning, and extending the framework may require new tools, scene signatures, and safety constraints; LLMs were used only to polish the manuscript, while authors developed and verified the research.

Limitations

  • The evaluation focuses on indoor 3D spatial reasoning; outdoor scenes, embodied navigation, and real-time robotics may require new tool interfaces, scene signatures, and safety constraints.These are stated as potential requirements for transferring the framework beyond the current evaluation.Source: Limitations · p. 9

A LLM Usage Claim

  • LLMs were used only for language polishing, grammar correction, and manuscript clarity; the authors developed, verified, and approved the method, experiments, analysis, and claims.Source: A LLM Usage Claim · p. 14

B.1 Efficiency Analysis.

On VSI-Bench, Skill-3D improves tool-use quality while reducing inference time relative to direct tool use and Think3D. Scene-aware skills select task-relevant evidence sources and avoid unnecessary expensive 3D reconstruction.

B.1 Efficiency Analysis

  • 70.0 average score, 78.7% effective tool usage, and 20.8s average inference time give Skill-3D the best reported VSI-Bench accuracy–efficiency trade-off.Direct tool use scores 58.2 with 39.2% effective tool usage, while Think3D scores 64.7 and takes 35.1s per query; Skill-3D adds 0.5s retrieval overhead.Source: §B.1 · p. 14, Table B.1 · p. 14
  • Pi3 takes about 21.35s on seven sampled frames, compared with 0.77s for segmentation, 1.51s for depth estimation, and 0.88s for orientation estimation.These differing tool costs help explain why selecting task-relevant evidence sources can improve the accuracy–efficiency trade-off.Source: §B.1 · p. 14
  • Skill-3D uses scene-aware skills to select task-relevant evidence and avoid unnecessary reliance on expensive 3D reconstruction.This is the paper's stated explanation for its improved accuracy–efficiency trade-off over direct tool use and Think3D.Source: §B.1 · p. 14

B.2 Cross-Benchmark Skill Transfer. · C Experimental Details

Dynamic skills transfer across related spatial-reasoning benchmarks, and pooling training benchmarks achieves the best results by combining complementary scene-task knowledge.

B.2 Cross-Benchmark Skill Transfer.

  • Skills learned from VSI-Bench transfer to MMSI-Bench and CV-3D, while MMSI-Bench skills also improve VSI-Bench.The cross-benchmark evaluation uses the same static skills and GPT-5.4 across settings.Source: §B.2 · p. 14, Table B.2 · p. 14
  • Pooling all training benchmarks consistently achieves the best results, indicating that the Skill Library benefits from complementary scene-task knowledge across datasets.Source: §B.2 · p. 14

C.1 Dataset Details · C.2 Hyperparameters · C.3 More Details about GRPO Objective

The appendix describes the four evaluation benchmarks and their split protocol, reports post-training hyperparameter settings, and details the GRPO trajectory, reward, and optimization procedure.

C.1 Dataset Details

  • Because the benchmarks lack official training splits for skill construction or post-training, the authors use category-wise random splits with question-level disjointness.Table C.3 presents dataset statistics and the train/test split; reported results use the held-out test set.Source: §C.1 · p. 14, Table C.3 · p. 14
  • VSI-Bench evaluates indoor spatial intelligence from egocentric observations across counting, distance, size, direction, route planning, and appearance-order reasoning (Yang et al., 2025a).Source: §C.1 · p. 14
  • BLINK’s multi-view spatial reasoning subset tests inference across visual observations (Fu et al., 2024), while CV-3D covers geometric reasoning including depth ordering and multi-view consistency.Source: §C.1 · p. 14
  • MMSI-Bench evaluates multimodal spatial intelligence, focusing mainly on positional relationships across diverse scene configurations (Yang et al., 2025c).Source: §C.1 · p. 14

C.2 Hyperparameters

C.3 More Details about GRPO Objective

  • For each query, Skill-3D samples G complete trajectories and scores answer correctness, format compliance, and tool-use efficiency.Tool-use efficiency considers tool calls relative to the maximum budget and successful non-empty outputs; it is zero when required tool evidence is absent.Source: §C.3 · p. 15
  • Group-normalized relative advantages guide a clipped-surrogate policy update, while a KL term to the SFT-initialized reference preserves learned skill-selection and tool-use behavior.The group-relative advantage favors trajectories with higher task success, better tool-use efficiency, and valid outputs.Source: §C.3 · p. 15

D Qualitative Results

In two qualitative cases, Skill-3D retrieves task-specific skills that correct Think3D’s distance overestimate and duplicate chair counting.

D Qualitative Results

  • 1.5m: Think3D overestimates the table–bathtub distance, while Skill-3D’s depth-distance skill combines Pi3 alignment, detection localization, and dense depth cues to predict 0.9m.Skill-3D grounds the closest object boundaries across partial views.Source: D Qualitative Results · p. 15
  • Five chairs: Think3D’s cross-view duplicates produce an incorrect count of five, while Skill-3D’s detection-counting skill uses layout consistency and object detections to correctly count four.Skill-3D grounds chair instances across views and suppresses duplicates.Source: D Qualitative Results · p. 15

E Prompt Design

The section presents prompt designs for Skill-3D’s framework stages and illustrates how retrieved skills guide tool use in distance reasoning and multi-view counting.

E Prompt Design

  • The prompt designs cover system and scene-context prompts, skill retrieval, tool planning and exclusion, and final answering.The prompt designs for framework stages are provided in Figures E.3–E.6.Source: E Prompt Design · p. 15, Figure E.3 · p. 18, Figure E.4 · p. 19
  • For the table-to-bathtub distance question, Skill-3D combines Pi3 reconstruction, object detection, and depth estimation to answer 0.9m, while Think3D overestimates it as 1.5m.The skill aligns room-level geometry with local depth cues for boundary-aware distance reasoning.Source: E Prompt Design · p. 16
  • For multi-view chair counting, Skill-3D combines Pi3 layout consistency with object detection to de-duplicate instances and count four chairs, while Think3D over-counts five.The retrieved detection-counting skill supports instance-level grounding across views.Source: E Prompt Design · p. 17

Related papers

Loading 2606.07436v2…