PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning

Haoyuan Li, Zhengdong Hu, Jun Wang, Hehe Fan, Yi Yang

arXiv:2606.07436v2cs.CV

TL;DR

Existing 3D tool-using MLLM agents often apply uniform strategies despite scene-specific evidence requirements, limiting the benefits of agency. Skill-3D learns self-evolving scene-aware skills from rollout memory and lessons, then uses them to guide tools and post-training. It reports improved tool utilization and spatial reasoning, while its evaluation remains focused on indoor 3D scenes.

  • Problem

    Existing agents often misuse tools and apply uniform strategies across heterogeneous scenes, yielding insufficient evidence and marginal gains over non-agentic baselines.

  • Method

    Skill-3D records scene-task trajectories in Scene Memory, distills successful similar-scene trajectories into reusable skills, attaches failures as lessons, and reinjects refined skills into later tool use and post-training.

  • Results

    Skill-3D improves effective tool usage from 39% to 78% and consistently improves benchmark performance, including a 67% lift for Gemini-3-Flash on MMSI-Bench.

  • Takeaways & Limitations

    Scene-aware skills specify task-relevant evidence and workflows, supporting improvements beyond tool availability or generic 3D reconstruction.

  • Takeaways & Limitations

    Evaluation focuses on indoor 3D spatial reasoning; outdoor, embodied, or real-time robotic applications may require new interfaces, scene signatures, and safety constraints.

Abstract

from arXiv · show

This paper explores agentic 3D spatial understanding, i.e., MLLM agents performing 3D reasoning through tool use. Existing methods often misuse tools and exhibit biased tool preferences under 3D scenarios, leaving the agentic paradigm with only marginal gains over non-agentic strategies. We reveal that 3D spatial reasoning tasks are heterogeneous across scenes, while these agents apply a uniform tool-use strategy to all scenes rather than selecting tools according to the specific scene and task. To address this, we propose Skill-3D, a framework that learns self-evolving scene-aware skills. Specifically, Skill-3D identifies the task scene and records the agent's tool-use trajectory into a Scene Memory, where successful trajectories from similar scenes are aggregated and distilled into a reusable scene-aware skill, with failed ones attached to the skill as lessons. During training, once a similar scene recurs, the corresponding skill is injected to guide the agent, producing new trajectories whose successes and failures further refine the skill, forming a loop in which the memory and the skill library co-evolve. Experiments show that Skill-3D substantially improves tool utilization in 3D spatial reasoning (from 39% to 78% on VSI-Bench), driving the agent toward correct and sufficient tool use. For instance, it improves Gemini-3-Flash by 67% on MMSI-Bench. Furthermore, we conduct agentic post-training over skill-guided trajectories, which boosts Qwen3-VL-8B by 60% on VSI-Bench.

1 Introduction

Skill-3D addresses the mismatch between heterogeneous 3D scenes and uniform tool-use strategies by learning reusable scene-aware skills. Its memory-driven framework and post-training improve tool utilization and benchmark performance.

  • Agentic 3D reasoning uses external tools to acquire spatial and geometric evidence that MLLMs may not infer alone.
  • Existing methods often favor dominant tools across scenes, producing insufficient evidence and only marginal gains over non-agentic baselines.
  • Scene heterogeneity means tasks require different evidence; for absolute object distance, depth is needed rather than only detection and 3D reconstruction.
  • Skill-3D identifies scene-task context, retrieves suitable skills, and invokes task-relevant perception tools such as depth estimation.
  • Successful trajectories are distilled into reusable skills, while failed trajectories become lessons that refine skills when similar scenes recur.
  • Skill-guided SFT and GRPO train agents to internalize skill retrieval, tool invocation, evidence accumulation, correctness, and structured output.
  • 78% effective tool usage, up from 39%, accompanies consistent improvements across representative 3D reasoning dimensions.

2 Related Work

Related work spans stronger MLLMs, spatial benchmarks, tool augmentation, memory-based agents, and reusable skills. Skill-3D differs by targeting perception-grounded tool workflows for heterogeneous 3D spatial reasoning.

  • MLLM spatial reasoning has advanced through stronger backbones, dedicated benchmarks, 3D reconstruction, depth cues, grounding, and spatial VQA data.
  • Other approaches use prompting, mental simulation, visual chain-of-thought, reinforcement learning, code-driven reasoning, and generative 3D imagination.
  • Tool augmentation lets multimodal models invoke external modules through prompting, structured APIs, or code generation across diverse visual tasks.
  • Memory-based agents store trajectories for reflection or replay, while skill-based agents distill reusable behavioral primitives for decision-time guidance or reinforcement learning.
  • Unlike prior skill-based agents focused on general automation, Skill-3D encodes perception-grounded workflows involving objects, geometry, and multi-view evidence.

3 Method

Skill-3D builds and retrieves scene-aware skills from rollout experience, then guides tool-use workflows and transfers the behavior through agentic post-training. Its design combines success-driven skill formation with failure lessons and efficiency-aware rewards.

  • Scene-Aware Skill Extraction: Skill-3D records cross-benchmark rollouts in Scene Memory and evolves a Skill Library from successful and failed spatial reasoning cases.
  • Scene-Aware Skill Extraction: Each rollout contains the question, observations, reasoning trace, selected skills, tool calls, outputs, and final answer for updating the library.
  • Scene-Aware Skill Extraction: Successful routines become new or merged dynamic skills only when they add useful scene, evidence, or workflow coverage; otherwise, success statistics are updated.
  • Scene-Aware Skill Extraction: Failed rollouts are diagnosed for errors such as wrong tool selection, missing evidence, invalid inputs, ignored outputs, and redundant calls, then attached as lessons.
  • Scene-Task Skill Retrieval: For each query, Skill-3D identifies task category, target entities, scene signature, and required evidence before retrieving and compacting relevant skills.
  • Skill-Guided Agentic Post-Training: Skill-guided post-training uses trajectories containing skill choices, tool interactions, reasoning, and answers to teach the structured interaction pattern.
  • Skill-Guided Agentic Post-Training: The composite reward combines answer correctness, format compliance, and tool-use efficiency, rewarding useful evidence with minimal redundant calls.
  • Skill-Guided Agentic Post-Training: Tool execution receives credit only when the trajectory obtains benchmark-required evidence, preventing selection of easier skills to inflate tool-use reward.

4 Experiments

Skill-3D is evaluated across multiple 3D spatial reasoning benchmarks and MLLM settings, showing stronger task performance and more effective, scene-aligned tool use than existing baselines.

  • Experimental Setup: Experiments cover VSI-Bench, BLINK, CV-3D, and MMSI-Bench, with representative spatial reasoning metrics reported on disjoint test samples.VSI-Bench includes eight indoor spatial reasoning categories; the other benchmarks assess multi-view reasoning, depth ordering and relative distance, and positional relationships.
  • Main Results: Skill-3D consistently outperforms non-agentic, direct tool-use, and Think3D baselines across all four closed-source MLLM agents.Its shared Skill Library is constructed from heterogeneous spatial reasoning benchmarks, enabling skills learned on one benchmark to transfer when similar scene-task contexts recur.
  • Main Results: 50.3% relative gain: averaged across four closed-source agents, Skill-3D raises the VSI-Bench average from 42.9 to 64.5 over the w/o Tools baseline.Consistent improvements are also observed on BLINK, CV-3D, and MMSI-Bench.
  • Main Results: 60.3% relative gain: Skill-3D-8B improves over the w/o Tools baseline on VSI-Bench, while Skill-3D-4B achieves a 59.7% relative gain.The gains span diverse spatial reasoning categories; the stronger 8B results indicate better exploitation of retrieved skills and tool evidence.
  • Ablation Study: ETU increases from 39.2% to 78.7% on VSI-Bench, from 36.4% to 79.2% on BLINK, from 31.8% to 87.5% on CV-3D, and from 30.5% to 80.3% on MMSI-Bench.Because ETU is normalized by total tool calls, the reported gains assess evidence-producing and evidence-integrating tool use rather than call volume alone.
  • Ablation Study: GPT-5.4 and Think3D show tool-selection bias across task groups, whereas Skill-3D produces a more task-aligned tool distribution.Think3D heavily relies on Pi3 and GPT-5.4 mostly calls GroundingDINO even when scene-task requirements differ.

5 Conclusion

Skill-3D addresses biased, uniform tool use in heterogeneous indoor 3D scenes by evolving reusable scene-aware skills from tool-use trajectories. Experiments show improved accuracy and effective tool usage, while skill-guided post-training transfers this behavior to compact agents.

  • Skill-3D constructs Scene Memory and evolves a Skill Library of reusable scene-aware skills from successful trajectories and lessons from failures.
  • Retrieved skills guide tool planning, evidence collection, and answer grounding during inference.
  • Skill-guided agentic post-training transfers scene-aware tool-use behavior into compact agents.
  • Skill-3D consistently improves accuracy and effective tool usage across the evaluated 3D spatial reasoning tasks.

Limitations

The evaluation focuses on indoor 3D spatial reasoning, and extending Skill-3D beyond that setting may require new interfaces, scene signatures, and safety constraints.

  • The current evaluation is limited to indoor 3D spatial reasoning.
  • Transfer to outdoor scenes, embodied navigation, or real-time robotic interaction may require new tool interfaces, scene signatures, and safety constraints.
  • The supplied limitation passage does not specify a broader empirical evaluation beyond indoor 3D spatial reasoning.

B.1 Efficiency Analysis.

Skill-3D achieves the strongest reported VSI-Bench efficiency and tool-use quality, combining higher performance and effective tool usage with lower inference time than the compared methods.

  • 70.0 is Skill-3D’s best VSI-Bench score, exceeding Direct tool use at 58.2 and Think3D at 64.7.
  • 78.7% effective tool usage is achieved by Skill-3D, compared with 39.2% for Direct tool use.
  • 20.8s is Skill-3D’s average inference time per query, versus 35.1s for Think3D, with 0.5s retrieval overhead.
  • Table B.1 reports average tool calls, effective tool usage, inference time per query, and performance gain for GPT-5.4 on VSI-Bench.

B.2 Cross-Benchmark Skill Transfer.

Dynamic skills transfer across the evaluated benchmarks, and combining training data from all benchmarks produces the strongest results by providing complementary scene-task knowledge.

  • Skills learned from VSI-Bench transfer effectively to MMSI-Bench and CV-3D, while MMSI-Bench skills also improve VSI-Bench.
  • Pooling all training benchmarks consistently achieves the best results across the evaluated transfer settings.
  • Table B.2 evaluates dynamic skills trained on one source benchmark against other target benchmarks using the same static skills and GPT-5.4.

C.1 Dataset Details

Skill-3D evaluates four 3D spatial reasoning benchmarks using category-wise random splits with question-level disjointness. Its post-training objective samples complete tool-use trajectories and optimizes rewards for correctness, format compliance, and tool-use efficiency.

  • Dataset Details: Category-wise random splits and question-level disjointness support fair comparison when benchmarks lack official training splits.The training set is used for skill construction and post-training, while reported results use the held-out test set.
  • Dataset Details: Four benchmarks—VSI-Bench, BLINK, CV-3D, and MMSI-Bench—cover indoor spatial intelligence, multi-view reasoning, geometric reasoning, and positional relationships.VSI-Bench includes counting, distance, size, route planning, and appearance-order tasks; the other benchmarks target complementary spatial capabilities.
  • Post-training Objective: Each trajectory contains skill choices, tool calls, tool outputs, intermediate reasoning, and a final answer, sampled in groups from the current policy.For each scene-task query, the policy observes the question, visual observations, and retrieved skill candidates before sampling G complete trajectories.
  • Post-training Objective: The scalar reward combines answer correctness, structured-format compliance, and tool-use efficiency, with tool efficiency based on execution and evidence obtained.Rexec is zero when no required tool evidence is obtained, and tool calls are penalized relative to the maximum tool budget.
  • Post-training Objective: GRPO normalizes rewards within each sampled group and optimizes a clipped surrogate objective using relative advantages.The KL term preserves SFT-learned skill-selection and tool-use behavior, while higher-success and more-efficient trajectories receive favorable advantages.

D Qualitative Results

Qualitative cases show Skill-3D replacing coarse, reconstruction-centric workflows with evidence-specific tool combinations. This improves both boundary-aware distance estimation and cross-view object counting.

  • Boundary-aware metric distance reasoning: Skill-3D predicts the correct 0.9m table–bathtub distance, whereas Think3D overestimates it as 1.5m.The Skill-3D workflow combines Pi3 reconstruction, object detection, and depth estimation to ground the closest object boundaries.
  • Boundary-aware metric distance reasoning: For metric distance estimation, Skill-3D combines room-level reconstruction and target localization with dense depth cues.Think3D relies on Pi3 reconstruction and object detection despite lacking explicit grounding of the closest boundaries.
  • Multi-view object counting: Skill-3D obtains the correct count of four chairs, while Think3D incorrectly counts five.Skill-3D uses Pi3 layout consistency and object detection to ground instances across views and suppress duplicates.
  • Multi-view object counting: The counting case attributes Think3D’s over-count to repeated chair appearances across adjacent views being treated as distinct instances.Skill-3D retrieves a detection-counting skill and applies cross-view de-duplication.

E Prompt Design

The prompt design specifies Skill-3D’s scene-aware agent identity, evidence-grounded tool-use requirements, skill retrieval, and structured reasoning outputs. It also illustrates these prompts through distance-estimation and object-counting cases.

  • Case Studies: The distance case asks for the closest-point distance between a table and bathtub, with a ground-truth answer of 0.9m.The associated example contrasts coarse reconstruction with a depth-distance workflow.
  • Case Studies: The counting case asks how many chairs are in the room, with a ground-truth count of four.The example uses a detection-counting skill and combines layout consistency with object detection to avoid duplicate counts.
  • System Prompt: The system prompt defines Skill-3D as a scene-aware multimodal spatial reasoning agent combining memory, skills, tools, and evidence-grounded final reasoning.It also requires detecting low-confidence outputs, tracking evidential support, and recording failures for future refinement.
  • Scene Context Prompt: The tool-use prompt requires tools for precise distances, closest-point boundaries, ambiguous object identity, cross-view targets, or answers not visible in one image.The internal format includes task type, scene context, final reasoning, skill choice, and tool calls.
  • Skill Retrieval Prompt: The skill-retrieval prompt ranks static and dynamic candidates using scene-task context and favors matched dynamic workflows.Available static skills include object counting, metric distance estimation, view selection, detection-depth fusion, and 3D reconstruction.
Loading 2606.07436v2…