PaperlayerView exact revision on arXiv

Source-linked AI summary

Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation

Senwei Xie, Yuntian Zhang, Ruiping Wang, Xilin Chen

arXiv:2603.02623v1cs.ROcs.LG

TL;DR

Skill-centric planners can be limited by fixed skill libraries, while novel skills may require manual demonstrations. Uni-Skill detects skill gaps, extends planning, and grounds requested skills using SkillFolder, a repository built from annotated robotic videos. It reports zero-shot gains across simulation and real-world settings.

  • Problem

    Skill-centric approaches often depend on fixed skill libraries, and new skills may require manual demonstrations or waypoint annotations at deployment.

    Source: §I · p. 1

  • Method

    Uni-Skill combines skill-aware planning that generates descriptions for missing capabilities with SkillFolder, a hierarchical repository of automatically annotated video demonstrations.

    Source: §I · pp. 1–2

  • Results

    On RLBench tasks outside predefined skills, Uni-Skill’s zero-shot success rate outperforms MOKA by 31.0%.

    Source: §I · p. 2

  • Takeaways & Limitations

    Retrieved SkillFolder examples provide behavior patterns and spatial-trajectory references for few-shot inference without deployment-time demonstrations.

    Source: §I · p. 2

    Skill retrieval currently uses semantic hierarchies; incorporating mechanical properties remains a direction for future work.

    Source: §V · p. 8

Abstract

from arXiv · show

While skill-centric approaches leverage foundation models to enhance generalization in compositional tasks, they often rely on fixed skill libraries, limiting adaptability to new tasks without manual intervention. To address this, we propose Uni-Skill, a Unified Skill-centric framework that supports skill-aware planning and facilitates automatic skill evolution. Unlike prior methods that restrict planning to predefined skills, Uni-Skill requests for new skill implementations when existing ones are insufficient, ensuring adaptable planning with self-augmented skill library. To support automatic implementation of diverse skills requested by the planning module, we construct SkillFolder, a VerbNet-inspired repository derived from large-scale unstructured robotic videos. SkillFolder introduces a hierarchical skill taxonomy that captures diverse skill descriptions at multiple levels of abstraction. By populating this taxonomy with large-scale, automatically annotated demonstrations, Uni-Skill shifts the paradigm of skill acquisition from inefficient manual annotation to efficient offline structural retrieval. Retrieved examples provide semantic supervision over behavior patterns and fine-grained references for spatial trajectories, enabling few-shot skill inference without deployment-time demonstrations. Comprehensive experiments in both simulation and real-world settings verify the state-of-the-art performance of Uni-Skill over existing VLM-based skill-centric approaches, highlighting its advanced reasoning capabilities and strong zero-shot generalization across a wide range of novel tasks.

I. INTRODUCTION

Skill-centric approaches face a fixed-library gap: when a needed skill is unavailable, existing methods may require manual supervision for new skills. Uni-Skill addresses this with skill-aware planning and SkillFolder, a hierarchical repository of automatically annotated demonstrations.

I. INTRODUCTION

  • Existing skill-centric methods are constrained by fixed skill sets, and novel skills can require manually curated demonstrations or waypoint annotations at deployment.Unstructured robotic videos contain potential demonstrations but lack the skill links and annotations needed for direct library expansion.Source: §I · p. 1
  • Uni-Skill checks whether basic skills suffice, generates descriptions for missing capabilities, and uses those descriptions to match unstructured demonstrations.The descriptions extend the repository and serve as semantic anchors for implementation.Source: §I · p. 1
  • SkillFolder organizes over 10,000 skill traces into 106 VerbNet classes and 1,659 unique formulations for retrieving semantically relevant demonstrations.Retrieved traces offer behavior patterns and spatial-trajectory references for few-shot inference without deployment-time demonstrations.Source: §I · p. 2
  • On RLBench tasks outside predefined skills, Uni-Skill’s zero-shot success rate outperforms MOKA by 31.0%.The authors also report real-world performance improvements of 20.0% on long-horizon tasks and 34.0% in unseen skills.Source: §I · p. 2

II. RELATED WORKS

Related work covers task decomposition and skill reuse, skill-library construction, and retrieval of robotic demonstrations. Uni-Skill differs by structuring demonstrations around reusable skills for semantic retrieval.

Robotic task decomposition

  • Task-decomposition approaches use language plans and code APIs to connect high-level instructions with reusable atomic skills.End-to-end methods can require additional fine-tuning for new environments and tasks.Source: §II · p. 2

Construction of skill repository

  • Prior skill repositories often rely on fixed APIs or human feedback and demonstrations, while unstructured robotic videos offer a potential source for scalable library expansion.The videos lack explicit annotations, motivating hierarchical skill labels for efficient example retrieval.Source: §II · p. 2

Behavior retrieval

  • Retrieval-based behavior cloning typically matches demonstrations at the instance level, whereas Uni-Skill organizes them hierarchically for direct retrieval through semantic labels.The cited approaches otherwise require repeated feature matching and offer limited compositional generalization.Source: §II · p. 2

A. Skill-Aware Planning

Skill-aware planning expands a basic skill library on demand when it cannot meet a language-conditioned manipulation task. Uni-Skill formalizes this as sufficiency checking, skill synthesis, and planning over the augmented APIs.

A. Skill-Aware Planning

  • For intricate tasks such as cloth folding, compositions of basic skills may be insufficient, so the repository can expand on demand into a hybrid set of base and extended skills.The extended set may be synthesized or retrieved for specific task requirements.Source: A. Skill-Aware Planning · p. 3
  • The sufficiency discriminator checks whether basic skills meet the instruction, the generator synthesizes additional skills when needed, and the planner creates plans using the augmented set.These three modules are the sufficiency discriminator E, skill generator G, and planner P.Source: A. Skill-Aware Planning · p. 3
  • Given visual observations and language instructions, the planner emits executable API calls and parameters from the base or extended library.The calls are conditioned on the self-augmented API library.Source: A. Skill-Aware Planning · p. 3
  • The discriminator, generator, and planner share a multimodal, code-based format aligned with VLM pipelines; the VLM was trained on 106K runtime-code samples from video demonstrations.VLMs are described as serving as planners and skill descriptors through contextual understanding and visual grounding.Source: A. Skill-Aware Planning · p. 3

B. Automatic Skill Evolution

Automatic skill evolution turns unstructured videos into a structured SkillFolder, then retrieves and adapts examples to implement requested skills. Retrieved examples supply both semantic guidance and trajectory references for generating executable actions.

Automatic skill annotation

  • A three-stage VLM pipeline extracts procedures, describes skills, and aligns annotations to video intervals, producing over 10,000 slices from 350 hours of DROID demonstrations.The Descriptor can synthesize new task-oriented skill descriptions when the basic skill set does not suffice.Source: B. Automatic Skill Evolution · pp. 3–4

Hierarchical skill organization

  • SkillFolder organizes skills in a four-layer tree, from abstract VerbNet classes to visually grounded example slices.Its layers distinguish verb classes, verb instances, object-centric interaction templates, and fine-grained skill examples.Source: B. Automatic Skill Evolution · p. 4
  • Top-down matching expands SkillFolder when no existing node fits, yielding 106 VerbNet classes and 1,659 distinct skill descriptions.Annotated examples are matched successively to verb instances and skill descriptions.Source: B. Automatic Skill Evolution · p. 4

Few-shot skill implementation

  • At deployment, requested skills are matched through hierarchy levels, then candidates are filtered for scene similarity and discarded if trajectories go out of view or remain inactive.CLIP feature similarity assesses coherence in viewpoint and spatial layout.Source: B. Automatic Skill Evolution · pp. 4–5
  • Retrieved example trajectories and semantic constraints guide target contact-point and waypoint generation from the target scene and parameterized skill.The output trajectory has variable length, supporting flexible representation of long-horizon behavior.Source: B. Automatic Skill Evolution · p. 5
  • Sampled orientation patterns are transferred from demonstrations into target local frames and paired with waypoints to form executable 6-DoF poses.Fig. 3 depicts hierarchical retrieval and the progression from in-context guidance to waypoints and poses.Source: Figure 3 · p. 4, B. Automatic Skill Evolution · p. 5

IV. EXPERIMENT

The experiments compare Uni-Skill with policy-code-generation and visual-prompting baselines across tasks covered by the basic skills and tasks requiring skill extension.

IV. EXPERIMENT

  • The evaluation compares policy code generation (CaP) and visual prompting (MOKA) with Uni-Skill, using GPT-3.5 or GPT-4o for CaP and GPT-4o for MOKA.All methods share the same basic skill set, including AnyGrasp grasping and a top-down placement policy.Source: §IV · p. 5
  • Tasks are divided between those fully covered by the basic skills and those requiring generalization beyond them.The covered-skill group emphasizes planning and instruction understanding.Source: §IV · p. 5
  • The baseline set represents two foundation-model skill-centric approaches: policy code generation and visual prompting.CaP represents code generation; MOKA represents visual prompting.Source: §IV · p. 5

A. Simulation Experiments

In simulation, Uni-Skill is evaluated on RLBench tasks both within and beyond the predefined skills. It outperforms MOKA across out-of-library trajectory tasks, while its planning and retrieved examples support skill extension and compositional execution.

A. Simulation Experiments

  • RLBench evaluation includes eight tasks solvable with the basic skill set and ten additional tasks requiring skill extension.Each task is evaluated over 25 episodes, repeated three times, with mean and standard deviation of success rates reported.Source: A. Simulation Experiments · pp. 5–6
  • On covered RLBench tasks, Uni-Skill improves over CaP when instructions require visual grounding, as in interpreting “close the jar” to identify and rotate the gray lid onto the jar.The passage links this advantage to aligning free-form instructions with visual observations.Source: A. Simulation Experiments · p. 6
  • CaP fails to generalize to 10 RLBench tasks beyond the predefined skills without manually annotated deployment APIs, while Uni-Skill uses retrieved examples to implement self-augmented skills.The tasks span Revolute, Prismatic, Flip, Unplug, Lift, and Pour categories.Source: A. Simulation Experiments · p. 6
  • Uni-Skill outperforms MOKA on a wide range of RLBench trajectory-based tasks outside the predefined skills.Retrieved skill-centric demonstrations provide semantic constraints and referential guidance; Uni-Skill produces more semantically meaningful and physically feasible trajectories, particularly for Revolute and Prismatic tasks.Source: A. Simulation Experiments · p. 6
  • Uni-Skill decomposes complex scenarios requiring long-horizon planning and cross-skill generalization, while MOKA frequently fails on compositional Unplug and Lift tasks.The comparison attributes Uni-Skill's structured reasoning to its approach of grounding planning in skill examples.Source: A. Simulation Experiments · p. 6

B. Real-World Experiments

Real-world experiments evaluate Uni-Skill on eight task categories, including predefined-skill tasks and novel trajectory-focused tasks. Results describe stable performance on trajectory inference and an advantage on tasks combining long-horizon and spatial reasoning.

B. Real-World Experiments

  • The real-world evaluation uses eight diverse task categories, including Fold Cloth and Stir Blocks, with ten trials per task and layouts and object instances reconfigured after each trial.Tasks include both scenarios resembling simulation and novel trajectory-focused tasks.Source: B. Real-World Experiments · p. 6
  • On real-world tasks combining compositional and spatial reasoning, Uni-Skill decomposes Stir Blocks into subtasks and performs repeated rotational stirring motions at the target.Relevant SkillFolder examples supervise the self-augmented stirring procedure.Source: B. Real-World Experiments · p. 6
  • Uni-Skill shows more stable performance than MOKA on trajectory-based tasks such as Fold Cloth, while Uni-Skill and CaP perform strongly on long-horizon pick-and-place tasks.The passage attributes Uni-Skill's Fold Cloth performance to retrieved skill-centric examples.Source: B. Real-World Experiments · p. 6

C. Ablation

The ablations identify skill updating as critical for generalizing beyond predefined skills and show that retrieved semantic and spatial references play different roles across task types.

C. Ablation

  • Disabling skill updating makes Uni-Skill fail to perform reasonably on most tasks outside the predefined skills.Source: C. Ablation · p. 7
  • Semantic constraints matter more for contact-sensitive, holistic tasks such as Revolute and Flip, while spatial references matter more for precise-reasoning tasks such as Unplug and Lift.Source: C. Ablation · p. 7
  • Retrieved examples’ semantic and spatial references play distinct roles in few-shot skill implementation without manual intervention.Source: C. Ablation · p. 7

D. Discussion on adaptation and robustness

Uni-Skill adapts its skill repository to different video sources, while filtering improves repository quality and a closed-loop strategy addresses several failure types.

D. Discussion on adaptation and robustness

  • DROID provides robot-centric videos, while hand detection enables skill-segment extraction from ego-centric videos such as sth2sth, which cover broader skill categories.Source: D. Discussion on adaptation and… · p. 7
  • Human-centric videos lack action annotations and may be blurred, degrading repository quality; they remain promising for scaling but require quality control and filtering.Source: D. Discussion on adaptation and… · p. 7
  • Failures span planning, grounding, directional, and execution errors; self-correction further improves Close Drawer and Seat Down success rates by 12%.Source: D. Discussion on adaptation and… · p. 7
  • Among 135 sampled SkillFolder slices, 2 had semantic mismatches, 7 were occluded or too short, and none had execution failures; length and boundary filtering reduced low-quality data below 2%.Source: D. Discussion on adaptation and… · p. 7

V. CONCLUSION AND FUTURE WORK

Uni-Skill targets zero-shot generalization beyond predefined skills through skill-gap detection and augmentation. SkillFolder connects automatically annotated demonstrations to a structured taxonomy, with retrieval design left open for further exploration.

V. CONCLUSION AND FUTURE WORK

  • Uni-Skill detects skill gaps and augments new skills to support zero-shot generalization beyond predefined skills.Source: §V · p. 7
  • SkillFolder bridges automatically annotated demonstrations with a structured skill taxonomy, and retrieved examples guide performance on diverse tasks and unseen skills.Source: §V · p. 8
  • The authors identify retrieval that jointly considers mechanical properties and semantic categories as a future direction to reduce skill overlap and improve sample retrieval.Source: §V · p. 8

Related papers

Loading 2603.02623v1…