PaperlayerView exact revision on arXiv

Source-linked AI summary

Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation

Yifeng Zhu, Peter Stone, Yuke Zhu

arXiv:2109.13841v2cs.ROcs.LG

TL;DR

Long-horizon robot manipulation requires reusable skills, but learning them from unsegmented demonstrations without manual temporal labels is challenging. BUDS builds hierarchical task structures, learns goal-conditioned skills, and trains a meta-controller to compose them. It outperformed both baselines across three tasks, with an average margin over 20%, and multi-task skills improved average performance by 8%.

  • Problem

    Learning reusable skills for long-horizon manipulation from unsegmented demonstrations remains challenging, particularly with high-dimensional sensory data and costly or ambiguous manual segmentation.

    Source: §I · p. 1

  • Method

    BUDS uses hierarchical clustering to find recurring segments in multi-task demonstrations, learns goal-conditioned sensorimotor skills, and trains a meta-controller to compose them.

    Source: §I · p. 2, B. Skill Segmentation with… · p. 4

  • Results

    BUDS outperformed both baselines for all three tasks by over 20% on average, and multi-task skills improved average task performance by 8% over individually learned skills.

    Source: B. Quantitative Results · pp. 6–7

  • Takeaways & Limitations

    BUDS combines reusable skills from multi-task demonstrations to solve long-horizon manipulation tasks, with evaluation in simulation and on real hardware.

    Source: §I · p. 2, §V · p. 7

    BUDS relies on offline demonstrations and requires a task-specific meta-controller for each task; composing skills for novel tasks without retraining one remains future work.

    Source: §V · p. 8

Abstract

from arXiv · show

We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations and use these skills to synthesize prolonged robot behaviors. Our method starts with constructing a hierarchical task structure from each demonstration through agglomerative clustering. From the task structures of multi-task demonstrations, we identify skills based on the recurring patterns and train goal-conditioned sensorimotor policies with hierarchical imitation learning. Finally, we train a meta controller to compose these skills to solve long-horizon manipulation tasks. The entire model can be trained on a small set of human demonstrations collected within 30 minutes without further annotations, making it amendable to real-world deployment. We systematically evaluated our method in simulation environments and on a real robot. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks. Furthermore, skills discovered from multi-task demonstrations boost the average task success by $8\%$ compared to those discovered from individual tasks.

I. INTRODUCTION

Long-horizon robot manipulation requires reusable sensorimotor skills, but building skill libraries from real-world sensory data without costly manual annotation remains challenging. BUDS discovers and composes skills from unsegmented human demonstrations, with reported gains in simulation and on a real robot.

I. INTRODUCTION

  • Prior skill-discovery methods can require high sample complexity, ground-truth physical states, or costly manual segmentation, while earlier hierarchical imitation work focused on individual tasks.Earlier segmentation methods also often fell short on high-dimensional sensor data.Source: §I · pp. 1–2
  • BUDS discovers sensorimotor skills from unsegmented human demonstrations and learns a meta-controller to select skills and predict their subgoals from the current state.The skills operate on raw images and robot proprioception, and both skills and meta-controllers are trained with imitation learning.Source: §I · p. 2
  • Bottom-up agglomerative clustering builds hierarchical task structures, while multisensory cues and multi-task demonstrations support coherent segmentation and reusable skills.The method trains goal-conditioned skills with hierarchical behavior cloning [14, 31, 32, 49].Source: §I · p. 2
  • BUDS achieved an average 66% success rate on three vision-based manipulation tasks and outperformed its most competitive baselines by over 20%.It also reached 56% success on a real-robot kitchen task, on par with simulation results.Source: §I · p. 2
  • Skills learned from multi-task demonstrations increased average task success by 8% over skills learned separately for individual tasks.The evaluation used 50–120 demonstrations per task, collected within 30 minutes.Source: §I · p. 2

II. RELATED WORK

Prior work studies skill discovery through self-exploration, demonstration segmentation, and hierarchical imitation. BUDS differs by learning reusable closed-loop sensorimotor skills from raw multisensory demonstrations across tasks.

Robot Skill Discovery

  • Self-exploration methods often require high sample complexity or ground-truth physical states, while demonstration-segmentation approaches can be difficult to scale to raw sensor data or require task-sketch annotations.BUDS instead operates directly on raw sensor data and requires no manual execution-stage labels.Source: §II · p. 2

Bottom-up Methods in Perception and Control

  • Bottom-up representations group fine-grained elements into hierarchies, and BUDS applies this principle to demonstrations for long-term manipulation behavior.The passage relates bottom-up processing to perception and behavior-based control.Source: §II · p. 2

Hierarchical Imitation in Robot Manipulation

  • Hierarchical imitation uses high-level subgoal prediction and low-level action policies, whereas BUDS extracts reusable skills across tasks and handles raw sensory data.Prior hierarchical imitation methods typically focus on single tasks.Source: §II · pp. 2–3

III. APPROACH

BUDS discovers reusable sensorimotor skills from multi-task, multisensory demonstrations and composes them to solve long-horizon tasks. Its approach has two stages: hierarchical clustering for segmentation, then hierarchical behavioral cloning for skills and meta-controllers.

III. APPROACH

  • BUDS discovers reusable sensorimotor skills from multi-task, multisensory demonstrations for composition into long-horizon tasks.The skills are learned as closed-loop sensorimotor policies.Source: §III · p. 3
  • Hierarchical agglomerative clustering segments unsegmented demonstrations into temporal segments that can be grouped into skills.The paper first formalizes the problem, then presents skill segmentation as its first approach step.Source: §III · p. 3
  • Hierarchical behavioral cloning learns the skills and meta-controllers that compose them.This is the second approach step described by the paper.Source: §III · p. 3

A. Problem Formulation

The paper formulates manipulation as learning a closed-loop policy over robot sensor data and motor actions. For long-horizon tasks, it learns goal-conditioned skills from recurring segments across tasks and trains a task-specific meta-controller to compose them.

A. Problem Formulation

  • The policy operates on robot sensor data, including raw images and proprioception, and outputs motor actions to maximize expected return.The task is formalized as a discrete-time Markov Decision Process.Source: A. Problem Formulation · p. 3
  • A two-level temporal hierarchy combines goal-conditioned low-level skills with a high-level meta-controller that selects a skill and specifies its goal.The meta-controller maps the current state to a skill index and a goal representation.Source: A. Problem Formulation · p. 3
  • The formulation assumes tasks share state and action spaces and transition probabilities, but differ in reward functions and initial-state distributions.Demonstrations are collected for instances of each task from different initial states.Source: A. Problem Formulation · p. 3
  • Hierarchical clustering identifies recurring temporal segments across the aggregated multi-task dataset, partitioning it into skill-training datasets.Training on the aggregated data augments data for individual tasks and supports skill reuse across tasks.Source: A. Problem Formulation · p. 3
  • A separate task-specific meta-controller is trained on each task's demonstrations to compose the learned skills.The low-level goal-conditioned skills are trained on the corresponding temporal-segment partitions.Source: A. Problem Formulation · p. 3

B. Skill Segmentation with Hierarchical Clustering

BUDS represents demonstration states using fused multisensory features, then builds hierarchical temporal segments and groups recurring segments across demonstrations into skill datasets.

B. Skill Segmentation with Hierarchical Clustering

  • BUDS fuses multi-view image and proprioceptive embeddings into a joint latent state representation trained to reconstruct current sensor inputs.The representation is intended to capture cross-modal statistical patterns while retaining information needed to decode the state.Source: B. Skill Segmentation with… · p. 4
  • Adjacent temporal segments are repeatedly merged by hierarchical agglomerative clustering according to the L2 distance between their average latent-state features.Clustering starts with segments of W = 10 steps and continues until each demonstration forms one tree.Source: B. Skill Segmentation with… · p. 4
  • A breadth-first search through each hierarchy selects intermediate segments, stopping branches at a minimum-length threshold and the search when the lowest-level segment-count threshold is met.The selected intermediate segments are used as temporal segments for the demonstration.Source: B. Skill Segmentation with… · p. 4
  • Spectral clustering with an RBF kernel groups temporal segments from all demonstrations into skill datasets, merging classes whose average segment length falls below a threshold into adjacent skills.Segment features concatenate representations of the first, middle, and last states; the final number of remaining classes is K.Source: B. Skill Segmentation with… · pp. 3–4

C. Policy Learning with Hierarchical Behavioral Cloning

Hierarchical behavioral cloning trains goal-conditioned sensorimotor skills and a task-specific meta controller to select skills and their subgoals. The controller learns from demonstrations augmented with cluster assignments and encoded subgoals.

C. Policy Learning with Hierarchical Behavioral Cloning

  • Each skill is trained on its corresponding temporal-segment dataset to map sensor observations and a subgoal vector to a robot action.The skill uses observed images and proprioception along with the subgoal.Source: C. Policy Learning with Hierarchical… · p. 4
  • A skill’s subgoal is the demonstrated future state H steps ahead, or the segment’s last state when the segment ends within H steps.This lets each skill pursue short-horizon goals rather than reason about the task’s long-term goal.Source: C. Policy Learning with Hierarchical… · p. 5
  • A task-specific meta controller predicts a skill index and latent subgoal conditioned on state, using a cVAE to model the multiple demonstrated choices available from the same state.At evaluation it samples a latent vector from the prior, and typically runs more slowly than the skills to avoid frequent switching.Source: C. Policy Learning with Hierarchical… · p. 5
  • Meta-controller training uses cluster assignments as skill-index labels and skill-encoder outputs as subgoal targets, optimizing an ELBO on demonstration data.The subgoal vectors are computed on demonstration states using encoders trained jointly with the skill policies.Source: C. Policy Learning with Hierarchical… · p. 5

IV. EXPERIMENTS

The experiments examine BUDS’s long-horizon manipulation performance, the effects of multi-task demonstrations and multimodal features, and operation on physical hardware.

IV. EXPERIMENTS

  • The evaluation asks whether BUDS performs well on long-horizon vision-based manipulation, whether multi-task demonstrations and multimodal features improve skill quality and reusability, and whether it works on real hardware.Source: §IV · p. 5
  • The illustrated simulation tasks are Tool-Use, Hammer-Place, Kitchen, and Multitask-Kitchen.Source: Figure 3 · p. 5
  • Real-Kitchen is the illustrated real-robot task in the experiment set.Source: Figure 3 · p. 5

A. Experimental Setup

The setup spans three single-task simulations, a multi-task kitchen domain, and a real-robot kitchen task, using teleoperated demonstrations and a Franka Panda arm.

A. Experimental Setup

  • Experiments use a 7-DoF Franka Emika Panda with a position-based operational-space controller and binary gripper command; the meta controller runs at 4 Hz and skills at 20 Hz.Source: A. Experimental Setup · p. 5
  • Tool-Use, Hammer-Place, and Kitchen require prolonged interactions involving tool use or articulated objects, with Kitchen comprising a sequence of cooking, serving, and stove-control subtasks.Source: A. Experimental Setup · p. 5
  • Multitask-Kitchen defines three tasks by differing drawer, stove, cube, and pot goal states, with three initial-configuration variants per task for testing skill reuse on new variants.Source: A. Experimental Setup · pp. 5–6
  • Real-Kitchen requires removing a pot lid, moving the pot and plate with a tool, and setting the tool down; observations come from workspace and eye-in-hand RGB cameras.Source: A. Experimental Setup · p. 6
  • Teleoperation demonstrations were collected with a SpaceMouse: 100 per single task, 120 per multi-task domain, and 50 for Real-Kitchen.Each single task took less than 30 minutes to collect; demonstrations contain images, proprioception, and actions.Source: A. Experimental Setup · p. 6

B. Quantitative Results

BUDS outperformed behavior cloning and change-point segmentation in single-task tests, while real-robot and multi-task experiments examined its performance and skill reuse.

Single Task Experiments

  • BUDS outperformed BC and CP by over 20% on average across Tool-Use, Hammer-Place, and Kitchen.Simulation success rates use 100 trials per task, repeated across five random seeds.Source: Table I · p. 6, B. Quantitative Results · p. 6

Failure Modes

  • Two reported BUDS failure modes were out-of-distribution skill selection and imprecise grasps, linked to limitations of imitation learning on small offline datasets.Across five repeated Kitchen runs, these accounted for 12.3% ± 2.9% and 9.0% ± 3.6% of evaluation trials, respectively.Source: B. Quantitative Results · p. 6

Comparisons to GTI

  • GTI's single-skill variant achieved 0.0% success from original Tool-Use starts, versus 58.6% for BUDS; on the shorter subtask, rates were 63.0% and 60.3%, respectively.The shorter subtask starts after the cube has been fetched and the tool placed down.Source: B. Quantitative Results · p. 6
  • Kitchen performance peaked with K = 6 skills; too few skills face diverse subgoals, while too many fragment training data.The authors used K = 6 for the main experiments.Source: B. Quantitative Results · p. 6, Table II · p. 7

Learning from Multi-task Demonstrations

  • Multi-task demonstrations improved average task performance by 8% over individual-task demonstrations, and learned skills composed to solve new task variants.Variant-3 had low test performance because it had more subtasks than its training counterparts, compounding skill execution failures.Source: Table III · p. 7, B. Quantitative Results · p. 7

Ablation Study on Demonstration State Representations

  • Multimodal state representations outperformed image-only and proprioception-only ablations; workspace-only images performed especially poorly under object occlusion.The study attributes the weaker workspace-only results to missing task-relevant object information under occlusion.Source: Table IV · p. 7, B. Quantitative Results · p. 7

Real Robot Experiments

  • BUDS achieved 56% success on Real-Kitchen, versus 18% for the CP baseline.The evaluation used 50 trials with varying initial configurations.Source: B. Quantitative Results · p. 7

V. CONCLUSION

BUDS discovers sensorimotor skills from recurring patterns in multi-task demonstrations and composes them for tasks; evaluations show effectiveness in simulation and on real hardware.

V. CONCLUSION

  • BUDS identifies recurring patterns in multi-task demonstrations, trains skills with imitation learning, and uses a meta-controller to compose them for tasks.The patterns are based on multisensory cues.Source: §V · p. 7
  • The results show BUDS is effective in simulation and on real hardware, and the paper examines model designs through ablation studies.Source: §V · p. 7
  • BUDS is limited by offline demonstration data and task-specific meta-controllers; future work proposes online experience and planning for novel tasks without a new controller.Source: §V · p. 8
Loading 2109.13841v2…