PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillWeave: Weaving Heterogeneous Demonstrations into Long-Horizon Manipulation Skills

Ryosei Tamura, Xiaoxiang Dong, Uksang Yoo, Yuemin Mao, Romina Mir, Jonathan Francis, Jeffrey Ichnowski

arXiv:2610.12046v1cs.RO

TL;DR

Long-horizon dexterous manipulation requires demonstrations for both broad task progression and precise contact-rich interaction. SkillWeave combines modality-matched demonstrations with object-mask conditioning and successor-aware handoffs, achieving 27% average end-to-end success across three real-world tasks.

  • Problem

    Long-horizon dexterous manipulation must combine free-space task progression with contact-rich interaction, while kinesthetic demonstrations create visual mismatch and independent policies can have incompatible handoff states.

    Source: §I · p. 2, B. Policy Composition and… · p. 4

  • Method

    SkillWeave pairs teleoperation for broad task coverage with kinesthetic teaching for contact-rich skills, using object-mask-conditioned policies and successor-aware terminal steering.

    Source: §I · p. 2

  • Results

    27% average end-to-end success across three tasks; mask-conditioned kinesthetic teaching achieved 65% average success on dexterous sub-tasks.

    Source: Sub-Task Policy Evaluation · p. 6, Long-Horizon Task Evaluation · p. 7

  • Takeaways & Limitations

    Across three real-world tasks, the heterogeneous policies were effectively composed for end-to-end execution.

    Source: §VII · p. 8

Abstract

from arXiv · show

Dexterous manipulation requires both large-scale task progression and precise contact-rich interaction, making it challenging to collect demonstrations that effectively support both regimes. We present SkillWeave, a heterogeneous demonstration framework for long-horizon dexterous manipulation that combines teleoperation for coarse reaching and transport with kinesthetic teaching for precise, contact-rich skills. To address the visual mismatch introduced by the demonstrator's presence during kinesthetic data collection, we propose an object-mask-conditioned diffusion policy that uses offline object segmentation for training supervision and a lightweight learned mask predictor at deployment, avoiding online segmentation and image inpainting. To mitigate distribution shift between independently trained sub-task policies, we introduce successor-aware terminal steering, which selects among actions sampled from the predecessor policy to guide the system toward states supported by the successor's demonstrated initial-state distribution. Across three real-world long-horizon tasks, SkillWeave achieves 27% average end-to-end success. Mask-conditioned kinesthetic policies improve dexterous sub-task success to an average of 65%, while successor-aware handoffs achieve an average composition efficiency of 87%. These results show that matching demonstration modality to interaction regime, explicitly addressing kinesthetic visual mismatch, and steering policy handoffs toward successor-supported states substantially improves long-horizon dexterous manipulation. Videos and code are available at skillweave-authors.github.io .

I. INTRODUCTION

SkillWeave matches teleoperation and kinesthetic teaching to different manipulation phases, while addressing visual mismatch and policy-boundary failures that complicate long-horizon execution.

I. INTRODUCTION

  • Independently trained sub-tasks can fail to compose because predecessor terminal states drift from the successor’s demonstrated initial-state distribution.The resulting out-of-distribution initialization can cause successor failure and compound across a sequence of sub-tasks.Source: §I · p. 2
  • Kinesthetic demonstrations create a visual mismatch because the demonstrator occludes objects during training but is absent during autonomous execution.The paper’s approach uses offline SAM 3 masks to supervise a learned mask predictor, avoiding online SAM 3 and image inpainting.Source: §I · p. 2
  • The experiments test modality matching, bridging into successor-supported initial states, and mask conditioning for robustness to background and scene changes.The stated scope is long-horizon, contact-rich manipulation.Source: §I · p. 2
  • SkillWeave combines teleoperation for task-level coverage and free-space reaching with kinesthetic teaching for precise, contact-rich phases.The paper argues these modalities fit different interaction regimes: teleoperation supports task progression, while kinesthetic teaching provides direct guidance through robot-feasible contact configurations.Source: §I · pp. 1–2
  • Successor-aware terminal steering guides predecessor policies toward states represented in the successor’s demonstrated initial-state distribution.The bridge addresses distribution shift at policy boundaries.Source: §I · p. 2

II. RELATED WORK

Prior work studies demonstration modalities, skill composition, and visual invariance, while SkillWeave combines modalities by task phase, bridges independently trained skills, and conditions policies on object masks.

A. Demonstration Collection for Manipulation

  • Teleoperation can suffer from human–robot embodiment mismatch during contact, while kinesthetic teaching avoids retargeting but prior approaches trade open-loop replay inconsistency against inpainting errors.The paper motivates pairing teleoperation coverage with kinesthetic contact-rich corrections; Diffusion Policy is cited as conditional denoising over action sequences for multimodal prediction and receding-horizon execution [4].Source: A. Demonstration Collection for… · p. 2

B. Skill Composition

  • Prior skill-composition methods address terminal-to-initial mismatch through transition policies, terminal-state regularization, or feasibility-guided switching, often modifying policies or requiring reinforcement-learning interaction.SkillWeave instead uses an explicit bridge while leaving the constituent policy parameters unchanged.Source: B. Skill Composition · p. 2

C. Visual Invariance in Demonstration Learning

  • Visual-invariance approaches include image editing, inpainting, object-centric representations, and robot masks, but can depend on reconstructing occluded pixels or observable objects.SkillWeave avoids image reconstruction and uses segmentation only as training-time supervision; Fig. 2 depicts its mask-conditioned policy architecture.Source: C. Visual Invariance in Demonstration… · pp. 2–3, Figure 2 · p. 3

III. PROBLEM STATEMENT

The paper studies long-horizon tasks composed of sequential sub-tasks, each learned from a selected demonstration modality, and targets visual mismatch and policy handoff failures.

III. PROBLEM STATEMENT

  • Each sub-task uses teleoperation for coarse arm motions such as reaching and transport, or kinesthetic teaching for precise, contact-rich manipulation.Demonstrations from the selected modality form a sub-task dataset used to train its policy.Source: §III · p. 3
  • The goal is to compose independently trained sub-task policies into a framework that completes the long-horizon task.The task is described as a sequence of K sub-tasks.Source: §III · p. 3
  • Two challenges are human-hand occlusion in kinesthetic demonstrations and incompatible terminal and initial states between independently trained policies.The visual mismatch arises because the demonstrator is absent at autonomous execution; unsuitable successor initialization can make handoffs fail.Source: §III · p. 3
  • The study addresses heterogeneous demonstration modalities, modality-specific visual mismatch, and reliable composition into long-horizon behaviors.These are the stated aims of the problem formulation.Source: §III · p. 3

IV. METHOD

SkillWeave trains diffusion policies for modality-matched sub-tasks, conditions kinesthetic policies on object masks, and steers predecessor actions toward successor-supported states at handoffs.

A. Sub-Task Policies

  • Each sub-task policy is trained from its corresponding demonstration dataset using a diffusion policy.Teleoperated policies aggregate image features with spatial softmax and concatenate visual representations with robot proprioception.Source: A. Sub-Task Policies · p. 3
  • Kinesthetic observations include the demonstrator’s hand and arm during collection, creating visual mismatch when those are absent at autonomous execution.The mismatch can draw visual attention away from the manipulated object.Source: A. Sub-Task Policies · p. 3
  • The policy pools visual features within the downsampled object mask to form an object-conditioned query; camera-view features are combined with qt to condition a 1D U-Net that denoises an action chunk.Each camera observation is encoded by a ResNet-18 visual encoder.Source: A. Sub-Task Policies · p. 3
  • Object-mask attention is supervised during training, and views where the object is not visible are excluded from the attention loss.The attention target is the normalized object mask.Source: A. Sub-Task Policies · pp. 3–4
  • At a handoff, candidate action chunks from the predecessor are evaluated by predicted terminal-state proximity to the successor’s demonstrated start-state distribution.The closed-loop procedure executes the best candidate and reduces boundary distribution shift without transition demonstrations or a separately trained policy.Source: Figure 3 · p. 4, A. Sub-Task Policies · p. 4
  • Offline SAM 3 masks supervise training, while lightweight per-view U-Nets predict masks from RGB observations at deployment.This retains original RGB observations and avoids online SAM 3 and image inpainting.Source: A. Sub-Task Policies · pp. 3–4

B. Policy Composition and Successor-Aware Handoff

Independently trained sub-task policies can produce handoffs that start outside the successor’s demonstrated distribution. Successor-aware terminal steering modifies the predecessor’s terminal phase to guide the system toward successor-supported states before transfer.

B. Policy Composition and Successor-Aware Handoff

  • Successor-aware terminal steering guides the predecessor toward the successor’s demonstrated start states, addressing out-of-distribution failures at policy boundaries.Independent policies can end in states unlike those represented in the successor’s demonstrations, even when each policy succeeds in isolation.Source: B. Policy Composition and… · p. 4
  • The method samples multiple candidate action chunks from the predecessor near task completion and selects the chunk whose predicted endpoint best matches the successor’s start-state distribution.During normal execution the predecessor is unmodified; steering proceeds through multiple replanning steps.Source: B. Policy Composition and… · p. 4
  • The successor’s start-state distribution is represented by the first Hs states from its training demonstrations, using wrist pose and arm and hand joint configurations.State components are normalized before distances are computed.Source: B. Policy Composition and… · p. 4
  • A state’s compatibility with the successor is estimated from its K nearest neighbors in the successor start-state set; smaller distance means greater compatibility.The procedure repeats after each executed chunk and hands control over when the state is sufficiently close to demonstrated successor starts.Source: B. Policy Composition and… · pp. 4–5
  • Candidates remain actions sampled from the predecessor, so steering avoids an arbitrary interpolated trajectory or a separately trained transition controller.The method requires neither additional transition demonstrations nor a separately trained transition policy, and leaves the successor unchanged.Source: B. Policy Composition and… · p. 5

V. EXPERIMENTS

The experiments evaluate SkillWeave on three real-world long-horizon manipulation tasks. They examine kinesthetic skill learning and the composition of trained sub-task policies.

V. EXPERIMENTS

  • The evaluation tests SkillWeave on three real-world, long-horizon manipulation tasks.Source: §V · p. 5
  • One experiment question is whether a mask-conditioned diffusion policy can learn kinesthetic skills and execute them without the demonstrator’s hand in view.Source: §V · p. 5
  • A second question is whether trained sub-task policies can be composed to achieve robust long-horizon task execution.Source: §V · p. 5

A. Demonstration Collection

Demonstrations were collected with an xArm7 and LEAP Hand using both teleoperation and kinesthetic teaching. The collection setup records synchronized visual and robot-state observations.

A. Demonstration Collection

  • The platform was a 7-DoF xArm7 arm equipped with a 16-DoF LEAP Hand, observed by third-person and wrist-mounted RGB cameras.Source: A. Demonstration Collection · p. 5
  • Each sub-task had 100 demonstrations, collected by teleoperation or kinesthetic teaching according to its required motion modality.Source: A. Demonstration Collection · p. 5
  • Teleoperation used a 6-DoF SpaceMouse for arm control and camera-tracked finger motion retargeted to the robot hand; kinesthetic teaching used drag/teach mode and a compliant controller.The controller yields to manual finger adjustments and holds the resulting poses.Source: A. Demonstration Collection · p. 5
  • Object positions were randomized, and synchronized camera images and robot proprioception were recorded at 30 Hz for separate sub-task policies.Proprioception included end-effector pose, arm joint angles, and hand joint angles.Source: A. Demonstration Collection · p. 5

B. Task Descriptions

Three long-horizon tasks combine teleoperated coarse movements with kinesthetically demonstrated dexterous interactions. The task decompositions and sub-task success criteria are summarized in Table I.

B. Task Descriptions

  • Across all three tasks, teleoperation demonstrates reaching or transport, while kinesthetic teaching supplies precise interaction skills.Source: B. Task Descriptions · p. 5
  • In cube reorientation and in-hand rotation, teleoperation covers reaching, grasping, and wrist reorientation, while kinesthetic teaching covers in-hand rotation.Source: B. Task Descriptions · p. 5
  • In nut removal and storage, teleoperation covers reaching and container placement, while kinesthetic teaching covers unscrewing the nut from the bolt.Source: B. Task Descriptions · p. 5
  • In kettle preparation and placement, teleoperation covers reaching and positioning under a faucet, while kinesthetic teaching covers handle grasping and pressing the lid release.Source: B. Task Descriptions · p. 5
  • Table I summarizes each long-horizon task’s sub-task decomposition and success criteria.Source: B. Task Descriptions · p. 5

C. Evaluation Metrics

The section reports individual sub-task success and end-to-end task success, then compares observed end-to-end success with the level expected from isolated sub-task performance to assess losses from policy composition.

C. Evaluation Metrics

  • Individual sub-task success S(j)_i and end-to-end task success S_i are reported as separate metrics.The supplied passages do not provide further definitions for these symbols.Source: C. Evaluation Metrics · p. 5
  • Expected success from isolated sub-task performance serves as a reference for quantifying losses introduced by policy composition.Values near 1 indicate that handoffs add little degradation beyond isolated sub-task failure rates.Source: C. Evaluation Metrics · p. 5

D. Baselines

SkillWeave combines teleoperation for coarse arm-level motions with mask-conditioned kinesthetic teaching for precise dexterous skills, then steers policy handoffs. Across three tasks, it averages 27% end-to-end success; the supplied results also show that modality suitability differs by sub-task.

D. Baselines

VI. RESULTS

  • Evaluation uses 20 randomized rollouts per policy or task; Figure 5 orders the three tasks from Cube Reorientation to Kettle Preparation and colors teleoperation blue and mask-conditioned kinesthetic frames red.The long-horizon evaluation compares complete tasks and representative SkillWeave rollouts.Source: Sub-Task Policy Evaluation · p. 6, Long-Horizon Task Evaluation · p. 7, Figure 5 · p. 7
  • 45%, 15%, and 20% end-to-end success on Cube Reorientation, Nut Removal and Storage, and Kettle Preparation yield 27% average; single-strategy baselines record no successes.Table II covers individual sub-task and end-to-end system success rates; results are over 20 trials.Source: Table II · p. 6, Long-Horizon Task Evaluation · p. 7
  • 64% average teleoperation success on arm-level reaching, reorientation, and placement contrasts with 5% for teleoperation and 65% for mask-conditioned kinesthetic teaching on dexterous hand tasks.The dexterous set comprises in-hand cube rotation, nut unscrewing, kettle-handle grasping, and button pressing.Source: Sub-Task Policy Evaluation · pp. 6–7
  • Across dexterous sub-tasks, raw kinesthetic observations achieve 10% average success, KineDex 20%, and mask-conditioned policies 65% when evaluated without the demonstrator’s hand.The comparison tests explicit object-mask conditioning against raw observations and inpainting.Source: Sub-Task Policy Evaluation · pp. 6–7
  • Without steering, composing the same sub-task policies substantially degrades end-to-end success; terminal steering guides handoffs toward successor-compatible states.Figure 6 qualitatively contrasts predecessor terminal finger-joint states with successor start states, with closer states under steering.Source: Long-Horizon Task Evaluation · p. 7, Figure 6 · p. 7

VII. CONCLUSIONS AND FUTURE WORKS

SkillWeave combines teleoperation for coarse arm motions with kinesthetic teaching for precise, contact-rich interaction, then uses mask-conditioned policies and bridging motions to support autonomous execution. Experiments on three real-world long-horizon tasks demonstrate complementary strengths of the modalities and effective policy composition; future work targets automation, reduced mask reliance, and broader skills and environments.

VII. CONCLUSIONS AND FUTURE WORKS

  • Experiments on three real-world long-horizon tasks show complementary strengths of teleoperation and kinesthetic teaching and effective composition of heterogeneous policies for end-to-end execution.Teleoperation handles coarse arm motions, while kinesthetic teaching supports precise, contact-rich interactions.Source: §VII · p. 8
  • SkillWeave combines teleoperation for coarse arm motions with kinesthetic teaching for precise, contact-rich interactions, using mask-conditioned diffusion policies and bridging motions for robust autonomous execution.The policies mitigate the visual mismatch in kinesthetic demonstrations, while bridging motions address handoff failures during skill composition.Source: §VII · p. 8
  • Future work includes automating task decomposition and modality choices, reducing reliance on explicit object masks, and extending the framework to larger skill libraries, novel objects, and more diverse environments.Source: §VII · p. 8
Loading 2610.12046v1…