PaperlayerView exact revision on arXiv

Source-linked AI summary

DexStitch: Learning and Composing Dexterous Skills from Human Videos via Contact-Aware Robotic Inpainting

Byeonggyeol Choi, Seongmin Jung, Unghui Nam, Youngbo Song, Sunhong Min, MinHyung Lee, Byonghyo Shim, Yongseok Lee

arXiv:2610.17789v1cs.ROcs.CV

TL;DR

Human videos can provide demonstrations, but they lack robot actions and skill labels, and collecting both through teleoperation and annotation is costly. DexStitch segments, converts, and composes those demonstrations into robot-executable skills and transitions. Its full model reaches 77.5% sequence success.

  • Problem

    Robot teleoperation and manual annotation of action and skill supervision are costly, while human videos lack robot actions and skill labels.

    Source: §I · p. 1

  • Method

    DexStitch segments human videos into skills and transitions, converts them into robot-domain observation–action data with skill labels, and composes them using a skill-conditioned policy.

    Source: §I · p. 2

  • Results

    77.5% sequence success for the full model, compared with 35.0% using only teleoperated data and 5.0% using only synthetic robot data.

    Source: E. Ablation Studies · p. 7

  • Takeaways & Limitations

    Human demonstrations can provide both executable robot supervision and reusable skill structure.

    Source: §VI · p. 8

Abstract

from arXiv · show

A long-horizon manipulation task consists of primitive skills and the transitions connecting them. Learning skills that can be executed in sequence requires supervision for both, which is costly to collect on a robot. Human videos offer a scalable source of demonstrations, but transferring them to robots requires extracting robot-compatible observations and actions together with their skill structure. We present DexStitch, a framework that cuts long-horizon human videos into skills and transitions, converts them into aligned robot observation-action-skill data, and stitches the learned skills into a requested sequence. A skill classifier first segments the videos into manipulation skills and transitions. DexStitch then converts the human demonstrations into robot observations and actions while retaining their corresponding skill labels. During conversion, per-finger human-object contacts guide dexterous hand retargeting and robotic inpainting, preserving contact geometry and finger-object occlusion ordering. To stitch the learned skills into a sequence, a shared skill-conditioned policy is pretrained on the resulting synthetic data and finetuned on a mixture of synthetic data and a small set of single-skill teleoperation episodes. Conditioning on the current and next skills guides each transition toward the following skill, while predicted skill progression enables closed-loop switching. On a held-out session, the classifier agrees with manual annotations on 92.4% of frames. Across four household skills, DexStitch achieves 80.0% single-skill success and 77.5% sequence success.

I. INTRODUCTION

DexStitch turns long human demonstrations into robot-domain data with explicit skill labels, then composes learned skills and transitions into requested sequences. Per-finger contact supports both dexterous retargeting and visual compositing, while a shared policy uses current and next skills to guide transitions.

Motivation

  • Robot teleoperation and manual skill annotation are costly, while human videos lack robot actions and skill labels; existing transfer approaches do not provide both explicitly labeled robot data and transition boundaries.Latent-skill transfer represents structure without explicit skill-labeled robot observation–action pairs, while motion retargeting and rendering alone do not identify skills and transitions.Source: §I · pp. 1–2

Framework

  • DexStitch segments long human videos into skills and transitions, converts them to the robot domain, and stitches them in a requested order.A classifier trained on framewise annotations segments the remaining videos; long demonstrations contain many transitions at once, avoiding separate recordings for every ordered skill pair.Source: §I · p. 2

Sequential composition

  • The shared policy is pretrained on synthetic demonstrations and finetuned with a small set of teleoperated single-skill episodes to learn skills and transitions.Transitions are supervised solely by human video; conditioning on the current and next skills guides each transition, and predicted skill completion advances the sequence.Source: §I · p. 2

Evaluation

  • The evaluation examines four household skills through executable-skill comparisons, data and policy ablations, conversion-quality comparisons, and held-out video segmentation.The classifier is compared with two vision–language models, but this passage does not report the evaluation outcomes.Source: §I · p. 2

Contact-aware conversion

  • DexStitch uses per-finger human–object contact to refine dexterous hand retargeting and determine finger–object occlusion during robot rendering.This conversion retains contact geometry while transferring frame-level skill labels to the synthetic robot data.Source: §I · p. 2

II. RELATED WORK

Prior work studies skill structure in human video, transfers human demonstrations into robot execution, and composes skills with different treatments of transitions. DexStitch makes transitions explicit labels and executes them with the same policy as other skills.

A. Learning Skill Structure from Human Video

  • DexStitch fixes a skill set in advance, labels transitions as skills, and automatically segments human videos into those labels, unlike latent-skill approaches that sequence continuous representations.Prior approaches include latent plans from play, reusable skill primitives, and cross-embodiment latent skills; XSkill and UniSkill do not tell the policy which skill comes next.Source: A. Learning Skill Structure from Human… · p. 2

B. Transferring Human Videos to Robot Execution

  • Robotic inpainting pairs rendered robot frames with retargeted actions, but prior methods leave skill and transition boundaries unmarked; DexStitch adds teleoperated episodes only during finetuning.DexStitch uses per-finger contact from RGB video both to refine the hand configuration and to determine which fingers pass behind the object during compositing, without depth or an object model.Source: B. Transferring Human Videos to Robot… · pp. 2–3, Figure 2 · p. 3

C. Skill Composition and Transition

  • Prior methods use dedicated transition policies, feasibility-based switching, planners, or implicit transitions; DexStitch labels transitions as skills and runs them with the same policy, conditioned on the next skill.The policy is told which skill comes next rather than treating transitions as implicit or assigning them to a separate module.Source: C. Skill Composition and Transition · p. 3

III. HUMAN-ROBOT DATA CONVERSION

DexStitch converts egocentric human video into robot-domain observations and actions while retaining each frame’s skill label. It estimates hand motion and per-finger contact, uses contact to refine dexterous retargeting, and smooths the resulting robot trajectory.

III. HUMAN-ROBOT DATA CONVERSION

  • The conversion estimates hand motion and contact, retargets the hand to the robot, and renders the robot into the source frames to produce aligned training data.Because conversion is frame-wise, each converted frame retains the skill label of its source frame.Source: §III · p. 3

A. Hand Pose and Contact Estimation

  • HaWoR reconstructs 3D hand motion, while HaCo estimates contact probabilities over MANO vertices and pools them into temporally stable per-finger contact states.Contacting vertices are retained as contact points that constrain retargeting and occlusion ordering.Source: A. Hand Pose and Contact Estimation · p. 3

B. Contact-Aware Retargeting

  • DexPilot initializes hand articulation by matching wrist-to-fingertip vectors, then contact-constrained optimization refines joint angles so contacting robot fingertips approach the demonstrated contact points.The refinement uses a surface-normal alignment constraint and a quadratic regularizer; the arm is obtained by inverse kinematics from the calibrated camera-to-base transform, and per-frame configurations are smoothed over time.Source: B. Contact-Aware Retargeting · pp. 3–4

C. Inpainting and Compositing

DexStitch removes the human from each frame, preserves the manipulated object, and composites the retargeted robot into the scene. Contact states guide which fingers appear behind the object, yielding robot-domain observations paired with action targets.

Inpainting and rendering

  • SAM2 tracks the human and object masks, ProPainter inpaints the human while protecting the object, and the robot is rendered using the calibrated camera placement and intrinsics.This produces robot-domain observations aligned with the retargeted trajectory.Source: C. Inpainting and Compositing · p. 4

Contact-aware occlusion

  • For an assumed antipodal grasp, contacting non-thumb fingers are treated as hidden behind the object, and overlapping object-mask regions are drawn in front of the robot.The method projects and dilates those fingers’ 3D contact points to identify the image region where the object takes precedence.Source: C. Inpainting and Compositing · p. 4

Layered compositing

  • Compositing orders the inpainted background, hidden robot fingers, source-frame object, and remaining robot pixels from back to front.The object comes from the source frame because inpainting alters the region where the hand met it; paired with the retargeted trajectory, the converted frames provide robot observations and action targets.Source: C. Inpainting and Compositing · p. 4

IV. SKILL LEARNING AND COMPOSITION

DexStitch uses a skill classifier and a skill-conditioned policy to recover and execute demonstrated skill sequences. The classifier labels manipulation and transition behavior using a frozen video encoder and a small trainable head.

IV. SKILL LEARNING AND COMPOSITION

  • The classifier and skill-conditioned policy are jointly used at inference to execute and compose demonstrated skills.The framework is trained from synthetic robot data and a small set of real robot demonstrations.Source: §IV · p. 4

A. Skill Classifier

  • The classifier treats transitions connecting manipulation behaviors as ordinary skill labels.This makes both behaviors and transitions part of the recovered skill sequence.Source: A. Skill Classifier · p. 4

1) Architecture Design and Training:

  • A frozen V-JEPA 2 encoder processes video at 2 Hz, while an MLP predicts the latest skill from a causal four-second window.Only 296K parameters are trainable, allowing fitting from a handful of annotated sessions.Source: 1) Architecture Design and Training · p. 4
  • The classifier predicts a skill label at each token, and each token’s label is held across the frames it represents.The encoder supplies one token per pair of frames sampled at 4 FPS.Source: 1) Architecture Design and Training · p. 4

2) Skill Annotation Generation:

The classifier’s frame-wise skill predictions are smoothed into an ordered sequence of segments, then paired with synthetic robot data for policy training.

2) Skill Annotation Generation:

B. Skill-Conditioned Policy

A shared policy learns individual skills and the motions connecting them from robot observations, actions, and corresponding human skill labels. It conditions on the current and next skills and predicts action chunks alongside skill progression for closed-loop switching.

B. Skill-Conditioned Policy

  • The shared policy learns both individual skills and the motions connecting them from synthetic robot observations, actions, and human skill labels.Source: B. Skill-Conditioned Policy · p. 5

1) Architecture Design:

  • Conditioning on the current skill specifies behavior, while the next skill guides how it terminates and connects to what follows.The policy predicts an action chunk and skill progression; predicted labels support closed-loop skill switching.Source: 1) Architecture Design · p. 5

2) Training:

DexStitch pretrains on synthetic demonstrations, then finetunes on a mixture of synthetic and limited teleoperated data. At inference, predicted skill progression advances execution through the requested sequence; experiments use long-horizon demonstrations and isolated teleoperated skill episodes.

2) Training:

  • Pretraining on complete synthetic sequences teaches skills and inter-skill motions without teleoperated transition demonstrations.Finetuning mixes synthetic data with a small set of teleoperated episodes to ground the policy in the target embodiment while maintaining coverage of human-derived skills and transitions.Source: 2) Training · p. 5

C. Compositional Inference

  • Execution advances when the policy consistently predicts the next skill, with consecutive predictions aggregated to avoid premature switches.The policy is conditioned on the current and next entries in the requested sequence.Source: C. Compositional Inference · p. 5

A. Experimental Setup

  • The dataset contains approximately 240 long-horizon human episodes and about 32 teleoperated episodes, roughly eight isolated demonstrations per skill.The teleoperated demonstrations are used to finetune the policy.Source: A. Experimental Setup · p. 5
  • The diffusion policy uses 224 × 224 RGB images and a 33-D proprioceptive-and-skill state, predicts 33-D targets, and executes 18-DoF actions at 20 Hz.It predicts 32 steps and executes the first 16, using 10 DDIM denoising steps at inference.Source: A. Experimental Setup · p. 5
  • Figure 4 compares three grasp scenes and shows DexStitch preserving hand–object occlusion that the other two methods lose.Source: Figure 4 · p. 6
  • Figure 5 plots redbox end-effector trajectories in 3D and x–y views for teleoperation, DexStitch, and UniSkill.Source: Figure 5 · p. 6

B. Skill Learning and Composition

DexStitch learns skills and transitions in a shared policy, using explicit transition skills and next-skill conditioning to compose four-skill sequences. It achieves 80.0% single-skill success and 77.5% sequence success, while UniSkill’s sequence success is 0%.

B. Skill Learning and Composition

  • 80.0% single-skill success and 77.5% sequence success; UniSkill achieves 45.0% average single-skill success and 0% sequence success.Both methods use the same synthetic robot data, pretraining steps, and teleoperated finetuning, but differ in skill representation.Source: B. Skill Learning and Composition · p. 6
  • DexStitch models transitions with an explicit transition skill and next-skill conditioning, while UniSkill has no explicit transition mechanism.Conditioning guides each transition toward the next skill; the policy is evaluated on 10 four-skill sequence rollouts.Source: B. Skill Learning and Composition · p. 6

C. Conversion Quality

DexStitch’s contact-aware conversion better preserves dexterous grasp appearance than the compared alternatives. Its classifier also identifies transitions and recovers skill segments in held-out human videos.

C. Conversion Quality

  • DexStitch preserves demonstrated finger configuration and renders the object over contacting fingers, unlike Masquerade’s gripper overlay and the no-occlusion variant.The qualitative comparison covers three grasps at contact: holding a sponge, lifting a snack carton, and hanging a mug.Source: C. Conversion Quality · pp. 6–7

D. Skill Classifier Evaluation

  • DexStitch achieves transition F1 0.89, compared with 0.41 for Qwen and 0.18 for InternVL.The baselines classify manipulation skills more accurately than transitions; Fig. 6 shows the confusion matrix.Source: D. Skill Classifier Evaluation · p. 7, Figure 6 · p. 7
  • The classifier recovers 242 segments against 241 in ground truth, matching the skill order in every episode but one.Segmentation is evaluated on held-out videos because compositional inference uses segment boundaries rather than isolated frame labels.Source: D. Skill Classifier Evaluation · p. 7

E. Ablation Studies

The ablations show that synthetic and teleoperated data, along with explicit transition and next-skill conditioning, contribute to sequence performance. The conclusion identifies wrap-grasp compositing and evaluation on one fixed four-skill order as limits.

E. Ablation Studies

  • 35.0% with only teleoperated data and 5.0% with only synthetic data, versus 77.5% for the full model; neither source alone suffices.The authors attribute broad skill coverage to synthetic data and grounding in robot execution to teleoperated data.Source: E. Ablation Studies · p. 7
  • 10.0% without the explicit transition skill and 7.5% without next-skill conditioning; both variants degrade most at transitions.The transition skill and next-skill signal provide context for moving toward the appropriate subsequent skill.Source: E. Ablation Studies · pp. 7–8
  • 10.0% success at a 50% teleoperation ratio rose to 77.5% at 90%, then fell to 50.0% at 100%.The authors adopt the 90% ratio, retaining a small fraction of synthetic data in the finetuning mixture.Source: E. Ablation Studies · p. 8

VI. CONCLUSION

  • DexStitch converts long-horizon human demonstrations into robot observation–action data and learns manipulation and transition skills with a skill-conditioned policy.A classifier extracts skill sequences from human video for closed-loop robot execution.Source: §VI · p. 8
  • Compositing assumes a wrap grasp, and composition evaluation covers only one fixed four-skill order.Pinch and top-down grasps fall outside the stated compositing rule; held-out orderings remain unevaluated.Source: §VI · p. 8
Loading 2610.17789v1…