Source-linked AI summary
XSkill: Cross Embodiment Skill Discovery
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, Shuran Song
TL;DR
Human videos offer a way to specify robot behavior, but embodiment differences and unobserved action parameters make reusable skill extraction difficult. XSkill learns shared skill prototypes from unlabeled human and robot videos, transfers them to robot actions, and composes them from a human prompt video; it reports stronger performance than baselines on unseen cross-embodiment tasks in simulation and the real world.
Problem
Extracting reusable robot manipulation skills directly from unstructured human videos is difficult because of embodiment differences and unobserved action parameters.
Source: §1 · p. 1Method
XSkill learns shared skill prototypes from unlabeled human and robot videos, transfers them with a skill-conditioned diffusion policy, and composes them for tasks specified by a human prompt video.
Source: §1 · p. 2Results
On unseen tasks with cross-embodiment prompts, XSkill outperforms all baselines, achieving 70.2% success in simulation and 60% in real-world evaluations.
Source: §4.1 · p. 7Takeaways & Limitations
XSkill demonstrates that cross-embodiment skill prototypes can support skill transfer and composition for unseen tasks, including from non-expert human demonstrations.
Source: §5 · p. 8The current benchmark uses videos from the same laboratory camera setup, so applicability to more diverse camera setups and environments remains to be investigated.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework. The benchmark, code, and qualitative results are on https://xskill.cs.columbia.edu/
1 Introduction
XSkill frames imitation from human demonstrations as discovering reusable skills, transferring them to robots, and composing them for new tasks. It learns shared human–robot skill prototypes, transfers them through a skill-conditioned policy, and uses a human prompt video for one-shot task generalization.
1 Introduction
- XSkill learns a shared embedding space in which video clips with similar action effects are closer, using learnable prototypes to align human and robot skills.The prototypes serve as representative anchors in the continuous embedding space.Source: §1 · p. 2
- At inference, a human prompt video defines a new task, and the skill alignment transformer aligns its skills with robot observations despite embodiment differences and unexpected execution failures.Source: §1 · p. 2
- Compared with direct imitation learning, XSkill decomposes long-horizon tasks into reusable low-level visuomotor skills that can be composed for new tasks.The introduction distinguishes this from single-embodiment skill discovery [8] [9], which relies on on-robot demonstrations.Source: §1 · p. 2
- XSkill discovers shared skill prototypes from unlabeled human and robot videos, then uses a skill-conditioned diffusion policy and a skill alignment transformer to transfer and compose skills.The framework targets unseen tasks specified by a single human demonstration.Source: §1 · p. 2
2 Related Work
Prior robot skill discovery often depends on environment interaction or robot demonstrations, while much imitation and human-video learning relies on task labels, reinforcement learning, or task pairing. XSkill instead studies cross-embodiment discovery and one-shot imitation from human video without training-time task labels or embodiment correspondence.
2 Related Work
- Many robot skill-discovery methods require environment interaction, and demonstration-based methods commonly rely on physical state or robot action trajectories; XSkill studies discovery across embodiments.BUDS [25] is cited as a raw-RGB exception among prior demonstration-based approaches.Source: §2 · p. 2
- Prior imitation methods often use same-embodiment demonstrations or require task labels for cross-embodiment training, whereas XSkill needs neither task labels nor embodiment correspondence.Source: §2 · pp. 2–3
- Many human-video approaches construct rewards and use reinforcement learning, while XSkill focuses on one-shot imitation without RL in the loop or task pairing during training.The paper also contrasts its representation focus with MimicPlay [62], which aims to minimize robot demonstration collection.Source: §2 · p. 3
3 Approach
XSkill discovers shared human–robot skill representations, transfers them into robot actions, and composes them from a single human prompt video to address unseen tasks. Its pipeline uses unaligned human and robot videos for discovery, robot teleoperation data for policy learning, and a human prompt for composition.
3.1 Discover: Learning Shared Skill Prototypes
- XSkill learns shared skill prototypes through soft-assignment clustering, with Sinkhorn-Knopp regularization ensuring all prototypes are used within single-embodiment batches.This design is intended to prevent different embodiments from mapping to different prototypes.Source: §3.1 · pp. 3–4
- Temporal contrastive learning makes prototype probabilities similar for nearby video clips and dissimilar for clips farther apart in time.A temporal encoder extracts representations from uniformly sampled video clips.Source: §3.1 · p. 4
3.2 Transfer: Skill conditioned imitation learning
- The skill-conditioned visuomotor policy is trained on robot teleoperation data to map robot state and skill representation to an action sequence.Robot state includes proprioception and visual observation.Source: §3.2 · pp. 4–5
3.3 Compose: Performing unseen task from one-shot human prompt video
- For an unseen task, XSkill encodes a human prompt video as a skill sequence and executes its skills through the learned policy.The sequence serves as a task execution plan.Source: §3 · p. 3, §3.3 · p. 5
- The Skill Alignment Transformer uses the robot’s current observation to select the next skill, helping address execution-speed differences and unexpected failures.The transformer compares the current state with the prompt-derived skill sequence to infer what remains to be executed.Source: §3.3 · pp. 5–6
4 Evaluation
Evaluation tests cross-embodiment task composition in simulation and a physical kitchen. XSkill outperforms baselines on unseen tasks, while performance remains constrained by robot-data diversity and camera-setting scope.
4 Evaluation
- Evaluation pairs simulated Franka Kitchen with a four-subtask physical kitchen, testing unseen compositions in prompt order against goal-conditioned and XSkill ablations.The physical benchmark uses human and teleoperation demonstrations; episodes end if the robot executes an undemonstrated sub-task.Source: §4 · p. 6
4.1 Key Findings
- Human and robot skills for the same objects align, including transitions, despite humans executing 2x faster; cross-embodiment prompts reduce seen-task performance by around 5%.The comparison is against same-embodiment prompts on seen tasks.Source: Figure 5 · p. 7, §4.1 · p. 7
- XSkill achieves 70.2% success in simulation and 60% in the real world on unseen tasks with cross-embodiment prompts, outperforming all baselines.The simulation and real-world result tables report these benchmark results.Source: Table 1 · p. 7, Table 2 · p. 7, §4.1 · p. 7
4.2 Limitation and Future Work
- Generalization remains limited by robot-data diversity, including missing transition dynamics, while the current benchmark uses a single laboratory camera setup.The authors identify diverse camera setups and environments as areas for future investigation.Source: §4.1 · p. 7, §4.2 · p. 8
4.1 Key Findings
- XSkill outperforms nearest-neighbor composition by more than 50% with cross-embodiment prompts, while SAT adjusts skills to task progress when speed differences disrupt visual matching.The authors attribute nearest-neighbor instability to morphology-based visual distraction and prompt videos that omit states captured during slower robot execution.Source: §4.1 · p. 8
5 Conclusion
XSkill extracts reusable manipulation skills from human and robot videos for transfer to robots and composition into new tasks. Experiments show improved performance over direct behavior cloning, especially on complex long-horizon tasks.
5 Conclusion
- XSkill extracts common manipulation skills from unstructured human and robot videos for transfer to robots and composition into new tasks.The framework can use non-expert human demonstrations to define new tasks, making it more cost-effective and scalable.Source: §5 · p. 8
- Experiments in simulation and the real world show XSkill improves on direct behavior cloning, especially for complex long-horizon tasks.Source: §5 · p. 8
A.1 Generalization to unseen transition
In simulation, XSkill generalizes to tasks with unseen skill compositions and transitions. Success rates are higher with same-speed cross-embodiment prompts than with prompts at ×1.5 speed across both transition-removal levels.
A.1 Generalization to unseen transition
- The evaluation tests generalization by withholding inference-task transitions at two difficulty levels: 25% for Level 1 and 50% for Level 2.The paper's main simulation experiment used the 25%-removed setting.Source: §A.1 · p. 13
- XSkill achieved success rates of 90.7% and 84.4% on Level 1 and Level 2 tasks, respectively, with same-speed cross-embodiment prompts; at ×1.5 speed, rates were 71.8% and 67.5%.Level 2 tasks lacked 50% of inference task transitions from the training data.Source: §A.1 · pp. 13–14
- Larger prototype counts of 256 or 512 significantly outperformed smaller counts on the harder Level 2 tasks, while choices were comparable on Level 1.The authors hypothesize that more prototypes provide finer representation granularity that facilitates interpolation.Source: §A.1 · p. 14
A.2.1 Number of skill prototypes
The prototype-count ablation finds that increasing K to 256 or 512 does not degrade performance, whereas reducing K harms it. The authors attribute robustness partly to the policy using continuous skill representations before prototype projection.
A.2.1 Number of skill prototypes
- Increasing the number of prototypes to 256 or 512 did not degrade performance, while reducing K adversely affected it.Source: §A.2.1 · p. 14, Table A2 · p. 14
- The authors hypothesize that small K limits representation capacity, potentially forcing distinct skills toward the same prototype and diminishing manipulation performance.Source: §A.2.1 · p. 14
- Performance is not significantly affected by K because the policy and skill-alignment transformer use continuous skill representations before prototype projection.Fine-tuning may still be needed: the authors note that seven Franka Kitchen subtasks may require larger K, while four real-world kitchen subtasks may suffice with K = 32.Source: §A.2.1 · p. 14
A.2.2 Time contrastive loss
The time-contrastive-loss ablation reports a significant performance drop when the loss is removed, including under same-embodiment evaluation. The authors say the loss helps representations capture temporal skill effects and benefits downstream manipulation.
A.2.2 Time contrastive loss
- Removing time contrastive loss caused a significant performance drop compared with XSkill, even under the same-embodiment setting.Source: §A.2.2 · p. 14, Table A3 · p. 15
- The authors state that time contrastive loss helps the representation capture temporal effects of skills, benefiting downstream manipulation tasks.Source: §A.2.2 · p. 14
A.3 Additional Experiment Results
Additional results link simulation generalization on unseen transitions to sufficient, diverse data, while real-world Drawer generalization is constrained by data diversity.
A.3 Additional Experiment Results
- Table A4 reports real-world per-task results using cross-embodiment prompts; the stated generalization bottleneck is diversity in robot teleoperation data.The supplied passages do not give Table A4 cell values.Source: §A.3 · p. 15, Table A4 · p. 15
- Sufficient, diverse data support generalization to previously unseen skill transitions in simulation.The real-world Drawer data lack multimodal properties, and XSkill struggles with unseen transitions involving that sub-task.Source: §A.3 · p. 15
- Real-world Drawer data lack multimodal properties, and XSkill struggles with unseen transitions involving Drawer.This is the stated boundary on extending generalization to transitions involving that sub-task.Source: §A.3 · p. 15
A.4 Implementation Details
The appendix specifies pseudocode and implementation choices for discovery, transfer, and composition, then describes the data sources and simulation and real-world setups.
A.4 Implementation Details
- Discovery updates the temporal encoder and prototypes; transfer trains SAT and policy with alignment and behavior-cloning losses; composition predicts skills and actions until success or episode end.The pseudocode lists prototype and temporal-contrastive discovery losses, then combines SAT and behavior-cloning losses for transfer.Source: §A.4 · pp. 15–16
A.4.1 Sinkhorn-Knopp Algorithm
- Sinkhorn projects skill representations onto prototypes and produces target assignments as columns of a doubly stochastic matrix.The objective retains projection similarity while enforcing a specified entropy level; three Sinkhorn iterations were sufficient in the experiments.Source: §A.4.1 · p. 16
A.4.2 Temporal Skill Encoder & Prototypes Layer
- The temporal encoder uses a 3-layer CNN and an 8-layer, 4-head transformer; the prototype layer is a normalized, bias-free linear layer.Its weights are frozen for the first three training iterations, and the TCN loss uses unnormalized prototype probabilities because softmax saturation destabilizes training.Source: §A.4.2 · p. 17
A.4.3 Skill Alignment Transformer
- SAT uniformly samples a fixed number of skill representations to retain timing information and accommodate different human and robot skill-list lengths.The appendix sets NSAT to 100 in simulation and 200 for real-world trajectories.Source: §A.4.3 · pp. 17–18
A.4.4 Diffusion Policy
- The diffusion policy uses the original Chi et al. [35] codebase and the same configuration in simulation and the real-world environment.The appendix refers readers to that paper for further details.Source: §A.4.4 · p. 18
A.5 Environment&Data Collections
- The data collection describes human demonstrations, robot teleoperation trajectories, and single human prompt videos as distinct sources.The trajectories include visual observations, and robot teleoperation data also include proprioception and end-effector actions.Source: §A.5 · pp. 18–19
A.6 Simulation
- The simulation dataset rerenders 600 Franka demonstrations with a visually distinct sphere agent and evaluates unseen combinations of four subtasks.Each trajectory completes four subtasks in randomized order; the tasks include opening the microwave, moving the kettle, switching the light, and sliding the cabinet.Source: §A.6 · p. 19
A.7 Realworld
- The real-world UR5 dataset covers four subtasks, with seven training tasks and 25 trajectories per task; inference includes unseen three- and four-subtask tasks.The four subtasks are opening the oven, grasping cloth, closing the drawer, and turning on the light.Source: §A.7 · p. 19