XIRL: Cross-embodiment Inverse Reinforcement Learning

Kevin Zakka; Andy Zeng; Pete Florence; Jonathan Tompson; Jeannette Bohg; Debidatta Dwibedi

Source-linked AI summary

XIRL: Cross-embodiment Inverse Reinforcement Learning

Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, Debidatta Dwibedi

arXiv:2106.03911v3Jun 7, 2021cs.ROcs.AIcs.CVcs.LG

TL;DR

Cross-embodiment imitation must learn policies from videos whose demonstrators and learners differ substantially in embodiment, without relying on difficult frame correspondences. XIRL uses temporal cycle-consistency to learn task-progress embeddings and derives rewards from distances to goal observations. The resulting rewards support reinforcement-learning transfer to unseen embodiments and improve sample efficiency in real-world human-to-simulated-robot transfer.

Problem
Third-person demonstrations often differ from the learner in embodiment and strategy, making frame-to-frame correspondences and action labels difficult to obtain.
Method
XIRL self-supervises an embodiment-invariant visual encoder with temporal cycle-consistency, then defines dense rewards from embedding-space distances to goal observations.
Results
XIRL rewards generalize to unseen embodiments and improve sample efficiency for transferring real-world human demonstrations to a simulated robot.
Takeaways & Limitations
Video-only, self-supervised reward learning can support cross-embodiment reinforcement-learning imitation without manually paired demonstration frames.
Takeaways & Limitations
The demonstration dataset contains observation data only and does not include the agents’ actions.

Abstract

from arXiv · show

We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -- shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings typically required alignment with a reference trajectory, which may be difficult to acquire under stark embodiment differences. We show empirically that if the embeddings are aware of task progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. Additionally, when transferring real-world human demonstrations to a simulated robot, we find that XIRL is more sample efficient than current best methods. Qualitative results, code, and datasets are available at https://x-irl.github.io

1 Introduction

XIRL addresses video imitation when demonstrations and learners differ in embodiment, making direct correspondence difficult. It learns embodiment-invariant rewards from demonstrations and uses them to train policies, including on unseen embodiments.

Motivation: Third-person demonstrations could let robots learn from abundant tutorial videos, but viewpoint, environment, expert, tool, strategy, and embodiment differences make policy learning difficult.The embodiment gap between human demonstrators and robot hardware is especially challenging.
Motivation: Different embodiments may complete the same task through distinct state-action trajectories, making labeled frame correspondences difficult to obtain.For example, a human may scoop several pens at once while a gripper picks them individually.
Approach: XIRL learns task-specific, embodiment-invariant rewards from expert videos using temporal cycle-consistency and uses them for downstream reinforcement learning.The method uses an encoder trained with TCC to define dense rewards through distances in the learned embedding space.
Results: Experiments across four simulated embodiments show that learned rewards generalize to new embodiments and can sometimes exceed ground-truth sparse rewards in sample efficiency.The paper also evaluates transfer from real-world human demonstrations to a simulated Sawyer arm.
Resources: XIRL introduces X-MAGICAL, a cross-embodiment benchmark, and X-REAL, a real-world dataset containing manipulation demonstrations from nine embodiments.The contributions also include comparisons on X-MAGICAL and a human-to-robot transfer benchmark.

2 Related Work

Prior work learns from observation-only demonstrations through action inference, learned rewards, or domain adaptation, but often assumes embodiment or alignment conditions that limit transfer. XIRL instead targets reward learning across different policy embodiments without manually labeled dense correspondences.

Observation-only imitation: Traditional imitation learning uses expert state-action trajectories, but third-person demonstrations often lack ground-truth actions.This motivates methods that infer actions or learn directly from observations.
Observation-only imitation: Action-inference and forward-prediction methods address observation-only learning but may not transfer skills across embodiments or exploit multiple embodiments for generalization.Their limitations concern either different policy embodiments or generalization to unseen configurations.
Learned reward functions: Latent-feature reward methods use distances between policy and expert observations, while time-index heuristics constrain tasks with differing speeds or non-monotonic progress.These approaches can rely on viewpoint-invariant or self-supervised visual features but still use weak temporal correspondence.
Learned reward functions: Other reward-learning work avoids time alignment but assumes the expert and learned policy share the same domain and embodiment.XIRL relaxes that shared-domain and shared-embodiment assumption.
Domain adaptation: Domain-adaptation methods translate expert observations between policy domains before constructing rewards for model-based or model-free reinforcement learning.Examples include pixel-level image translation and generative models.
Reinforcement learning with demonstrations: RLV combines labeled expert-policy state pairs with adversarial training and an inverse dynamics model, whereas XIRL avoids human-labeled dense state correspondences through self-supervision.The comparison concerns human labeling requirements for cross-embodiment transfer.

3 Approach

XIRL learns embodiment-invariant task representations from observation-only demonstrations using temporal cycle-consistency, then converts those representations into dense rewards for reinforcement learning. The reward is the scaled negative embedding distance to a goal representation, allowing one reward to support agents with different trajectories and embodiments.

Problem Formulation: XIRL extracts an agent-invariant task definition from videos of agents that may differ in shape, dynamics, end-effectors, and strategies.The framework targets observation-only demonstrations and does not require action trajectories.
Representation Learning: TCC trains an image encoder to capture semantic temporal correspondences across videos of the same task despite different execution speeds and embodiments.The method cycles between frame sequences and minimizes the error between the starting frame index and the retrieved cycle-back index.
Reward Function: The learned reward is the scaled negative distance between the current state embedding and the mean embedding of demonstration-ending frames.The scale parameter keeps distances in a range suitable for reinforcement learning.
Reward Function: Because the reward is dense and encodes task progress, it avoids reference-trajectory correspondence and supports trajectories with embodiment-specific lengths.Different strategies can be mapped to a common notion of task progress in the learned embedding space.
Reinforcement Learning: The frozen encoder and learned reward define an MDP that can be used for any agent, including embodiments unseen during encoder training.The formulation can also augment sparse task-success rewards during policy training.

4 Experimental Setup

The experiments use X-MAGICAL, a simulated cross-embodiment sweeping benchmark, and compare XIRL with alternative reward functions under SAC. The setup varies agent shapes and end-effectors, uses observation-only demonstrations, and represents states with stacked temporal features.

X-MAGICAL Benchmark: X-MAGICAL is a PyMunk-based benchmark in which agents push three objects into a predefined zone in a long-horizon sweeping task.Environment reward is the fraction of debris swept into the zone at episode end.
X-MAGICAL Benchmark: The benchmark varies embodiment through stick agents of three lengths and a circular gripper agent with two actuated arms.Agents use rotation and forward/backward translation actions, while the gripper also has finger actuation.
State Representation: Each agent receives a 16-dimensional state vector describing its pose and each debris item’s position and distances, then three frames are stacked into 48 dimensions.The stacked representation encodes temporal and velocity information.
Evaluation: The experiments compare XIRL with ImageNet, goal-classifier, and other learned reward baselines while using SAC for policy learning.The comparison is conducted on the X-MAGICAL sweeping task.
Demonstrations: Demonstrations contain 1000 successful sweeping trajectories per agent, with debris initialized at random positions.The dataset contains observation-only, agent-specific demonstrations and highlights differing state-visitation patterns.

5 Experiments

XIRL is evaluated on same-embodiment, unseen-embodiment, real-world-demo, and qualitative reward-learning settings. Across these experiments, learned rewards support reinforcement learning, generalize across embodiments, and improve sample efficiency.

5.1 Same-Embodiment Demonstrations: XIRL is more sample-efficient than other learned reward baselines in same-embodiment reinforcement learning.The authors attribute this to TCC embeddings encoding task progress and guiding exploration toward objects and goal zones.
5.2 Cross-Embodiment Demonstrations: XIRL generalizes significantly better to unseen agents than TCN, LIFS, and goal-frame-classifier reward baselines.Each experiment holds out one embodiment and trains the encoder on demonstrations from the other three.
5.3 Real-World Cross-Embodiment Demonstrations: XIRL improves sample efficiency over RLV and the environment reward when learning in simulation from real-world human demonstrations.The real-only encoder uses human videos without human-labeled paired-frame correspondences.
5.4 Qualitative Reward Comparison: XIRL rewards correlate highly with ground-truth rewards on successful and unsuccessful demonstrations across simulated and real-world tasks.The visualizations cover Sweeping, State Pusher, and Drawer Opening.
5.4 Qualitative Reward Comparison: For a failed drawer-collision trajectory, XIRL gives partial reward for moving toward the drawer while the sparse environment reward remains zero.This illustrates how the learned reward can represent partial task progress.

6 Conclusion

The paper presents XIRL as a self-supervised framework for learning embodiment-invariant visual rewards from expert videos and using them for reinforcement learning. It reports improved sample efficiency and generalization, while noting that real-robot policy learning remains unshown.

Conclusion: XIRL learns vision-based rewards from videos of experts with different embodiments using TCC-supervised visual representations.Rewards are generated from distances between current-state and goal-observation embeddings.
Conclusion: The framework enables unseen agents to learn demonstrated tasks without manually paired video frames.The same reward function can support agents whose embodiments were absent during encoder training.
Conclusion: XIRL is more sample-efficient than multiple baselines, including TCN, LIFS, and RLV.The conclusion summarizes results across simulated and real-world videos.
Limitations: The experiments demonstrate policy learning in simulation, but not policy learning on a real robot.The authors identify real-robot policy learning as future work.
Related Work: Unlike time-indexed correspondence methods, XIRL avoids assuming demonstrations execute at the same speed or follow strictly monotonic progress.Prior time-index heuristics are limited when task speeds differ or subtask ordering is ambiguous.

B.2 Detailed Task and Embodiment Description

The Sweeping benchmark uses randomized debris arrangements and embodiment-specific horizons, with image observations for reward generation and state observations for policy learning. Demonstrations are collected from successful oracle-policy rollouts.

Task Description: The Sweeping task requires pushing three debris blocks into a pink goal zone.The agent starts below the debris, whose positions are randomized at each reset.
Task Description: The environment uses an embodiment-specific horizon, including H =50 time steps for the longstick agent.
Environment Reward: The environment reward measures the fraction of debris inside the goal zone.The reward expression sums indicators for debris membership in the goal zone and normalizes by three.
Demonstrations: A film-strip visualization shows each embodiment solving Sweeping with environment reward plotted over time.For this visualization, agents are manually teleoperated and the horizon limit is disabled.
Demonstrations: The dataset contains 1000 successful demonstrations per embodiment collected from SAC oracle-policy rollouts.Potentially unsuccessful demonstrations are discarded during collection.

Appendix C X-REAL: A Real-World Cross-Embodiment Dataset

X-REAL evaluates XIRL on real-world videos of diverse end-effectors performing a multi-step pen-transfer task. The learned rewards distinguish successful from unsuccessful trajectories without object, end-effector, or state annotations.

Dataset: X-REAL contains 93 videos of different embodiments transferring five pens to two cups consecutively.The embodiments include a human hand and six tools with different end-effectors.
Dataset: The X-REAL embodiments vary visually and differ substantially in how long and how they solve the task.The task requires lifting pens into one cup and then moving them to a separate cup.
Data Collection: Demonstrations are recorded with a tripod-mounted GoPro Hero8 at 1920×1080 resolution and 30 frames per second.
Reward Learning: XIRL handles X-REAL’s visual complexity without annotations of end-effectors, objects, or their states.The encoder is trained on all training-set embodiments and evaluated on validation demonstrations.
Reward Results: Successful and unsuccessful trajectories receive clearly different learned rewards for both the RMS Grabber Reacher and 1 Hand 5 Fingers.Leaving pens in the first cup produces roughly half the reward of successful completion for the hand embodiment.

D.1.2 Data Augmentation & Preprocessing

The appendix lists the augmentation and preprocessing pipeline used for representation learning, including spatial, color, grayscale, blur, and normalization transformations applied to video frames.

Representation learning uses RandomResizedCrop, ColorJitter, ToGray, GaussianBlur, and Normalize transformations.The cited table identifies representation-learning hyperparameters, while the listed passages specify the transformations.
The same randomly sampled transformations are applied to all sampled frames from one video, while each minibatch frame stack receives independent samples.The composed order is N◦B◦G◦J◦C.

D.1.3 Training and Evaluation

The appendix describes representation diagnostics, policy-learning evaluation, and implementation settings for training and assessing XIRL across environments and embodiments.

Representations are optimized with ADAM using β1=0.99, β2=0.999, and weight decay of 10^-5.Downstream policy-learning performance, quantitative metrics, and qualitative results are used to evaluate the representations.
Kendall’s Tau measures temporal alignment between two sequences on a scale from −1 to 1.
Nearest-neighbor alignment videos test whether embedding-space neighbors preserve task progress across embodiments.A demonstration is selected as a reference, and a test video is aligned to it using nearest neighbors.
The SAC implementation uses clipped double Q-learning with three-layer ReLU MLP actor and critic networks of hidden size 1024.The actor is tanh-diagonal-Gaussian, and both networks use orthogonal weight initialization with zero biases.
Policies stack three consecutive state vectors, producing flattened inputs in R48 for X-MAGICAL and R15 for Puck Pushing.
Training begins with 5000 uniformly random-policy observations, then performs one gradient update per new environment observation.Evaluation averages final success over 50 episodes.

Appendix E Additional Experiments

Additional experiments compare XIRL with ground-truth rewards and SimCLR baselines, showing competitive sample efficiency and stronger downstream performance in the reported cross-embodiment evaluations.

E.1 XIRL vs. Ground-truth Environment Reward: XIRL is at least as sample-efficient as the ground-truth environment reward in both same- and cross-embodiment settings.The authors attribute this to denser reward information from encoded task progress compared with the sparse environment reward.
E.2 XIRL vs. SimCLR: XIRL is compared with SimCLR trained on X-MAGICAL and with an ImageNet-pretrained SimCLR ResNet18.The comparison reports longstick and mediumstick X-MAGICAL embodiments.
E.2 XIRL vs. SimCLR: XIRL performs significantly better than both SimCLR baselines for the reported mediumstick comparison.The X-MAGICAL-only SimCLR objective performs poorly, while ImageNet-pretrained SimCLR does better on the easier longstick embodiment but poorly on mediumstick.
E.2 XIRL vs. SimCLR: The results indicate that cross-embodiment demonstration pretraining and the TCC loss inductive bias are both needed for strong downstream reinforcement-learning performance.

Appendix F Qualitative Results

Qualitative analyses visualize XIRL’s embedding structure and alignment behavior, emphasizing overlap across demonstrations and distances that track task progress.

F.1 t-SNE Visualizations: The t-SNE analysis compares XIRL and Goal Classifier embeddings for four shortstick demonstrations.
F.1 t-SNE Visualizations: Different demonstration trajectories overlap and align well in XIRL’s embedding space, unlike the less structured Goal Classifier space.
F.1 t-SNE Visualizations: Distances to the goal in the XIRL projection correlate with task progress.The goal is shown in the top-left corner of the XIRL projection.
Supplementary videos provide t-SNE visualizations, nearest-neighbor alignments, policy rollouts, and interactive learned-reward visualizations.The materials cover Sweeping, Puck Pushing, X-MAGICAL, and X-REAL examples.
XIRL encoder hyperparameters are mostly shared across environments, with embedding dimension and sampled-frame count as the main variations.The appendix covers Sweeping, Puck Pushing, and X-REAL representation and policy learning.