Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning

Juntao Ren; Priya Sundaresan; Dorsa Sadigh; Sanjiban Choudhury; Jeannette Bohg

Source-linked AI summary

Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning

Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, Jeannette Bohg

arXiv:2501.06994v2Jan 13, 2025cs.ROcs.AIcs.LG

TL;DR

Imitation learning is limited by the labor and scale of teleoperated robot data, while human videos lack robot action labels. MT-π represents actions as image-space motion tracks shared by human hands and robot end-effectors, then reconstructs 6DoF trajectories from two views. It achieves 86.5% average success across four real-world tasks, 40% above the stated baselines, and generalizes to scenarios seen only in human videos.

Problem
Teleoperated robot demonstrations are labor-intensive, whereas scalable human videos lack the robot action labels needed for direct imitation-learning training.
Method
MT-π co-trains on human and robot demonstrations by predicting short-horizon 2D keypoint trajectories and grasp states in image space, with auxiliary embedding and domain-adaptation losses.
Results
86.5% average success across four real-world tasks, 40% higher than state-of-the-art imitation-learning approaches without human video or the proposed action space.
Takeaways & Limitations
The shared action space enables learning primarily from minutes of human video with only tens of robot demonstrations and generalization to scenarios present only in human videos.
Takeaways & Limitations
Independent per-view predictions do not explicitly enforce cross-view consistency, which can cause triangulation errors and imprecise actions; the method is also sensitive to noise in human videos.

Abstract

from arXiv · show

Teaching robots to autonomously complete everyday tasks remains a challenge. Imitation Learning (IL) is a powerful approach that imbues robots with skills via demonstrations, but is limited by the labor-intensive process of collecting teleoperated robot data. Human videos offer a scalable alternative, but it remains difficult to directly train IL policies from them due to the lack of robot action labels. To address this, we propose to represent actions as short-horizon 2D trajectories on an image. These actions, or motion tracks, capture the predicted direction of motion for either human hands or robot end-effectors. We instantiate an IL policy called Motion Track Policy (MT-pi) which receives image observations and outputs motion tracks as actions. By leveraging this unified, cross-embodiment action space, MT-pi completes tasks with high success given just minutes of human video and limited additional robot demonstrations. At test time, we predict motion tracks from two camera views, recovering 6DoF trajectories via multi-view synthesis. MT-pi achieves an average success rate of 86.5% across 4 real-world tasks, outperforming state-of-the-art IL baselines which do not leverage human data or our action space by 40%, and generalizes to scenarios seen only in human videos. Code and videos are available on our website https://portal-cornell.github.io/motion_track_policy/.

I. INTRODUCTION

Imitation learning can require costly teleoperated robot demonstrations, while scalable human videos lack robot action labels. MT-π addresses this gap with a shared image-space action representation, achieving strong performance from limited data.

State-of-the-art imitation-learning policies may require hundreds to tens of thousands of teleoperated demonstrations.Teleoperation is time-consuming, labor-intensive, and often requires practice before collection begins.
Human videos scale more easily than robot demonstrations but lack the robot action labels needed to train imitation-learning policies directly.Existing datasets provide thousands of hours of human demonstrations.
Existing hybrid approaches align representations but remain limited by robot datasets for output actions, restricting behavioral expressivity.A robot trained only on rightward drawer-closing demonstrations may fail to generalize to leftward closing.
MT-π unifies human and robot actions as short-horizon 2D image-space trajectories, requiring approximately 10 minutes of human video and a few dozen robot demonstrations.Two camera views and multi-view geometry recover 6DoF end-effector trajectories at test time.
MT-π reaches an average 86.5% success rate across four real-world tasks, 40% higher than state-of-the-art imitation-learning approaches without human video or this action space.The policy also generalizes to scenarios captured only in human videos.

A. Imitation Learning

Imitation learning maps demonstrations to robot actions but remains constrained by teleoperation data requirements. Human video offers scalable supervision, yet embodiment differences and action-label gaps complicate transfer; MT-π uses an aligned image-based action space instead.

A. Imitation Learning: Imitation learning trains agents to mimic expert behavior from demonstrations and maps sensory inputs to motor actions.
A. Imitation Learning: Large-scale teleoperation datasets can exceed 80K trajectories, but collecting data at that scale is often infeasible and performant learning remains difficult.
B. Sample-Efficient State-Action Representations: Alternative 3D representations can reduce demonstration needs but generally require robot-only RGB-D data and calibrated camera extrinsics.
B. Sample-Efficient State-Action Representations: Image-based methods often reparameterize inputs while retaining robot proprioceptive action spaces, limiting policies to actions present in teleoperated data.
C. Learning from Human Video: Human-video pretraining is scalable but pretrained representations can be brittle for task-specific use because of dataset diversity.
C. Learning from Human Video: MT-π represents human and robot actions as shared 2D image-plane trajectories, disentangling motion differences from visual differences despite morphological gaps.

III. PROBLEM FORMULATION

MT-π learns visuomotor actions from shared image-space keypoints across human and robot demonstrations. It predicts short-horizon keypoint tracks and grasp states, then triangulates tracks from two views into 6DoF robot actions.

The policy is trained mostly on human video and a small amount of robot data using shared pixel-level keypoints as image-space actions.
The dataset combines human and robot demonstrations, with at least one camera during collection and two cameras with known extrinsics at test time.
MT-π extracts keypoints for human hands and robot grippers, with grasp represented as a binary variable.
Motion tracks forecast future pixel locations of end-effector keypoints over a horizon H.
The predicted action includes grasp indicators for each future timestep.
Short-horizon motion tracks are triangulated to recover 6DoF actions executed by the robot.

IV. APPROACH

MT-π maps human and robot demonstrations into a common 2D keypoint representation and predicts future keypoint locations in image space. Co-training in this unified action space supports learning from both datasets.

MT-π maps human and robot demonstrations to common 2D manipulator keypoints and predicts their future pixel locations.The shared representation is designed for visuomotor control in image space.

A. Data Preprocessing

The preprocessing pipeline converts robot and human demonstrations into corresponding image-plane keypoints and grasp states, enabling a shared action representation.

Robot demonstrations project calibrated 3D end-effector positions into 2D image-plane keypoints.The process uses captured viewpoint images, robot end-effector positions, and known camera-to-robot extrinsics.
The robot gripper uses five keypoints—two per finger and one central point—to support grasping actions.The gripper’s open/close state is represented separately as a binary grasp variable.
Human demonstrations use RGB videos and hand-pose detection, selecting five keypoints that roughly match the robot gripper’s structure.The selected points include the wrist and two points each on the thumb and index finger.
A proximity heuristic between hand keypoints and segmented objects infers per-timestep grasp actions from human videos.The heuristic uses object masks and fingertip proximity to determine the binary grasp state.
Matching keypoint positioning and ordering creates an explicit correspondence between human-hand and robot-gripper action representations.

B. Training Pipeline

The training pipeline retargets embodiment-specific keypoints, predicts short-horizon motion tracks from image features, and aligns human and robot representations with auxiliary losses.

Keypoint Retargeting Network: A Keypoint Retargeting Network maps robot keypoints toward the spacing observed during training, reducing embodiment-specific differences.The network is trained by denoising human keypoints and then frozen for training and testing; for human keypoints, it acts as an identity map.
Motion Track Network: The motion track network combines image embeddings with current pixel-space keypoints and predicts future 2D keypoint offsets and gripper states over horizon H.It uses a Diffusion Policy objective and processes one viewpoint image at a time.
Motion Track Network: The single-viewpoint motion track network is viewpoint-pose agnostic, enabling adaptation to Internet-scale human videos from a single RGB viewpoint.
Limitation: HaMeR’s 3D hand-pose estimates can be noisy along the camera viewing direction because multiple poses may produce similar 2D projections.
Image Representation Alignment: Two auxiliary losses align human and robot visual embeddings by minimizing feature-distribution divergence and applying domain adaptation.The total auxiliary loss combines KL-divergence and domain-adaptation terms, with the domain-adaptation weight tuned by the human-to-robot demonstration proportion.

C. Action Inference

At inference, MT-π predicts motion tracks from two calibrated camera views, reconstructs 3D keypoints, and converts their motion into executable 6DoF end-effector actions.

The system concatenates retargeted keypoints with each image embedding before predicting per-view motion tracks and gripper states.
Stereo triangulation with known camera extrinsics recovers 3D positions for the predicted tracks at each timestep.
Relative rigid transformations between consecutive 3D keypoint configurations become directly executable 6DoF end-effector delta actions.The transformation consists of a rotation matrix R ∈ SO(3) and translation vector t ∈ R^3.
Consistent, unimodal human and robot demonstrations help motion tracks agree across views, improving action recovery reliability.

V. EXPERIMENTS

The experiments evaluate MT-π’s benefits from human video, its motion-track action space, and its generalization capabilities.

MT-π is evaluated on a suite of tabletop tasks to assess human-video transfer, the motion-track action space, and generalization.

A. Experimental Setup

The evaluation compares MT-π with DP and ACT under matched tabletop-task conditions, while distinguishing their training data, action spaces, and camera inputs.

Evaluation Protocol: MT-π, DP, and ACT are evaluated on tabletop manipulation tasks, with all methods trained from 25 teleoperated robot demonstrations.MT-π additionally uses cross-embodiment human data, while the baselines are restricted to robot data.
Policy Comparison: MT-π shares DP’s diffusion backbone but predicts image-based motion tracks instead of the 6DoF proprioceptive end-effector commands used by DP and ACT.Motion tracks provide a cross-embodiment action space for human and robot demonstrations.
Policy Comparison: The baselines receive additional wrist-camera observations, whereas MT-π does not because wrist cameras are typically absent from human videos.These design choices aim to attribute performance differences to training-data distribution and action space rather than other factors.

B. Key Results and Findings

MT-π outperforms robot-only imitation-learning baselines across four real-world tasks, benefits from human video when robot data are limited, and transfers motions appearing only in human demonstrations.

Cross-Task Performance: MT-π significantly outperforms DP and ACT across Fork on Plate, Fold Cloth, Serve Egg, and Put Away Socks.All methods use 25 teleoperated robot demonstrations, while MT-π also receives 10 minutes of human video.
Baseline Behavior: With only 25 robot trajectories, DP and ACT have difficulty generalizing to small changes in starting-state distributions and rollout states.Their performance improves when robot demonstrations are scaled up or when the task’s reset distribution is fully covered.
Sample Efficiency: MT-π without human demonstrations matches DP and ACT with the same robot-data quantity on Serve Egg.This supports image-space action prediction as a scalable representation even when training uses only robot data.
Sample Efficiency: MT-π matches baseline performance using 40% fewer minutes of robot demonstrations by leveraging approximately 10 minutes of human demonstrations.The comparison is made on the medium-complexity Serve Egg task.
Generalization: Human video enables MT-π to generalize drawer-closing motions to the left when robot demonstrations contain only rightward closures.DP and ACT show no success for the direction present only in human videos.
Data Distribution: On Serve Egg, MT-π achieves strong performance in the low-robot-data regime using 5–10 minutes of human video.Teleoperated collection takes nearly 5× longer than human videos: 100 robot demonstrations require approximately 65 minutes versus 70 human demonstrations in approximately 10 minutes.

VI. LIMITATIONS AND FAILURE MODES

MT-π’s main failure modes arise from independently processing camera views and from sensitivity to difficult human-video conditions. The authors identify viewpoint inconsistency, in-the-wild video noise, and grasp detection as boundaries for the current system.

VI. LIMITATIONS AND FAILURE MODES: Independent predictions across camera views can cause triangulation errors and imprecise robot actions.The policy does not explicitly enforce consistency between tracks from separate views.
VI. LIMITATIONS AND FAILURE MODES: The motion-centric method remains sensitive to drastic viewpoint shifts, egocentric motion, rapid temporal changes, and hand occlusion in human videos.These conditions currently limit handling of truly in-the-wild videos.
VI. LIMITATIONS AND FAILURE MODES: Human grasp detection remains an open challenge because the current pipeline uses a heuristic approach to infer hand–object contact.The approach leverages foundation models for contact inference.
VI. LIMITATIONS AND FAILURE MODES: Future work targets viewpoint-consistency losses, object-centric representations, truly in-the-wild videos, and more complex manipulation tasks.These directions are presented as extensions rather than capabilities established by the current system.