PaperlayerView exact revision on arXiv

Source-linked AI summary

Human2Robot: Learning Robot Actions from Paired Human-Robot Videos

Sicheng Xie, Haidong Cao, Zejia Weng, Zhen Xing, Haoran Chen, Shiwei Shen, Jiaqi Leng, Zuxuan Wu, Yu-Gang Jiang

arXiv:2502.16587v4cs.RO

TL;DR

Existing human-robot learning methods often rely on coarse video alignment, limiting access to fine-grained dynamics needed for generalization. Human2Robot introduces synchronized H&R data and a video-prediction-based framework whose representations guide action decoding. It achieves strong seen-task performance and one-shot generalization across novel task conditions, while its data collection remains constrained by the human-hand–robot-gripper embodiment gap.

  • Problem

    Coarsely aligned human-robot videos constrain models to global or task-level features instead of the fine-grained dynamics required for complex manipulation and generalization.

    Source: §1 · pp. 1–2

  • Method

    H&R provides 2,600 synchronized human-robot episodes, while Human2Robot generates robot videos from human inputs and uses Video Prediction Model representations to guide a decoupled action decoder.

    Source: §1 · p. 2, §4.1 · p. 3

  • Results

    Human2Robot performs well on seen tasks and demonstrates one-shot generalization to novel positions, appearances, instances, backgrounds, and task types.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Fine-grained paired videos and video-prediction representations support human-video-conditioned robot control across seen and unseen task conditions.

    Source: §1 · p. 2, §6 · p. 7

    The embodiment gap between human hands and robot grippers remains unresolved, so data collection focuses on pick-and-place tasks and leaves harder tasks such as screwing for future work.

    Source: §3.1 · p. 3, §5.1 · p. 5

Abstract

from arXiv · show

Distilling knowledge from human demonstrations is a promising way for robots to learn and act. Existing methods, which often rely on coarsely-aligned video pairs, are typically constrained to learning global or task-level features. As a result, they tend to neglect the fine-grained frame-level dynamics required for complex manipulation and generalization to novel tasks. We posit that this limitation stems from a vicious circle of inadequate datasets and the methods they inspire. To break this cycle, we propose a paradigm shift that treats fine-grained human-robot alignment as a conditional video generation problem. To this end, we first introduce H&R, a novel third-person dataset containing 2,600 episodes of precisely synchronized human and robot motions, collected using a VR teleoperation system. We then present Human2Robot, a framework designed to leverage this data. Human2Robot employs a Video Prediction Model to learn a rich and implicit representation of robot dynamics by generating robot videos from human input, which in turn guides a decoupled action decoder. Our real-world experiments demonstrate that this approach not only achieves high performance on seen tasks but also exhibits significant one-shot generalization to novel positions, objects, instances, and even new task categories.

1 Introduction

Existing human-video-conditioned robot policies often learn only coarse task representations because paired videos lack fine-grained alignment. Human2Robot addresses this gap with synchronized H&R data and video generation that learns robot dynamics for action decoding and generalization.

Motivation

  • Coarsely aligned human-robot videos limit existing methods to global or high-level task features, missing frame-level dynamics needed for complex manipulation and generalization.The paper describes this as a dataset-method cycle that motivates changes in both learning methodology and data curation.Source: §1 · pp. 1–2

Approach

  • Human2Robot treats fine-grained human-robot alignment as conditional video generation, predicting robot motion frame by frame from human demonstrations.The generative process is intended to internalize how a task is performed rather than only what task is performed.Source: §1 · p. 2

Data

  • H&R provides 2,600 precisely synchronized human-hand and robot-arm episodes collected through VR teleoperation across basic and long-horizon tasks.The dataset is designed to supply the dense paired supervision required for fine-grained alignment.Source: §1 · p. 2

Approach

  • The framework trains a Video Prediction Model to translate human videos into robot videos, then conditions a decoupled action decoder on the resulting predictive representations.This two-stage design uses implicitly learned robot-dynamics features to guide policy learning.Source: §1 · p. 2

Results

  • Human2Robot performs well on seen tasks and demonstrates one-shot generalization to novel positions, appearances, instances, backgrounds, and task types.The introduction also reports a KNN-enhanced variant for executing previously seen tasks without a human video at test time.Source: §1 · p. 2

2 Related Work

Prior work uses human videos, paired representations, and diffusion models for robot learning, but the paper positions Human2Robot as combining paired data with diffusion-based alignment for stronger generalization.

Learning from Human Videos

  • Human-video learning methods often require strong prior knowledge or struggle to transfer to robots, while human-video-conditioned alignment methods remain inefficient and limited in generalization.The paper contrasts these approaches with its use of paired data and diffusion models.Source: §2 · p. 2

Diffusion Models for Video Generation

  • Diffusion models have advanced video generation, visual and motion transfer, and visual understanding, motivating their use for human-robot alignment and robot-policy learning.The paper specifically uses diffusion features to guide final policy learning.Source: §2 · pp. 2–3

3 H&R Dataset

The H&R dataset uses coordinate-aligned VR teleoperation to collect paired human and robot videos, while remaining limited by the embodiment gap between hands and grippers.

3.1 Coordinate Alignment for Teleoperation

  • Coordinate mismatch can make human-hand and robot-arm movements appear visually inconsistent, while the embodiment gap prevents teleoperation from supporting some tasks such as screwing.These are the two core issues identified for collecting aligned videos.Source: §3.1 · p. 3
  • Three corresponding real-world points establish a shared coordinate system and consistent scale, matching the robot arm’s movement range to the human hand’s.The alignment procedure addresses the coordinate-system mismatch rather than the embodiment gap.Source: §3.1 · p. 3
  • The embodiment gap remains unresolved, so the dataset focuses on pick-and-place data and evaluates generalization after training on relatively simple tasks.The authors leave this teleoperation limitation as future work.Source: §3.1 · p. 3

3.2 Statistics of H&R Dataset

  • H&R contains 2,600 episodes spanning four basic task types and six long-horizon tasks, with 300–600 frames per episode.It is presented as the first video dataset with perfectly aligned human and robot videos.Source: §3.2 · p. 3

4 HUMAN2ROBOT

HUMAN2ROBOT learns robot actions from human videos through a two-stage pipeline: video prediction first captures robot motion, then action decoding converts those features into actions. The framework also supports task conditioning by retrieving a matching human demonstration when no video is explicitly provided.

4 HUMAN2ROBOT

  • HUMAN2ROBOT uses a two-stage pipeline in which a Video Prediction Model generates robot videos before an action decoder predicts robot actions.The VPM is trained first, then frozen while the action decoder learns from its motion features.Source: §4 · p. 3, Figure 3 · p. 4

4.1 Stage1 : Video Prediction Model (VPM)

  • The VPM combines robot-arm spatial features, human behavior cues, and temporal modeling to predict high-fidelity future robot frames.The Spatial UNet extracts robot-arm features, while the Spatial-Temporal UNet models temporal dynamics using behavior-extractor features and noise latents.Source: §4.1 · p. 3

4.2 Stage2 : Action Decoding.

  • The action decoder uses first-step denoising features from the frozen VPM as action priors, aggregates them with a Video Former, and conditions a diffusion policy through cross-attention.The first upsampling-layer output is used because a single denoising step already contains action and position information.Source: §4.2 · pp. 4–5

4.3 KNN + HUMAN2ROBOT.

  • The KNN+HUMAN2ROBOT variant retrieves a conditioning human video by matching current-scene features to training episodes using DINOv2 and CLIP.The closest and most frequent matching episode supplies the human demonstration used for task execution.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate HUMAN2ROBOT on seen and varied unseen tasks, comparing video-conditioned learning with baselines and ablations. Results show strong basic-task performance, broad generalization, and the importance of video-generation pretraining.

5.1 Experimental Setups

  • Experiments train on simple pick-and-place tasks and test variations in appearance, position, instance, background, task combinations, and entirely new tasks.The setup evaluates both seen tasks and multiple levels of generalization, using 20 trials per task in the reported tables.Source: §5.1 · p. 5, Figure 4 · p. 6, Table 2 · p. 6

5.2 Main results

  • HUMAN2ROBOT achieves the highest success rate across all basic tasks, leading VPP by 10–20 percentage points and outperforming other baselines.XSkill success rates are 30–50% lower, while VPP reaches closer performance but remains behind the proposed method.Source: §5.2 · pp. 5–6, Table 1 · p. 6
  • HUMAN2ROBOT with KNN outperforms all other baselines without direct demonstrations, although success decreases by 10–20% relative to HUMAN2ROBOT.The results support strong performance on seen tasks even when demonstrations are unavailable at inference.Source: §5.2 · p. 6

5.3 Ablation Study

  • Directly conditioning the action decoder on human videos yields jittery executions and a 23% average success rate, whereas VPM features provide a more reliable motion prior.The human-video ablation frequently misses grasps, motivating the intermediate video-prediction representation.Source: §5.3 · p. 6
  • Removing video-generation pretraining reduces success to 20% on push-and-pull tasks and 10% on pick-and-place, showing that pretraining is essential for effective task completion.The ablation freezes the initialized VPM and trains only the downstream action decoder.Source: §5.3 · pp. 6–7

5.4 Generalization

  • HUMAN2ROBOT generalizes to appearance, position, instance, background, combined, and brand-new tasks, outperforming baselines across these settings.Reported success rates include 100% versus 80% for appearance, 80% under large position shifts, 70% for novel instances, and 80% for unseen backgrounds; only HUMAN2ROBOT completes the described writing task.Source: §5.4 · p. 7, Figure 5 · p. 7

6 Conclusion

The paper presents paired human-robot data collection and HUMAN2ROBOT, which uses video prediction for alignment and robot-dynamics features to guide action decoding. Evaluations report effectiveness on both seen and unseen tasks.

6 Conclusion

  • VR teleoperation produces perfectly paired human-robot data, while HUMAN2ROBOT uses a VPM for alignment and robot-dynamics features to guide action decoding.The reported evaluations show strong performance on both seen and unseen tasks.Source: §6 · p. 7

A System Setup Details

The system combines a VR headset, a 7-DoF robot arm, and two cameras to capture synchronized human and robot actions. Software retargets headset hand coordinates to the robot arm after aligning the two operating spaces.

A System Setup Details

  • The H&R system uses a Meta Quest 3, a 7-DoF xArm, and two Intel Realsense D435 cameras to record human and robot actions.The cameras capture synchronized views at 240 × 424 resolution and 30Hz.Source: A System Setup Details · p. 8
  • OpenTeach connects the hardware, while headset coordinates are retargeted to the robot arm across two matched operating environments.The setup addresses scale and orientation inconsistency between the human and robot coordinate systems.Source: A System Setup Details · p. 8

B Data Collection Details

Data collection begins by recording robot reference positions and requiring operators to establish workspace anchors. Six anchor points align human and robot coordinate systems before synchronized demonstrations are recorded.

B Data Collection Details

  • The preparation phase records three robot anchor points and the arm’s initial position before data collection begins.The robot does not follow hand movements until the operator sets the three workspace anchors.Source: B Data Collection Details · p. 8
  • Six anchor points—three human and three robot—align the coordinate systems, with small headset-origin variations requiring object-position adjustments.The operator uses the right hand for demonstrations and the left hand to start and stop recording.Source: B Data Collection Details · p. 8

B.1 Coordinate System Alignment

The alignment procedure maps human-hand and robotic-arm positions into a shared coordinate system and preserves their relative rotations. It uses real-world anchors, scale adjustment, and initial-state-relative orientation matching.

Position alignment

  • Three real-world anchors establish shared human and robot coordinate systems with consistent scale, aligning the robot’s movement range to the human hand.The operating spaces are matched so corresponding movements have comparable visual extent.Source: §B.1 · p. 8
  • The position mapping includes η, a hyperparameter that scales motion amplitude along the z-axis.The mapped robotic coordinate is computed from the human hand coordinate after applying the alignment transformation.Source: §B.1 · p. 8

Rotation alignment

  • Rotation alignment matches the human hand’s and robot arm’s relative rotations with respect to their respective initial coordinate systems.Human and robot rotation matrices are derived at the same time t and related through a coordinate-system transformation.Source: §B.1 · pp. 8–9

C Supplementary Experiment Results

The system’s control accuracy depends on both hand-tracking precision and errors introduced by mapping human movements to the robot. MCP tracking and controlled latency are used to reduce jitter while preserving accuracy.

C.1 System stability

  • Control accuracy is governed by hand-gesture recognition precision and accuracy loss from mapping hand movements to the robotic arm.The Meta Quest 3 provides highly accurate hand tracking under sufficient lighting conditions.Source: §C.1 · p. 9
  • Tracking the index-finger MCP joint avoids wrist topological inconsistency and thumb-index jitter during grasping.The MCP choice preserves structural consistency without the differing fingertip speeds that introduce jitter.Source: §C.1 · p. 9
  • Allowing 100–300ms control latency trades execution speed for smoother, less jittery motion.Serial control makes system sensitivity depend on execution speed, so added latency improves stability.Source: §C.1 · p. 9

D.1 Task Details

The H&R dataset covers eight base manipulation tasks and six long-horizon combinations. Each of its 2,600 episodes pairs synchronized human-robot videos with robot states, hand transforms, retargeted actions, and timestamps.

D.1 Task Details

  • H&R defines eight base tasks spanning pick-and-place, pushing, pulling, upright placement, and handwriting behaviors.The base tasks vary object type, color, initial position, and target placement conditions.Source: §D.1 · pp. 9–10, Figure 8 · p. 10
  • Six long-horizon tasks combine base behaviors, including sequential cube placement, plate movement, and returning to the initial position.These combinations include picking designated objects, moving plates, and completing tasks before returning.Source: §D.1 · p. 10

D.2 Episode Details

  • Each of the 2,600 episodes contains paired human-hand and robot-arm videos with 200–600 frames, plus robot state and joint-velocity data.The frame count varies by task.Source: §D.2 · p. 10
  • Episode metadata also includes human-hand transforms and keypoints, retargeted robot actions, and timestamps.These fields accompany the synchronized videos and robot measurements.Source: §D.2 · p. 11

Related papers

Loading 2502.16587v4…