PaperlayerView exact revision on arXiv

Source-linked AI summary

Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration

Tyler Ga Wei Lum, Olivia Y. Lee, C. Karen Liu, Jeannette Bohg

arXiv:2504.12609v3cs.ROcs.AI

TL;DR

Dexterous robot learning is hard to scale with wearable- or teleoperation-based demonstrations, and human videos do not directly provide robot action labels. HUMAN2SIM2ROBOT trains simulation RL policies from one human RGB-D video using an object-centric reward and a pre-manipulation pose. In the single-demo regime, it outperforms object-aware trajectory replay by >55% across real-world tasks.

  • Problem

    Collecting high-quality dexterous-hand demonstrations is costly to scale, while human videos lack robot action labels and direct retargeting can produce suboptimal trajectories across embodiments.

    Source: §1 · pp. 1–2

  • Method

    HUMAN2SIM2ROBOT uses one human RGB-D video to extract an object pose trajectory for an object-centric reward and a pre-manipulation hand pose to initialize and guide simulation RL.

    Source: §1 · p. 2

  • Results

    Across real-world tasks in the single-human-demo regime, HUMAN2SIM2ROBOT outperforms object-aware trajectory replay by >55%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework reports zero-shot real-world dexterous manipulation across grasping, non-prehensile manipulation, and complex multi-step tasks without task-specific reward tuning.

    Source: §1 · p. 2

    The method focuses on single-object pose trajectories and assumes a high-quality object tracker and simulator; its tasks use rigid-body objects and environments.

    Source: §6 · p. 9

Abstract

from arXiv · show

Teaching robots dexterous manipulation skills often requires collecting hundreds of demonstrations using wearables or teleoperation, a process that is challenging to scale. Videos of human-object interactions are easier to collect and scale, but leveraging them directly for robot learning is difficult due to the lack of explicit action labels and human-robot embodiment differences. We propose Human2Sim2Robot, a novel real-to-sim-to-real framework for training dexterous manipulation policies using only one RGB-D video of a human demonstrating a task. Our method utilizes reinforcement learning (RL) in simulation to cross the embodiment gap without relying on wearables, teleoperation, or large-scale data collection. From the video, we extract: (1) the object pose trajectory to define an object-centric, embodiment-agnostic reward, and (2) the pre-manipulation hand pose to initialize and guide exploration during RL training. These components enable effective policy learning without any task-specific reward tuning. In the single human demo regime, Human2Sim2Robot outperforms object-aware replay by over 55% and imitation learning by over 68% on grasping, non-prehensile manipulation, and multi-step tasks. Website: https://human2sim2robot.github.io

1 Introduction

Dexterous robot learning is difficult to scale because collecting high-quality demonstrations is costly, while human videos lack robot action labels and direct retargeting can fail across embodiments. HUMAN2SIM2ROBOT uses one human RGB-D video to guide simulation RL through object pose trajectories and a pre-manipulation hand pose, outperforming the cited baselines across real-world tasks.

1 Introduction

  • High-quality dexterous-hand demonstrations often require wearable sensors or teleoperation systems, making collection expensive and difficult to scale.Videos of human-object interactions are a less costly, scalable alternative, but do not directly provide robot action labels.Source: §1 · p. 1
  • Human hand-pose estimation is vulnerable to occlusion and sensor noise, and even accurate retargeting can produce suboptimal robot trajectories because of morphological differences.These problems are especially consequential for contact-rich dexterous manipulation.Source: §1 · p. 2
  • Imitation learning depends on accurate correspondences between demonstrated and learned behaviors, while real-world RL faces task-specific reward engineering and unfavorable sample complexity.Simulation RL is used to learn with the robot's own embodiment instead.Source: §1 · p. 2
  • A single human RGB-D video supplies an object 6D pose trajectory and one pre-manipulation hand pose, without requiring task-specific reward engineering.The method does not require high-fidelity 3D human motion data.Source: §1 · p. 2
  • The object trajectory defines an embodiment-agnostic, object-centric reward, while the pre-manipulation pose initializes RL and facilitates exploration.The demonstration specifies and guides the task rather than directly supplying state-action mappings, allowing deviations when the human strategy is unsuitable for the robot.Source: §1 · p. 2
  • The framework reports robust real-world dexterous manipulation from one human RGB-D demonstration across grasping, non-prehensile manipulation, and complex multi-step tasks.The authors describe it as the first system to achieve this, to the best of their knowledge.Source: §1 · p. 2
  • By >55% over object-aware trajectory replay and >68% over imitation learning, HUMAN2SIM2ROBOT performs better across all real-world tasks in the single-human-demo regime.The tasks include grasping, non-prehensile manipulation, and complex multi-step tasks.Source: §1 · p. 2

2 Related Work

Related approaches use demonstrations for action-label generation, trajectory replay, or reward inference, but face constraints in data requirements, embodiment transfer, or reactive behavior. HUMAN2SIM2ROBOT instead guides simulation RL from one human video to train policies for a full arm-and-hand action space.

Visuomotor Imitation Learning for Robotics

  • Visuomotor imitation learning commonly relies on many demonstrations collected through teleoperation or specialized wearables, while human-video retargeting can yield infeasible or suboptimal actions.Pose-estimation noise and embodiment differences undermine the high-quality action labels on which these methods depend.Source: §2 · p. 2

One-Shot Imitation Learning (OSIL)

  • One-shot imitation methods retarget or adapt a single demonstration, but replaying its modifications may not provide closed-loop recovery under contact variation or rollout perturbations.HUMAN2SIM2ROBOT uses the video for task specification and RL guidance, allowing robot-specific strategies rather than rigid imitation.Source: §2 · p. 3

Reinforcement Learning for Robotics

  • Sim-to-real RL remains underexplored for full arm-and-hand dexterous manipulation; prior approaches may require many same-embodiment demonstrations, lack error recovery, or use roughly 100 demonstrations for inverse reward inference.HUMAN2SIM2ROBOT trains in simulation over a full arm-and-hand action space and uses an explicit object-centric reward from one demonstration.Source: §2 · p. 3

Pre-grasp Initialization

  • Prior pre-grasp methods use poses from similar embodiments for grasping in simulation, whereas HUMAN2SIM2ROBOT targets real-world transfer and both prehensile and non-prehensile manipulation.Its policies can be deployed zero-shot in real-world environments.Source: §2 · p. 3

3 Method

The method turns one human RGB-D demonstration into object-motion targets and a pre-manipulation hand pose, then uses simulation RL to learn robot-specific actions.

3.1 Real-to-Sim & Human Demo

  • A digital twin is built from scene and object meshes, while the video yields object-pose and human-hand trajectories.FoundationPose estimates object poses; HaMeR estimates hand poses, with depth-assisted ICP alignment.Source: §3.1 · pp. 3–4
  • Two-step inverse kinematics retargets the human pose: the robot arm aligns the middle knuckle, then the hand aligns corresponding fingertips.This produces a feasible robot pre-manipulation pose for the object pose at the selected timestep.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Simulation-based Policy Learning

  • The object trajectory specifies the task through an object-centric tracking reward, while the pre-manipulation pose initializes exploration as a rough prior.The pose prior need not be extremely precise; the initial state distribution perturbs the retargeted hand pose.Source: §3.1 · p. 4, §3.2 · p. 5
  • Simulation training prioritizes desired object motion over human action imitation, using the pre-manipulation pose as a rough prior.The policy is trained in an Isaac Gym environment matching the real robot, scene, and object.Source: §3.2 · p. 4
  • Domain randomization, noisy object-pose observations, and random forces prepare an LSTM policy for uncertain dynamics and zero-shot real-robot deployment.The trained policy is deployed without additional fine-tuning.Source: §3.2 · p. 5

4 Experiments & Results

Across real-world tasks, HUMAN2SIM2ROBOT outperforms replay and behavior-cloning baselines. The experiments evaluate task and initialization choices in simulation as well as robot performance.

4 Experiments & Results

  • Experiments test embodiment-specific RL, object-trajectory rewards, initialization, and single-pose sufficiency in simulation and on a real robot.Simulation ablations use the plate-pivot-lift-rack task and three random seeds.Source: §4 · pp. 5–6, §4.1 · p. 6

4.1 Experimental Setup

  • The tabletop setup uses a 16-DoF Allegro hand on a 7-DoF KUKA arm; tasks span three objects and single- and multi-step skill types.Tasks include grasping, non-prehensile manipulation, extrinsic manipulation, and composed skill sequences.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2 Importance of Embodiment-Specific RL

  • Across all tasks, HUMAN2SIM2ROBOT policies outperform Replay by 67%, Object-Aware (OA) Replay by 55%, and Behavior Cloning (BC) by 68%; each task’s success rate is evaluated over 10 policy rollouts.Source: Figure 6 · p. 6, §4.2 · p. 7

4.3 Importance of the Object Pose Trajectory

  • The reward ablation compares trajectory tracking with fixed targets, interpolated endpoints, and downsampled key poses.The downsampled trajectory reduces the demonstration's temporal resolution to a series of key poses.Source: Figure 7 · p. 7, §4.3 · p. 7

4.4 Importance of Pre-Manipulation Pose Initialization

  • Initialization ablations contrast default-rest and overhead starts with pre-manipulation pose, including a variant moved 20 cm from the object.The far variant retains the pre-manipulation hand pose while moving the palm away from the object.Source: §4.4 · p. 7, Figure 8 · p. 8

4.5 Sufficiency of Single Pre-Manipulation Pose

  • The single pre-manipulation pose is compared with full-hand-trajectory alternatives: a hand-tracking reward and an open-loop residual policy.The residual policy replays the retargeted trajectory while learning delta PD joint targets.Source: Figure 9 · p. 8, §4.5 · p. 8

5 Conclusion

HUMAN2SIM2ROBOT trains dexterous manipulation policies from one human RGB-D video using object-centric rewards and a pre-manipulation pose. The paper reports improvements across manipulation task types and presents the approach as a step toward scalable, robust real-world policy training.

5 Conclusion

  • HUMAN2SIM2ROBOT trains dexterous manipulation policies from one human RGB-D video using object-centric rewards and a pre-manipulation hand pose.The framework trains policies in simulation for real-world use.Source: §5 · p. 8
  • The framework addresses efficient exploration, reward-engineering effort, the human-robot embodiment gap, and sim-to-real transfer.The conclusion reports significant improvements across grasping, non-prehensile, and extrinsic manipulation tasks.Source: §5 · p. 8
  • The authors present the approach as a step toward scalable and robust training of real-world dexterous manipulation policies.Source: §5 · p. 8

6 Limitations & Future Work

The framework has evaluation and tracking scope limits, while its appendices describe human-demo processing, retargeting, reward flexibility, and RL initialization. Several steps rely on rigid objects, object tracking, and simulator assumptions.

6 Limitations & Future Work

  • Evaluation covers a Kuka arm and Allegro hand; broader embodiment generality remains unverified beyond preliminary LEAP Hand and UMI gripper experiments.The framework focuses on single-object pose-trajectory tasks, assumes a high-quality object tracker and simulator, and evaluates rigid-body objects and environments.Source: §6 · p. 9

A.1 Digital Twin Construction Details / A.2 Human Demo Processing Details

  • Scene scanning and alignment take about five minutes once, while scanning each object takes about two minutes; demo processing takes 5–10 minutes after a 30-second recording.The workflow uses separate apps for workspace and object scans, then segments video, estimates the object trajectory, and estimates and depth-aligns the pre-manipulation hand pose.Source: §A.1 · p. 15, §A.2 · p. 15

A.3 Depth Alignment of HaMeR Hand Pose Estimates

  • HaMeR’s RGB-only hand estimates can have 20–30 cm 3D errors, so depth alignment and ICP improve alignment in most frames but degrade under substantial occlusion.Occlusion can occur when fingers grasp around an object and face away from the depth camera.Source: §A.3 · pp. 15–16

B Human-to-Robot Retargeting Details

  • Human-to-robot retargeting uses arm and fingertip IK, seeds successive poses from prior solutions, skips failed poses, and stretches timing when joint velocities exceed limits.Arm IK targets derive from the middle knuckle and wrist orientation; hand IK targets the four fingertips.Source: B Human-to-Robot Retargeting Details · p. 16

C Reward Function Details

  • Anchor points make the pose-tracking reward adaptable to rotational symmetry, and the experiments use the same L hyperparameter across tasks apart from symmetry adjustments.For symmetric objects, points orthogonal to the symmetry axis can be removed from both reward computation and policy observation.Source: C Reward Function Details · p. 17, Figure 11 · p. 17

D Initial State Distribution Sampling

  • RL initialization samples the object pose near its demonstrated starting pose, then derives a corresponding wrist pose and adds small noise to the wrist and hand joints.The described object perturbations use yaw within ±20° and horizontal translation within ±0.1 m, with roll, pitch, and height fixed.Source: D Initial State Distribution Sampling · p. 17

E Simulation Training Details

Simulation training combines parallel RL, domain and force randomization, and privileged critic information with deployment-time object tracking, policy control, and pose re-registration.

E Simulation Training Details

  • Asymmetric actor-critic training gives the critic privileged simulation state while the policy receives only observations available at real-world deployment.The authors state that privileged critic information improves the speed and quality of policy training.Source: E Simulation Training Details · p. 18
  • Training uses 4096 parallel agents, PPO, and domain randomization of observations, actions, gravity, object and table properties, and robot properties.Random object forces occur with 5% probability per timestep at magnitude 50 times the object’s mass, promoting tracking of displaced objects and robust grasps.Source: E Simulation Training Details · pp. 17–18

F.1 Policy Inference-Time Details / F.2 Real-Time Perception Details

  • At deployment, object poses are tracked at 30 Hz, policy actions run at 15 Hz, and a geometric fabric controller produces targets for a 200 Hz joint PD controller.The pose tracker can degrade with rapid motion or heavy occlusion; a mask IoU below 0.1 triggers re-registration.Source: §F.1 · p. 18, Figure 12 · p. 18, §F.2 · p. 19

G Full Task List

The task suite spans pushing, pivoting, grasp-and-pour, and multi-step placement, with success criteria defined by contact, pose, or completion of sequential stages.

G Full Task List

  • Snackbox-push and plate-push succeed when the object is pushed across the table into contact with a static box.Both start within a randomized 4 cm × 4 cm region; the snackbox begins face-down and the plate flat.Source: G Full Task List · p. 19
  • Snackbox-pivot tests turning a face-down snackbox sideways against a static box into a stable orientation.The starting position is randomized within a 1 cm × 4 cm region, and the robot’s initial position restricts movement in one direction.Source: G Full Task List · p. 19
  • Pitcher-pour requires grasping a pitcher by its handle, lifting it, and positioning its spout above a saucepan.The pitcher begins upright, with its position randomized within a 4 cm × 4 cm region.Source: G Full Task List · p. 19
  • Snackbox-push-pivot, plate-lift-rack, and plate-pivot-lift-rack grade sequential stages with partial credit when later stages fail.The three-stage plate task scores pivot, lift, and rack completion as 0, 0.33, 0.66, or 1.Source: G Full Task List · pp. 19–20

H.1 Full Human Hand Trajectory Estimation

The appendix details trajectory retargeting and replay baselines, including ways to address noisy, inconsistent, or unreachable hand poses. It also describes object-aware replay and generating behavior-cloning data from one demonstration.

H.1 Full Human Hand Trajectory Estimation

  • Framewise IK retargeting can fail from hand-pose errors, temporal inconsistency, and unreachable targets; filtering, seeded IK, and skipping frames address these issues.Even after downsampling and velocity checks, severe finger occlusion or a hand far from the camera can still produce inaccuracies; a pre-manipulation pose is generally easier to estimate.Source: §H.1 · p. 20

H.2 Replay Details

  • Replay uses calibrated camera, object, and hand transforms to obtain the object trajectory and an IK-derived robot configuration trajectory, which can be tracked with joint PD control.The frame convention defines a relative transform from frame A to frame B and composes transforms across frames.Source: §H.2 · pp. 20–21

H.3 Object-Aware Replay Details

  • Object-aware replay preserves the demonstrated object-to-middle-finger relationship while adjusting the trajectory for a changed initial object pose.It assumes the object follows the same relative motion as in the demonstration after the new starting pose.Source: §H.3 · p. 21

H.4 Behavior Cloning Details

  • Behavior-cloning data are augmented by perturbing the object pose and recomputing robot joint trajectories with adjusted IK targets.The observation includes robot joints, palm pose, and object pose; actions are joint-position targets relative to current robot position.Source: §H.4 · pp. 21–22

I Qualitative Analysis

The qualitative analysis shows that the Allegro hand can develop a manipulation strategy different from the human’s, while failures arise from simulation inaccuracies or pose-estimation error. The policy also requires initialization near the pre-manipulation pose.

I Qualitative Analysis

  • For plate-pivot-lift-rack, the Allegro hand pivots the plate with its ring finger and then clips it between its middle and index fingers, unlike the human strategy.The authors report that small differences in pre-manipulation hand pose can lead to different learned strategies given human-robot morphology differences.Source: I Qualitative Analysis · p. 22
  • Failures typically stem from policies exploiting simulation inaccuracies or from pose-estimation errors caused by occlusion.Imperfect tabletop and static-object friction modeling can produce policies that exploit inaccurate parameters; thin plates make plate-pivot-lift-rack especially challenging.Source: I Qualitative Analysis · p. 22
  • The policy must be initialized in the rough region of the pre-manipulation pose; collision-free planning or an approach-stage reward are proposed as ways to reach it from rest.The authors identify these techniques as potential additions to navigate from a default rest pose to the pre-manipulation pose.Source: I Qualitative Analysis · p. 22

J Other Embodiments

Preliminary snackbox-push experiments demonstrate Human2Sim2Robot on a LEAP Hand and UMI gripper as well as its primary Kuka arm and Allegro hand setting. These embodiments work without reward tuning, with robot-specific configuration changes.

J Other Embodiments

  • Initial snackbox-push experiments show successful Human2Sim2Robot results on a LEAP Hand and UMI gripper without reward tuning.Figure 14 presents qualitative results; the green box marks the snackbox target pose.Source: J Other Embodiments · p. 22, Figure 14 · p. 23
  • Adapting the system required changes to the robot URDF and configurations for retargeting, the geometric fabric controller, and the simulation environment.The listed changes include link names, orientations, default joint configurations, collision spheres, and action/observation dimensions.Source: J Other Embodiments · pp. 22–23

K Additional Related Work

The related work highlights limitations of direct imitation and replay under noisy observations and embodiment differences. Human2Sim2Robot instead uses a single video to guide simulation RL, aiming to learn robot-suited strategies across dexterous tasks.

K Additional Related Work

  • Visuomotor imitation learning commonly depends on many expert demonstrations collected through teleoperation or wearable equipment, and video-based retargeting faces pose noise and embodiment mismatch [4] [5] [7] [8] [26] [9] [23] [24].Human videos are described as less expensive and more intuitive to collect, but do not provide explicit action labels.Source: K Additional Related Work · p. 23
  • Simulation RL guided by a single human video allows policies to deviate from unsuitable human actions and can yield recovery and retry behavior without explicit demonstrations.The approach trains manipulation policies in simulation rather than directly mapping observations to actions.Source: K Additional Related Work · p. 23
  • One-shot replay and retargeting approaches have limited generalization beyond demonstrated actions, while noisy pose estimates and embodiment differences can make retargeted trajectories infeasible.Prior approaches include object-pose estimation [27] [28], object-aware retargeting [29], scene adaptation before replay [30], and simulation-based trajectory generation from a small set of teleoperated demonstrations [31].Source: K Additional Related Work · pp. 23–24
  • Prior robotic RL work spans sim-to-real tasks, but many dexterous manipulation studies remain in simulation with non-physical floating-hand robots [37] [38] [39] [40].The cited examples also include legged locomotion [32] [33], drone racing, bipedal soccer [61], and in-hand manipulation [36].Source: K Additional Related Work · p. 24
  • Prior sim-to-real methods fine-tune behavior-cloning policies [16] or learn residual actions around an open-loop motion [3], requiring same-embodiment demonstrations or limiting adjustments to the base motion.The passage notes that both approaches require accurate human action sequences from motion capture or teleoperation.Source: K Additional Related Work · p. 24
  • Lum et al. [44] use sim-to-real RL for dexterous grasping, whereas Human2Sim2Robot incorporates human guidance for prehensile and non-prehensile manipulation.The cited method is limited to simple grasping and typically learns a top-down grasp from an above-object initialization without human guidance.Source: K Additional Related Work · p. 24
  • Prior pre-grasp methods focus on grasping, use pre-grasps from similar embodiments, and have been tested only in simulation [42] [43].Human2Sim2Robot instead targets real-world transfer across the human-robot embodiment gap and acquires pre-manipulation poses from one human video.Source: K Additional Related Work · p. 24
Loading 2504.12609v3…