Source-linked AI summary
HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos
Haoyang Weng, Yitang Li, Nikhil Sobanbabu, Zihan Wang, Zhengyi Luo, Tairan He, Deva Ramanan, Guanya Shi
TL;DR
Robust whole-body humanoid-object interaction is difficult because suitable 3D human-object motion data are scarce and contact-rich RL is challenging. HDMI learns these skills from monocular RGB videos by extracting structured trajectories and training robot-object co-tracking policies with unified interaction designs. On Unitree G1, it achieved 67 consecutive door traversals, 6 real-world loco-manipulation tasks, and 14 simulated tasks.
Problem
Whole-body HOI is challenging because 3D human-object motion data are scarce and RL must learn contact and balance under imperfect references.
Method
HDMI extracts and retargets human and object trajectories from monocular RGB videos, then trains RL policies for robot-object co-tracking with unified representations, residual actions, and interaction rewards.
Results
HDMI achieves 67 consecutive door openings and traversals, 6 distinct real-world loco-manipulation tasks, and 14 simulated tasks.
Takeaways & Limitations
The results establish HDMI as a simple and general framework for acquiring interactive humanoid skills directly from human videos.
Takeaways & Limitations
The current system relies on ground-truth motion-capture data, motivating policies that operate directly from onboard sensing in uninstrumented environments.
Abstract
from arXiv · showhide
Enabling robust whole-body humanoid-object interaction (HOI) remains challenging due to motion data scarcity and the contact-rich nature. We present HDMI (HumanoiD iMitation for Interaction), a simple and general framework that learns whole-body humanoid-object interaction skills directly from monocular RGB videos. Our pipeline (i) extracts and retargets human and object trajectories from unconstrained videos to build structured motion datasets, (ii) trains a reinforcement learning (RL) policy to co-track robot and object states with three key designs: a unified object representation, a residual action space, and a general interaction reward, and (iii) zero-shot deploys the RL policies on real humanoid robots. Extensive sim-to-real experiments on a Unitree G1 humanoid demonstrate the robustness and generality of our approach: HDMI achieves 67 consecutive door traversals and successfully performs 6 distinct loco-manipulation tasks in the real world and 14 tasks in simulation. Our results establish HDMI as a simple and general framework for acquiring interactive humanoid skills from human videos.
I. INTRODUCTION
HDMI addresses scarce whole-body humanoid-object interaction data and difficult contact-rich RL by learning interactive skills directly from human RGB videos. It combines structured trajectory extraction, unified interaction-aware control designs, and sim-to-real validation.
- Whole-body HOI data are scarce, and RL must guide contact behavior and balance with objects despite imperfect references.
- The pipeline extracts and retargets human and object trajectories, trains robot-object co-tracking with RL, and deploys the policy on humanoids.
- HDMI learns autonomous whole-body humanoid-object interaction skills directly from human videos as a general framework.
- Three unified components support training: diverse-object representation, residual actions for exploration, and contact-promoting interaction rewards.
- 67 consecutive bi-directional door openings and traversals were achieved on Unitree G1.
- 6 distinct loco-manipulation tasks were deployed on real hardware, alongside 14 tasks in simulation.
B. Humanoid Loco-Manipulation
HDMI frames interactive skill acquisition as a pipeline from monocular human videos to structured reference trajectories and interaction-centric policy training. Its representation includes robot, object, articulated-joint, and contact information.
- B. Humanoid Loco-Manipulation: Existing loco-manipulation methods remain limited in generality and robustness, using reward engineering, trajectory optimization, specialized architectures, or hierarchical policies.
- B. Humanoid Loco-Manipulation: Video-based locomotion methods do not explicitly model object dynamics, while video-based manipulation methods are typically constrained.
- C. Robot Learning from Human Videos: HDMI retargets monocular RGB videos into structured reference trajectories for interaction-centric policy training through robot-object co-tracking.
- A. Scalable Skill Specification from Human Videos: Human videos are processed with pose estimation and retargeting, then object trajectories and contact signals are annotated into reference motions.
- A. Scalable Skill Specification from Human Videos: Object state includes position and orientation, with articulated-object joint state θ_obj and a binary signal indicating intended contact.
B. Interactive Skill Learning via Robot-Object Co-Tracking
HDMI formulates interactive skill learning as robot-object co-tracking: an RL policy simultaneously follows robot and object reference states. Training uses randomized reference initialization, phase conditioning, tracking-based termination, and PPO.
- Interactive skill learning is formulated as whole-body robot-object co-tracking with reinforcement learning.
- Episodes initialize robot and object states from randomized reference frames with small perturbations to improve robustness.
- Episodes terminate when robot or object states deviate excessively from the reference, while PPO optimizes tracking and regularization rewards.
- Three targeted solutions address robust whole-body object interaction challenges.
1) Unified Object Representation:
HDMI represents object pose and desired contact points in the robot’s local root frame, alongside proprioception and phase, while residual actions anchor exploration to reference poses.
- Unified Object Representation: Object pose and reference contact points are expressed in the robot’s local root frame for spatially invariant observations.The representation supports diverse object geometries and types without architectural changes.
- Unified Object Representation: The policy combines local object observations and contact points with proprioceptive state and a phase variable.
- Unified Object Representation: Reference contact points specify desired robot–object contact locations and are observed during training and deployment.
- 2) Residual Action Space: Residual actions add corrective offsets to reference joint positions instead of directly predicting absolute joint targets.This centers exploration around the current reference pose, helping with motions such as kneeling.
- 2) Residual Action Space: Residual exploration improves sample efficiency and speeds convergence for motions far from the default standing pose.
3) Unified Interaction Reward:
HDMI uses a unified interaction reward to correct imperfect kinematic references by promoting accurate, stable, and bounded contact across active end-effectors.
- 3) Unified Interaction Reward: Kinematic video-retargeted trajectories may lack precise contacts or contain penetration artifacts, making motion-tracking rewards insufficient.
- 3) Unified Interaction Reward: The unified interaction reward promotes stable contact when the reference indicates intended interaction.It is gated by the contact signal.
- 3) Unified Interaction Reward: The reward combines end-effector position alignment with a force term that favors sufficient but not excessive contact force.The force term is capped by Fthres for deployment safety.
- 3) Unified Interaction Reward: The interaction reward is averaged across end-effectors designated for contact.Nc denotes the number of end-effectors desired to have contact.
- Training Robustness: Domain randomization varies robot and object inertial and friction properties during training to improve robustness.
IV. REAL-WORLD EXPERIMENTS
The real-world evaluation tests HDMI on contact-rich, long-horizon whole-body interactions using policies trained in simulation and deployed directly on a Unitree G1.
- Experiment Scope: Five real-world interaction tasks test contact-rich interaction, combined locomotion and manipulation, and long-horizon whole-body coordination.
- Experiment Setup: Policies are trained in IsaacSim with shared hyperparameters and directly deployed on a Unitree G1 humanoid.
- Data Sources and Scope: Reference motions come from RGB videos except suitcase manipulation, which uses the Omomo dataset.Omomo motions involving dexterous prehensile manipulation are outside the study’s scope.
A. Case I: Door Open and Traversal
The real-world demonstrations evaluate HDMI on contact-rich, long-horizon whole-body behaviors that combine locomotion, manipulation, and expressive motion. The policy remains robust across repeated door traversal, varied object handling, and complex stair interactions.
- Door traversal: 67 continuous door runs tested bidirectional hand pushing, walking, turning, foot kicking, and return traversal before failure.The robot also completed approximately 7 runs after a wooden floorboard was removed and adapted to 10–30 cm randomized positional offsets.
- Object manipulation: 7 consecutive suitcase runs combined kneeling, lifting, and walking with load, while bread-box carrying completed 2 full trials.Foam-mat relocation also succeeded across grasping, sidestepping, and placement behaviors.
- Truman’s Bow: 3 continuous executions completed a staircase sequence involving climbing, bowing, sitting, waving, jumping off, and returning.Operators occasionally provided light back support because of stair-climbing risks.
- Overall capability: The demonstrations collectively show long-horizon whole-body behaviors spanning contact-rich object-scene interaction and precise full-body pose control.The reported behaviors include door interaction, object transport, stair sitting, bowing, and waving.
V. SIMULATION ABLATIONS
The simulation ablations evaluate HDMI design choices across parallel IsaacSim environments using task completion as the success criterion and reporting aggregate metric variability.
- Evaluation setup: 4096 parallel IsaacSim environments were used, with mean and standard deviation computed for evaluation metrics.Each evaluation started at the reference beginning and counted as successful only if the policy finished without triggering termination.
A. Interaction Reward
The interaction reward is most important when references are imperfect or precise contact locations are required, while its removal often leaves final success unchanged on other tasks.
- Interaction Reward: For the majority of tasks, removing interaction reward and contact-based termination does not significantly affect final performance.The ablation compares variants with both components, without interaction reward only, and without both components.
- Interaction Reward: Training succeeds without interaction reward when a successful policy supplies a perfect reference motion.This isolates the reward’s role in addressing imperfections in reference motions rather than making every task successful by itself.
- Interaction Reward: Interaction reward guides pushing-box policies toward precise end-effector placement on the box edge.Without it, end effectors often contact the box’s vertical surface, producing unstable contact.
- Interaction Reward: Interaction reward is crucial for move-suitcase success when the reference is imperfect.Without it, the policy rigidly follows flawed retargeted motion and can fail to establish a grasp.
B. Residual Action Space
Residual actions stabilize exploration and accelerate learning, especially for challenging poses far from the default standing configuration. Removing residual actions increases tracking errors and can prevent intended kneeling behavior.
- Residual Action Space: The full method consistently achieves the lowest joint and body tracking errors across 8 tasks.Removing residual actions produces slower convergence and fails to reach the same performance level, with or without tracking-error termination.
- Residual Action Space: Without residual actions and tracking-error termination, move-suitcase policies fail to learn kneeling and instead bend at the waist with both feet flat.This is a suboptimal strategy relative to the intended kneeling motion.
- Residual Action Space: Residual actions center exploration around the reference pose, avoiding abrupt pop-up behavior when episodes begin in kneeling.This targeted exploration produces higher-quality samples and supports faster convergence for complex motions.
- Residual Action Space: Residual action policies achieve low, stable tracking errors from the beginning and converge faster to high success rates.Policies without residual actions start with higher errors and converge much more slowly.
VI. LIMITATIONS AND FUTURE DIRECTIONS
HDMI demonstrates object-interaction skills across 14 simulated tasks, but its current deployment scope is limited by motion-capture dependence and task-specific specialist policies.
- 14 simulated tasks demonstrate HDMI’s effectiveness, while two key limitations remain.
- The current system relies on ground-truth motion-capture data, including object poses, rather than onboard sensing.Future policies should operate from cameras or other onboard modalities for uninstrumented environments.
- A separate specialist policy is currently trained for each task, limiting the system’s breadth to task-specific models.A proposed future direction is a unified generalist model trained from multiple skills.