Source-linked AI summary
Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanya Shi
TL;DR
Real-time whole-body teleoperation of full-sized humanoids lacks scalable feasible-motion data and practical sensing. H2O addresses this with reinforcement learning, sim-to-data filtering, and RGB-camera pose tracking, achieving diverse dynamic teleoperation in real-world tests while exposing representation and embodiment gaps for future work.
Problem
Real-time whole-body humanoid teleoperation requires diverse feasible motions, but human-to-humanoid differences make naively retargeted datasets contain motions the humanoid cannot perform.
Method
H2O filters retargeted motions with a privileged motion imitator, trains a robust RL humanoid motion imitator with sim-to-real regularization, and tracks RGB-camera human pose goals.
Results
H2O enables real-time whole-body teleoperation of dynamic humanoid motions including ball kicking, walking, and back jumping despite noisy, perspective-biased RGB pose estimates.
Takeaways & Limitations
The framework demonstrates the feasibility of learning-based whole-body humanoid teleoperation using only an RGB camera and minimal hardware.
Takeaways & Limitations
Future scalability is constrained by a trade-off between expressive motion representations and RL sample efficiency, alongside the remaining embodiment gap between humans and humanoids.
Abstract
from arXiv · showhide
We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.
I. INTRODUCTION
H2O targets real-time whole-body teleoperation of a full-sized humanoid using only an RGB camera. It combines feasible-motion data filtering, reinforcement-learning control, and camera-based human pose tracking to demonstrate diverse dynamic behaviors.
- Whole-body control becomes especially difficult when full-sized humanoids must replicate freeform human movements in real time.
- Existing model-based teleoperation methods simplify humanoid dynamics and depend on contact measurements, external exoskeletons, or force sensors.
- H2O introduces a scalable sim-to-data process that filters retargeted human motions into motions feasible for the real-world humanoid.
- The system trains an RL whole-body tracking controller for sim-to-real transfer and uses an RGB camera with 3D human pose estimation during teleoperation.
- The demonstrated system performs diverse whole-body motions including walking, pick-and-place, stroller pushing, boxing, handwaving, and ball kicking.
II. RELATED WORKS
Prior humanoid motion-control approaches rely on physics-based animation or model-based optimization but face real-world applicability and deployment constraints. H2O positions learning-based whole-body teleoperation as a marker- and force-sensor-free alternative using RGB capture.
- A. Physics-Based Animation of Human Motions: Physics-based simulation can generate realistic avatar motions, but simulated humanoid animation does not guarantee real-world applicability.
- B. RL-based Humanoid Controllers: Traditional model-based optimization tracks retargeted motions under stability and contact constraints, requiring predefined or sensor-estimated contacts.
- B. RL-based Humanoid Controllers: RL-based controllers offer an alternative to explicit contact information, but prior work did not accurately track human motions or precise lower-body movements.
- C. Teleoperation of Humanoids: Humanoid teleoperation includes task-space, upper-body-retargeted, and whole-body approaches, with partial morphology use in the first two categories.
- C. Teleoperation of Humanoids: H2O is presented as the first learning-based whole-body teleoperation system and uses an RGB camera without capture markers or force sensors on the human operator.
III. PRELIMINARIES
H2O formulates whole-body humanoid teleoperation as goal-conditioned reinforcement learning, training a single policy to track versatile human motion from proprioception and motion goals.
- The goal-conditioned formulation trains one RL control policy to track versatile human motions frame by frame.
- H2O models teleoperation as a Markov Decision Process with state, action, transition dynamics, reward, and discount factor.
- The policy state combines humanoid proprioception with a goal state representing the teleoperator’s whole-body motion.
- Actions specify joint target positions that a PD controller uses to actuate the humanoid’s degrees of freedom.
B. Parametric Human Model and Human Motion Dataset
SMPL provides a parametric representation of human shape and motion, while AMASS supplies a large motion-capture dataset. H2O fits SMPL body shape to humanoid keypoints to improve retargeted motion feasibility.
- SMPL represents human shape, pose, and translation parametrically and maps them to a 6,890-vertex triangular mesh.
- Fitting the SMPL shape to humanoid keypoints improves position-based retargeting from unstable in-toed motion to balanced humanoid motion.
- AMASS contains 40 hours of motion capture expressed in SMPL parameters.
IV. RETARGETING HUMAN MOTIONS FOR HUMANOID
H2O retargets human motions by first matching SMPL body shape to the humanoid structure, then uses this alignment to generate more feasible humanoid motion data. The process produces 10k retargeted sequences from 13k AMASS sequences.
- Naively retargeted human motions can be impossible for the humanoid and hinder motion-imitation training.
- H2O optimizes SMPL shape parameters using 12 corresponding human–humanoid joints before retargeting motion sequences.
- Using the fitted SMPL shape creates motion that is more feasible for the humanoid.
- 10k retargeted motion sequences are generated from an AMASS set containing 13k motion sequences.
B. Simulation-based Data Cleaning
H2O uses a privileged simulation policy to identify and remove implausible retargeted motions, then trains an imitator with state variables intended to be available during real-world teleoperation. The deployed policy avoids privileged physics information unavailable on the robot.
- The sim-to-data procedure uses a privileged motion imitator to filter implausible retargeted motions from a large dataset.
- The privileged policy observes global rigid-body positions, orientations, linear velocities, and angular velocities.
- Real-time teleoperation requires policy states containing quantities available in the real world, unlike simulation-only approaches with physics information such as foot contact force.
- The agile policy uses 19-dimensional joint targets, which a PD controller converts into joint torques.The controller applies τ = Kp(at − qt) − Kd q̇t.
B. Reward Design
The reward design combines penalties, regularization, and task rewards to support sim-to-real transfer and real-time whole-body tracking. Although the state uses only eight selected body positions, training receives full-body reward signals across all joints.
- Reward components: The reward function combines penalty, regularization, and task-reward terms.These components respectively address undesired behaviors, motion refinement, and successful whole-body tracking.
- Full-body supervision: Eight selected body positions in the state support six full-body reward terms covering joint and body motion quantities.The terms include DoF position and velocity, body position and rotation, and body velocity and angular velocity.
- Full-body supervision: The expressive reward terms provide denser training signals for efficient reinforcement learning.
C. Domain Randomization
H2O uses domain randomization to improve robustness and generalization during sim-to-real transfer. The randomized factors span simulated dynamics, control, disturbances, delays, and terrain.
- Purpose: Domain randomization is used as a key source of robustness and generalization for sim-to-real transfer.
- Randomized factors: The dynamics-randomization design includes simulated dynamics, external perturbations, and randomized terrain.These categories are identified as important for sim-to-real transfer and for boosting robustness and generalizability.
- Randomized factors: H2O randomizes ground friction, link mass, torso center-of-mass position, PD gains, applied-torque noise, control delay, and terrain types.Link mass and PD gains are randomized independently per link or joint, while the remaining factors are randomized episodically.
D. Early Termination Conditions
Early termination improves training efficiency by ending episodes when the robot falls, becomes excessively tilted, or deviates too far from the reference motion. Simulation experiments compare retargeting, state-space, and training variants on 10k uncleaned AMASS sequences.
- Termination rules: Training terminates when base height falls below 0.3m, projected gravity exceeds 0.7 on the x or y axis, or average link distance exceeds 0.5m.
- Simulation experiments: The simulation study evaluates four baselines to isolate the effects of retargeting, goal-state design, and sim-to-real training techniques.The baselines are the privileged policy, H2O-w/o-sim2data, H2O-reduced, and full H2O.
- Simulation experiments: The baselines are evaluated on 10k uncleaned retargeted AMASS sequences using success rate and global and root-relative MPJPE metrics.
- Simulation results: H2O significantly outperforms H2O-w/o-sim2data and H2O-reduced, showing the importance of motion filtering and informative goal-state design.
- Simulation results: The success-rate gap between H2O and the privileged policy reflects H2O’s less informative observation space and sim-to-real regularization and randomization.These choices reduce simulation performance while supporting robustness and stability for real-world transfer.
- Simulation results: H2O achieves higher success rates than the no-sim-to-data and reduced-state variants despite training on less data after infeasible motions are filtered.
B. Real-world Demonstrations
H2O enables real-time whole-body teleoperation from RGB-camera pose estimates, including dynamic motions such as walking, kicking, and back jumping. The system also maintains balance under external force disturbances.
- Deployment: Real-world deployment uses a standard 1080P webcam and HybrIK 3D human pose estimation at 30Hz.
- Deployment: Robot linear velocity is estimated with a 50Hz motion-capture system, while other proprioception comes from built-in sensors at 200Hz.The authors note that onboard visual or LiDAR odometry could replace motion capture.
- Dynamic demonstrations: Despite noisy and perspective-biased RGB-camera pose estimates, H2O generalizes to real-time estimated motion goals.
- Dynamic demonstrations: The robot performs precise whole-body motions including ball kicking, walking, and back jumping in real-time teleoperation.
- Robustness: H2O maintains balance under external force disturbances in real-world tests.The disturbance experiments are presented as evidence of system robustness.
VII. DISCUSSIONS, LIMITATIONS, AND FUTURE WORK
H2O demonstrates real-time whole-body humanoid teleoperation with an RGB camera, while identifying representation, embodiment, and sim-to-real gaps that limit broader generalization.
- Universal teleoperation: More expressive motion representations support finer and more diverse motions, but their increased dimensionality reduces scalable RL sample efficiency.The paper identifies this representation gap as a trade-off for future work.
- Universal teleoperation: Motion feasibility varies across robots, and training on infeasible or damaged motions can substantially harm performance.Hardware constraints create an embodiment gap that lacks systematic feasibility-identification algorithms.
- Universal teleoperation: Successful sim-to-real transfer requires regularization and domain randomization, but over-regularization or over-randomization can hinder motion learning.The best trade-off for universal humanoid control remains unresolved.
- Real-time teleoperation: RGB-camera latency and pose-estimation error create an efficiency–precision trade-off, while current operators receive only visual feedback.The paper suggests investigating multimodal feedback such as force, verbal, and conversational interaction.
- Whole-body teleoperation: Lower-body tracking remains necessary for agile human motions such as sports and dancing, especially where legged robots offer advantages over wheeled robots.The discussion highlights stepping stones and kicking as scenarios requiring diverse lower-body movements.
- Conclusions: H2O enables real-time whole-body humanoid teleoperation using only an RGB camera across diverse dynamic tasks.Demonstrated motions include ball kicking, walking, and back jumping, alongside other whole-body interactions.