Source-linked AI summary
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, Guanya Shi
TL;DR
Humanoid whole-body control needs to combine stable locomotion, dexterous manipulation, and scalable demonstration collection despite costly interfaces and limited humanoid datasets. OmniH2O uses kinematic pose, teacher-student sim-to-real learning, and multiple teleoperation or autonomous inputs; it demonstrates whole-body capabilities and releases a humanoid dataset for imitation learning. Its scope remains bounded by noisy root odometry and missing safety guarantees for extreme disturbances or out-of-distribution motion goals.
Problem
Whole-body humanoid control must coordinate stable locomotion with dexterous manipulation, while expensive interfaces and limited humanoid datasets constrain scalable demonstrations.
Method
OmniH2O uses kinematic pose as a universal interface and trains a deployable sparse-input policy by distilling an RL-trained privileged teacher, with retargeted motion and teleoperated data.
Results
OmniH2O supports dexterous whole-body loco-manipulation through VR, RGB, language, frontier-model, and imitation-learning control, while improving motion imitation over prior art and matching the teacher’s success rate.
Takeaways & Limitations
The system provides a common route from human motion or instructions to humanoid actuation and supports whole-body learning from teleoperated demonstrations.
Takeaways & Limitations
The system requires robot root odometry, and it lacks guarantees or safety checks for extreme disturbances and out-of-distribution motion goals.
Abstract
from arXiv · showhide
We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.
1 Introduction
OmniH2O addresses the difficulty of stable, dexterous whole-body control and the costly interfaces that hinder scalable humanoid demonstrations. It proposes a robust learning pipeline with a universal kinematic-pose interface for teleoperation, autonomy, and whole-body skill learning.
- Motivation: Whole-body humanoid control must coordinate dexterous manipulation and robust locomotion across upper and lower bodies.Existing approaches often focus on the lower body or decouple upper- and lower-body control.
- Motivation: Expensive motion-capture and exoskeleton setups hinder scalable collection of humanoid demonstration data.The paper identifies accessible interfaces such as VR as a requirement for scalable data collection.
- Approach: OmniH2O trains a robust whole-body motion-imitation policy using teach-student distillation and designs the motion distribution, rewards, state space, and history utilization for stability.The training distribution emphasizes standing and squatting, regularization rewards use a curriculum, and history can replace global linear velocity.
- Approach: Kinematic pose serves as an intermediate control representation compatible with VR, RGB cameras, language, and autonomous agents such as GPT-4o.The interface bridges human instructions and humanoid actuation while supporting multiple control sources.
- Contributions: The paper reports a robust policy for whole-body dexterous loco-manipulation, validates motion imitation in simulation and the real world, and releases the first humanoid loco-manipulation dataset.The dataset supports evaluation of imitation learning for whole-body skill learning from teleoperated datasets.
2 Related Works
Related work shows that humanoid research has separately advanced locomotion, teleoperation, control interfaces, and robotic datasets, but whole-body mobile manipulation and humanoid-scale data remain limited. OmniH2O positions itself as an end-to-end whole-body system with accessible interfaces and a full-sized humanoid dataset.
- Learning-based Humanoid Control: Most learning-based humanoid studies emphasize locomotion, while whole-body loco-manipulation requires the lower body to support precise upper-body movement.OmniH2O instead learns an end-to-end policy coordinating upper and lower bodies.
- Humanoid Teleoperation: Prior humanoid teleoperation systems use task-space, upper-body-retargeted, or whole-body control, with RGB-based approaches limited by pose-estimation delay and inaccuracy.The cited prior work also requires global linear velocity estimation, which creates additional practical constraints.
- Control Interfaces: Exoskeleton, motion-capture, and VR interfaces have been proposed, but VR-based systems largely focus on animation rather than real-world mobile manipulation.OmniH2O applies its interface to a real humanoid robot performing manipulation tasks.
- Datasets: Public robotic datasets are scarce for humanoids, and existing datasets largely use fixed-base arms or exclude humanoid whole-body loco-manipulation.The paper presents its dataset as the first for full-sized humanoid whole-body loco-manipulation.
3 Universal and Dexterous Human-to-Humanoid Whole-Body Control
OmniH2O builds a whole-body control system around retargeted human motion, privileged teacher learning, and a deployable student policy using sparse real-world observations. Kinematic pose provides a common interface for teleoperation and autonomous motion goals, while dataset balancing, rewards, and history improve stability and deployment feasibility.
- Sim-to-Real Training: The system formulates whole-body motion imitation as goal-conditioned reinforcement learning and uses a teacher-student framework because simulation exposes information unavailable from real devices.The teacher receives privileged proprioception and motion goals, while the student is designed for real-world inputs.
- Universal Interface: OmniH2O defines kinematic pose as joint rotations and positions, using it as a universal intermediate representation for tracking real-time motion input.The representation can be supplied by VR headsets, pose generators, cameras, language systems, or autonomous agents.
- Human Motion Retargeting: Retargeted AMASS motions are augmented with fixed standing and squatting lower-body sequences to improve stable standing and squatting.The stable dataset variant fixes the root position and lower body to standing or squatting configurations.
- Reward Design: Regularization rewards and domain randomization train a privileged imitation policy suitable as a teacher for a deployable student policy.The proposed max-feet-height reward encourages standing still and taking large steps instead of balance-maintaining stomping.
- Student Policy: The real-world student uses sparse observations and 25-step history, allowing it to infer velocity without global linear-velocity input.This removes the need for motion capture used in prior work and improves feasibility for in-the-wild deployment.
- Policy Distillation: DAgger distills the privileged teacher by rolling out the student, querying teacher actions from privileged states, and minimizing action discrepancy.The student is updated using the teacher’s reference actions computed from simulated states and reference poses.
- Dexterous Hands: VR-estimated hand poses are converted through inverse kinematics into joint targets for a low-level dexterous-hand controller.The hand-pose interface could also be replaced by motion-capture gloves or RGB cameras.
4 Experimental Results
OmniH2O is evaluated for motion tracking, versatile real-world control, and autonomy from frontier models or teleoperated demonstrations. Across these settings, the system supports whole-body humanoid behavior through shared motion-tracking and learning components.
- Motion Tracking: OmniH2O is evaluated on motion tracking in simulation and in the real world, including 14k augmented AMASS sequences and 20 standing motions.The real-world evaluation is limited to 20 standing sequences because of laboratory space and evaluation difficulty.
- Motion Tracking: The deployable student policy significantly improves motion imitation over prior art while achieving a similar success rate to the privileged teacher policy.This comparison is reported for the first three rows of Table 1.
- Ablations: DAgger improves performance overall, especially for policies using history input, whereas direct RL training makes the student policy struggle to learn.The comparison is reported in the simulation ablation on DAgger and RL.
- Ablations: Policies without explicit linear velocity input outperform policies using VIO or MLP/GRU velocity estimators in the real-world ablation.The result suggests that history-based policies can track motions without explicit linear velocity input; the reported evaluation uses Table 2(a).
- Control and Autonomy: The same motion-tracking policy supports VR and RGB teleoperation, language-generated motion goals, GPT-4o-selected motion primitives, and learning from teleoperated demonstrations.The experiments include dexterous loco-manipulation, robustness tests, frontier-model control, and the OmniH2O-6 dataset with approximately 40 minutes of demonstrations.
5 Limitations and Future Work
OmniH2O supports dexterous whole-body humanoid loco-manipulation through teleoperation, universal control interfaces, scalable demonstration collection, and autonomy. The system remains constrained by odometry quality and the absence of safety guarantees for extreme or out-of-distribution motion goals.
- OmniH2O enables dexterous whole-body loco-manipulation through teleoperation, universal control interfaces, scalable demonstration collection, and autonomy.
- Robot root odometry is required to transfer teleoperation pose estimates into robot-frame motion goals.Noisy or discontinuous VIO can cause motion goals to deviate from desired control.
- The policy has no guarantees or safety checks for extreme disturbances or out-of-distribution motion goals.The paper gives large motion-goal discontinuities as an example.
- Future work includes stair traversal with sparse upper-body goals and learning from demonstrations using additional sensors and improved algorithms.Suggested sensors include LiDAR, wrist cameras, and tactile sensors.
B Simulation Baseline and Ablations
The simulation baselines and ablations vary privileged information, training procedure, observation history, motion-goal keypoints, architecture, and velocity inputs. These variants isolate design choices in the OmniH2O deployment policy.
- The privileged policy uses complete environment, motion-goal, and proprioception information, whereas H2O uses RL without DAgger or history and observes 8 keypoints.
- OmniH2O is the deployment policy trained with DAgger, 3 motion-goal keypoints, and 25-step historical observations.
- Training ablations remove DAgger, historical observations, or both to compare their contributions.
- Motion-goal input ablations compare 22, 8, and 3 keypoints, with 3 keypoints defining the standard OmniH2O setting.
- Additional baselines vary global linear velocity input and include the privileged teacher policy trained with RL.
D LfD Baselines
The LfD baselines benchmark diffusion and behavior-cloning methods under different data, observation, action, architecture, reward, randomization, and state-estimation settings.
- The study benchmarks Diffusion Policy with DDIM or DDPM against behavior cloning for learning from demonstrations.DDIM and DDPM use a single image step to predict 8 action steps; behavior cloning uses ResNet plus MLP.
- Dataset-size ablations train DDPM using 25%, 50%, or 100% of the dataset.
- Input-output ablations compare single-step observations and actions with sequence observations or sequence action predictions.The sequence setting uses 4 image steps and predicts 8 action steps; single-observation sequence-action uses 1 image step and predicts 8 actions.
- Reward components combine penalties, regularization, and task rewards, while domain randomization covers dynamics, perturbations, and terrain.
- The velocity estimator is trained by supervised learning from simulated policy rollouts, using root linear velocity as supervision at each environment step.
H Ablation on Dataset Motion Distribution
The motion-distribution ablation shows that augmentation is important for stable whole-body behavior. Without motion data augmentation, the policy struggles to stand still and perform upper-body movements.
- Without motion data augmentation, the policy is hard to stand still and make upper-body moves.
I Additional Physical Teleoperation Results
Additional demonstrations show OmniH2O’s physical teleoperation capabilities across VR- and RGB-based interfaces, alongside the released whole-body control dataset.
- VR- and RGB-based teleoperation demonstrations provide additional physical showcases.
- The additional physical results are presented as demonstrations of whole-body humanoid control.
- The OmniH2O-6 dataset is presented as a separate whole-body control dataset resource.
J Dataset and Imitation Learning
This section documents the OmniH2O-6 learning-from-demonstration dataset and its autonomous-task evaluation across four tasks.
- Dataset: The OmniH2O-6 dataset contains demonstrations for six learning-from-demonstration tasks, including catching, squatting, hammer catching, and rock-paper-scissors.The Catch-Release, Squat, and Hammer-Catch tasks contain 13234, 8535, and 12759 frames, respectively.
- Dataset: Additional dataset tasks include Boxing and Basket-Pick-Place, which require visually conditioned, coordinated hand actions.
- Evaluation: Quantitative performance for four autonomous learning-from-demonstration tasks is documented in Table 17.
K Sim2real Training Hyperparameters
The supplementary materials provide training hyperparameters for reinforcement-learning, DAgger, and learning-from-demonstration components, together with GPT-4o prompt examples for autonomous control.
- Sim2real training: Reinforcement-learning and DAgger policy-training hyperparameters are listed in Table 18.
- Imitation learning: Learning-from-demonstration training uses a diffusion policy, with default hyperparameters listed in Table 19.
- GPT-4o control: The autonomous-control prompts map visual situations to discrete motion choices for boxing and human greetings.The examples specify left- or right-hand actions, remaining motionless, handshakes, and hugs.
- GPT-4o control: GPT-4 can also be prompted to generate motion goals directly rather than select among motion primitives.