PaperlayerView exact revision on arXiv

Source-linked AI summary

CLONE: Closed-Loop Whole-Body Humanoid Teleoperation for Long-Horizon Tasks

Yixuan Li, Yutang Lin, Jieming Cui, Tengyu Liu, Wei Liang, Yixin Zhu, Siyuan Huang

arXiv:2506.08931v2cs.RO

TL;DR

Humanoid teleoperation needs coordinated whole-body control with accurate positioning over long horizons, but existing systems decouple body control or operate without real-time feedback. CLONE addresses this with an MoE-based system, closed-loop LiDAR correction, and the CLONED dataset, achieving precise long-range tracking with minimal input while retaining important limitations in fine-grained stability and highly dynamic motion.

  • Problem

    Long-horizon humanoid teleoperation requires whole-body coordination and accurate global positioning, while existing methods struggle with both coordination and accumulated positional drift.

  • Method

    CLONE combines an MoE architecture, LiDAR-based closed-loop error correction, and the CLONED dataset, using minimal head-and-hand MR tracking for teleoperation.

  • Results

    5.1 cm mean tracking error over 8.9 m straight-path trajectories, with a maximum deviation of 12.0 cm, demonstrating mitigation of accumulated errors during extended teleoperation.

  • Takeaways & Limitations

    CLONE enables coordinated whole-body teleoperation over long trajectories, object retrieval from ground level, and robust operation across diverse operators and environments.

  • Takeaways & Limitations

    Minimal head-and-hand input limits fine-grained stability control in some scenarios, while highly dynamic movements such as jumping show reduced performance.

Abstract

from arXiv · show

Humanoid teleoperation plays a vital role in demonstrating and collecting data for complex humanoid-scene interactions. However, current teleoperation systems face critical limitations: they decouple upper- and lower-body control to maintain stability, restricting natural coordination, and operate open-loop without real-time position feedback, leading to accumulated drift. The fundamental challenge is achieving precise, coordinated whole-body teleoperation over extended durations while maintaining accurate global positioning. Here we show that an MoE-based teleoperation system, CLONE, with closed-loop error correction enables unprecedented whole-body teleoperation fidelity, maintaining minimal positional drift over long-range trajectories using only head and hand tracking from an MR headset. Unlike previous methods that either sacrifice coordination for stability or suffer from unbounded drift, CLONE learns diverse motion skills while preventing tracking error accumulation through real-time feedback, enabling complex coordinated movements such as ``picking up objects from the ground.'' These results establish a new milestone for whole-body humanoid teleoperation for long-horizon humanoid-scene interaction tasks.

1 Introduction

Humanoid teleoperation must coordinate the whole body while preserving movement accuracy and global positioning over extended tasks. CLONE addresses these challenges with MoE-based coordination, closed-loop correction, and minimal MR input.

  • Motivation: Whole-body humanoid teleoperation remains difficult because long-horizon tasks require coordinated movement, sustained accuracy, and global positioning.These demands create a capability gap between human operators and humanoids.
  • Challenges: Existing systems often decouple upper- and lower-body control, limiting natural synergies needed for actions such as reaching while walking.Motion-capture-based approaches may also produce conservative motions constrained by training distributions.
  • Challenges: Without real-time feedback, small pose errors compound over successive steps, degrading spatial awareness and potentially causing task failure.The problem is especially acute when manipulation requires precise positioning relative to environmental objects.
  • CLONE: CLONE combines an MoE architecture for diverse motion skills with LiDAR-based closed-loop correction to prevent positional drift accumulation.The system is designed for long-horizon humanoid tasks requiring whole-body coordination and accurate positioning.
  • Results: Using only head and hand tracking from a commercial MR headset, CLONE supports coordinated whole-body motion, minimal positional drift, and complex tasks such as retrieving objects from the ground.The approach also reports robust performance across operator configurations and environmental conditions.

2 Related Work

Prior humanoid teleoperation and control methods face limitations in motion diversity, whole-body coordination, and long-horizon operation. CLONE combines tailored motion data, teacher-student learning, MoE specialization, and LiDAR-based state feedback to address these gaps.

  • Whole-Body Humanoid Teleoperation: Existing whole-body teleoperation methods struggle to reproduce the full diversity and fluidity of human motion, partly because monolithic policies face conflicting motion objectives.Walking, crouching, and reaching require different control strategies.
  • Long-Horizon Loco-Manipulation: Humanoid teleoperation remains largely limited to short-horizon motion replication and open-loop operation because real-time global state estimation is difficult for bipedal systems.This distinguishes humanoid teleoperation from longer-horizon work on fixed-base, mobile, and aerial manipulators.
  • Datasets for Training Humanoids: Existing motion-capture datasets underrepresent robotics-relevant kinematic configurations and dynamic transitions despite containing semantically distinct actions.This limits their coverage of motions needed for humanoid control.
  • Whole-Body Control: Whole-body control coordinates multiple competing tasks, traditionally through hierarchical optimization and more recently through learning-based policies.CLONE treats the humanoid’s degrees of freedom as a unified whole-body control policy.
  • CLONED Dataset: CLONED augments AMASS through motion editing and additional human motion-capture data tailored to humanoid controllers.The expansion increases coverage of motions and transitions relevant to humanoid control tasks.

3 The CLONE Framework

CLONE combines sparse head-and-wrist control with teacher-student learning, MoE-based motion coordination, closed-loop positional correction, and robustness-oriented training to support whole-body teleoperation.

  • Control Interface: CLONE uses only the head position and both wrists’ 6D poses from an Apple Vision Pro headset as its control interface.These three tracked points provide the complete operator input without additional hardware or complex calibration.
  • Policy Learning: A teacher-student strategy distills privileged-information policy behavior into a student policy using real-world observations.The teacher uses comprehensive state information, while the student relies on onboard temporal state observations.
  • Mixture of Experts: The MoE architecture routes inputs to specialized experts, allowing one policy to handle conflicting motion patterns such as walking, crouching, and reaching.A balancing loss encourages more uniform expert selection and helps prevent collapse to only a few experts.
  • Closed-Loop Correction: Closed-loop correction compares humanoid and teleoperator global positions so the student policy can reduce positional drift during extended operation.FAST-LIO2 combines IMU and LiDAR data for real-time state estimation, and the policy consumes the position difference directly.
  • Robustness and Lower-Body Behavior: Velocity-dependent SDE noise models LiDAR odometry errors during training, while AMP rewards regularize lower-body behavior from upper-body references.The noise increases with movement speed and includes a minimum level; forward kinematics propagates randomized head motion to other body positions.
  • Training Data: CLONED combines 149 edited AMASS sequences with 14 custom Xsens motion-capture sequences to expand coverage of whole-body motions and transitions.The dataset is designed to support robust generalization for humanoid controller training.

4 Real-World Experiments

Real-world experiments on a Unitree G1 evaluate global positioning, whole-body skill tracking, and long-horizon mixed navigation. CLONE maintains accurate tracking across extended paths while coordinating diverse motions.

  • Global Position Tracking: 5.1 cm mean error was achieved in straight-path tracking, with 12.0 cm maximum deviation at 8.9 m.The experiment tested target distances of 3 m, 6 m, and 8.9 m with ten repetitions per condition.
  • Global Position Tracking: p > 0.05 across tested distance comparisons indicated no significant tracking degradation from 3 m to 8.9 m.The reported comparisons were 3 m versus 6 m (p = 0.871) and 6 m versus 8.9 m (p = 0.963).
  • Global Position Tracking: 20 cm mean error and 2° mean rotational drift were measured during curved-path tracking.The maximum curved-path error was 27 cm.
  • Whole-Body Motion Tracking: CLONE accurately tracks waving, squatting, standing from squats, and jumping on the Unitree G1.The results emphasize whole-body fidelity for dynamic skills requiring coordinated balance control and force application.
  • Long-Horizon Mixed Navigation: Over a navigation sequence spanning more than 15 m, the robot tracked walking, turning, side-stepping, and returning to its starting position with minimal drift.The sequence combined locomotion and whole-body motion control across an extended teleoperation session.

5 Simulations

Simulation studies examine motion tracking, stance robustness, ablations, and expert behavior. They show that both the MoE architecture and CLONED dataset improve reference tracking, while CLONE favors local fidelity over absolute positioning in difficult stances.

  • Motion Tracking: Both the MoE architecture and CLONED dataset contribute significantly to accurate reference motion tracking relative to the ablations.CLONE† replaces MoE with an MLP, while CLONE∗ uses OmniH2O data instead of CLONED.
  • Tracking Diverse Stances: Across head heights from 1.2 m to 0.6 m, the stance evaluation tests unseen poses ranging from standing to deep squatting.The motions were created by systematically editing CLONED sequences.
  • Tracking Diverse Stances: CLONE underperforms baselines in absolute position accuracy but outperforms them in local motion metrics, velocity error, and hand orientation.All methods show increased tracking errors at lower heights, indicating greater difficulty in squatting postures.

6 Conclusion

CLONE combines closed-loop error correction with MoE-based whole-body control to address tracking errors during long-horizon humanoid teleoperation. It achieves low global-position error over long trajectories while supporting coordinated movements from minimal headset input.

  • 5.1 cm mean global position tracking error over 8.9 m trajectories demonstrates accurate long-horizon positioning.
  • CLONE coordinates diverse whole-body skills, including object retrieval from ground level.
  • The system requires only head and hand tracking from a single commercial MR headset.
  • The MoE architecture and LiDAR-based closed-loop correction jointly address whole-body coordination and accumulated tracking errors.

7 Limitations

CLONE's minimal sensing configuration limits fine-grained stability control in some scenarios, and performance decreases during highly dynamic movements such as jumping. Future work targets richer sensing and expanded dynamic-motion data.

  • The minimal input configuration limits fine-grained stability control in certain scenarios.
  • CLONE exhibits reduced performance during highly dynamic movements like jumping.
  • Training-data constraints and balance-control challenges contribute to reduced performance during jumping.
  • Future work should add sensing modalities and expand motion datasets with specialized reward functions for dynamic behaviors.

A.2 LiDAR Odometry and Closed-Loop Error Correction

The closed-loop system estimates robot and operator global positions with LiDAR and MR odometry, then uses their difference to correct teleoperation drift. Its policy uses a multi-expert architecture for diverse motion skills.

  • FAST-LIO2 combines onboard LiDAR and IMU data to build a real-time 3D map and estimate the humanoid's global position.
  • LiDAR odometry supplies the robot's global position, while Apple Vision Pro odometry supplies the operator's global position for closed-loop correction.
  • The implementation includes reward-structure and domain-randomization details documented in Tables A1 and A2.
  • The student policy uses 3 MoE layers with 4 experts per layer and activates the top 2 experts through router weights.

C.3 Policy Training

Policy training uses IsaacGym with large-scale parallel simulation on A800 and 3090 Ti GPUs. The supplied passages identify training durations and reference the reward and domain-randomization specifications.

  • The teacher policy trains for 1M iterations with 8192 parallel environments, while the student trains for 600K iterations with 4096 environments.
  • Teacher training takes approximately 24 hours on one A800 GPU, while student training takes approximately 48 hours on one 3090 Ti.
  • Table A1 documents the primary reward functions used during training.
  • Table A2 documents the primary domain-randomization scheme used during training.

D.1 Evaluation Metrics

CLONE is evaluated on motion tracking using five metrics covering success, positional accuracy, velocity, and hand orientation. An ablation identifies a three-layer, four-expert architecture with 25 timesteps of history as the strongest overall configuration.

  • Evaluation Metrics: Evaluation uses success rate, mean per-keybody position error, root-relative mean per-keybody position error, average joint velocity error, and hand orientation tracking error.Success requires maintaining balance and staying within the specified average per-keybody distance from the reference motion.
  • Ablation Study: 25 timesteps of history, three MoE layers, and four experts per layer yield optimal performance across most evaluation metrics.The configuration is selected from systematic ablations of history length and MoE parameters.
  • Ablation Study: Shorter histories and more experts can marginally lower root-relative position error while increasing global tracking errors.The ablation indicates a trade-off between local and global motion fidelity.

D.3 Qualitative Results Comparsion

Qualitative comparisons show that CLONE tracks crouching and object-retrieval motions more reliably than CLONE*, while expert activations specialize by motion type. Ablations support three MoE layers and four experts as the final balance of performance and training cost.

  • D.3 Qualitative Results Comparsion: CLONE* fails on crouching and squatting to retrieve objects, whereas CLONE tracks these motions accurately and robustly.CLONE achieves these manipulation skills using about 20% of the data used to train CLONE* despite CLONE* having more than 8k motions.
  • D.3 Qualitative Results Comparsion: Experts specialize by motion type: the first-layer experts favor standing or squatting, while dynamic motions activate all four experts collaboratively.Similar specialization appears in later layers with less variation across motion categories.
  • D.4 The Choice of the Number of MoE Layers and Number of Experts: With N = 8, only half of the experts activate in each layer, indicating redundancy for the current training-data distribution.The analysis reports that four experts are sufficient for the current data distribution.
  • D.4 The Choice of the Number of MoE Layers and Number of Experts: A single MoE layer can activate different experts but performs worse than CLONE on tracking.The authors attribute the lower performance to insufficient parameters for learning diverse motions.
  • D.4 The Choice of the Number of MoE Layers and Number of Experts: Three MoE layers and four experts are selected to balance training cost and policy performance.Four- and five-layer variants show similar activation patterns, motivating the three-layer choice.
Loading 2506.08931v2…