Source-linked AI summary
CLOT: Closed-Loop Global Motion Tracking for Whole-Body Humanoid Teleoperation
Tengjie Zhu, Guanyu Cai, Yang Zhaohui, Guanzhu Ren, Haohui Xie, ZiRui Wang, Junsong Wu, Jingbo Wang, Xiaokang Yang, Yao Mu, Yichao Yan
TL;DR
Whole-body humanoid trackers often neglect global pose feedback, leaving long-horizon operation vulnerable to accumulated drift, particularly on full-sized robots. CLOT combines closed-loop localization with a data-driven observation pre-shift strategy, and reports stable, high-fidelity real-world teleoperation.
Problem
Whole-body trackers can omit global pose feedback, allowing drift to accumulate during extended teleoperation, particularly for full-sized humanoids.
Source: §I · pp. 1–2Method
Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Long-horizon whole-body humanoid teleoperation remains challenging due to accumulated global pose drift, particularly on full-sized humanoids. Although recent learning-based tracking methods enable agile and coordinated motions, they typically operate in the robot's local frame and neglect global pose feedback, leading to drift and instability during extended execution. In this work, we present CLOT, a real-time whole-body humanoid teleoperation system that achieves closed-loop global motion tracking via high-frequency localization feedback. CLOT synchronizes operator and robot poses in a closed loop, enabling drift-free human-to-humanoid mimicry over long timehorizons. However, directly imposing global tracking rewards in reinforcement learning, often results in aggressive and brittle corrections. To address this, we propose a data-driven randomization strategy that decouples observation trajectories from reward evaluation, enabling smooth and stable global corrections. We further regularize the policy with an adversarial motion prior to suppress unnatural behaviors. To support CLOT, we collect 20 hours of carefully curated human motion data for training the humanoid teleoperation policy. We design a transformer-based policy and train it for over 1300 GPU hours. The policy is deployed on a full-sized humanoid with 31 DoF (excluding hands). Both simulation and real-world experiments verify high-dynamic motion, high-precision tracking, and strong robustness in sim-to-real humanoid teleoperation. Motion data, demos and code can be found in our website.
I. INTRODUCTION
CLOT addresses global-pose drift in long-horizon humanoid teleoperation with closed-loop localization feedback and a training strategy for smoother global corrections. The paper reports stable, varied real-world teleoperation and introduces a curated motion dataset, Transformer policy, and adversarial motion prior.
I. INTRODUCTION
- Whole-body trackers enable coordinated motion but often omit global pose feedback, allowing drift during extended operation, especially on full-sized humanoids.Earlier teleoperation approaches also faced a trade-off between agility and stability; one closed-loop system did not support full whole-body tracking.Source: §I · pp. 1–2
- CLOT uses high-frequency localization feedback to close the loop between human and robot global poses, with online motion retargeting through Pinocchio inverse kinematics.The approach trains a general global body-tracking controller from motion-capture trajectories.Source: §I · p. 2
- Observation Pre-shift randomizes future target poses in the observation while keeping tracking rewards aligned to the current time, teaching motion interpolation from data.The strategy is intended to avoid aggressive corrections associated with directly applying global tracking rewards.Source: §I · p. 2
- CLOT’s training combines over 20 hours of curated human motion, a Transformer policy, and an adversarial motion prior to suppress unnatural artifacts.The collection protocol excludes unstable behaviors such as tiptoeing and excessive center-of-mass deviation.Source: §I · p. 2
- CLOT maintains long-term stability during extended teleoperation and performs varied loco-manipulation tasks, with diverse skills and disturbance rejection in real-world tests.The policy was deployed on a full-sized humanoid with 31 DoF.Source: §I · p. 2
A. Learning-Based Humanoid Whole-Body Control
- Recent unified trackers commonly draw on public motion datasets, while CLOT’s in-house data is curated for humanoid dynamics and task-agnostic teleoperation conditions.Many open-source datasets were designed primarily for animation and can differ from physical humanoid kinematics and dynamics.Source: A. Learning-Based Humanoid Whole-Body… · p. 2, §I · p. 2
B. Humanoid Teleoperation Systems
Prior teleoperation systems moved from decoupled upper- and lower-body control toward whole-body tracking, but global drift remains a challenge in extended operation. The paper formulates tracking as goal-conditioned reinforcement learning and describes a motion dataset designed for humanoid control.
B. Humanoid Teleoperation Systems
- Whole-body tracking enables more fluid and diverse human-motion mimicry than joystick- or exoskeleton-based decoupled control, but local-frame tracking can accumulate global drift during extended operation.CLOT incorporates real-time global pose feedback into whole-body tracking.Source: B. Humanoid Teleoperation Systems · pp. 2–3
A. Problem Formulation
- The tracking task is a goal-conditioned reinforcement-learning problem in which proprioceptive state and target motion guide joint-target actions and scalar rewards.The policy is trained using Proximal Policy Optimization (PPO) [19].Source: A. Problem Formulation · p. 3
- The teleoperation pipeline includes human-motion collection, motion retargeting, reinforcement-learning policy training, and closed-loop system construction.Hand and finger motions use direct PD tracking of retargeted joint references.Source: A. Problem Formulation · p. 3
B. Human Motion Data Collection
- The dataset contains approximately 20 hours of motion-capture data, spanning basic locomotion and high-dynamic whole-body movements, with attention to physically consistent contacts and humanoid constraints.The collection explicitly considers foot-contact consistency and humanoid actuation limits.Source: B. Human Motion Data Collection · p. 3
C. Real-time Motion Retargeting
The paper describes real-time human-to-humanoid motion retargeting and RL training components aimed at accurate, safe tracking. The framework combines whole-body IK, history-based observations, tracking rewards, and simulation randomization.
C. Real-time Motion Retargeting
- Real-time retargeting minimizes errors between robot frames and corresponding human keypoints, then clamps computed velocities and integrated joint positions for safety.Pinocchio solves the whole-body IK; v denotes joint velocity and e_i the tracking error for each task.Source: C. Real-time Motion Retargeting · p. 3
D. Basic Components for RL Training
- A history buffer of length H=10 stacks recent proprioceptive states, body-position features, and actions to mitigate partial observability.The actor–critic observation design is asymmetric.Source: D. Basic Components for RL Training · p. 3
- Tracking rewards encourage accurate teleoperation while penalizing unsafe behavior and constraint violations, with tracking terms decaying exponentially with error.The error tolerance σ is adaptively optimized during training, producing smooth gradients and promoting stable optimization.Source: D. Basic Components for RL Training · pp. 3–4
- Physical parameters and observation conditions are randomized to reduce the simulation-to-reality gap.The randomized factors include body mass and inertia, joint damping and friction, actuator strength, contact properties, and observation noise.Source: D. Basic Components for RL Training · p. 4
- Curriculum learning addresses the difficulty of early policy learning under humanoid robots’ high-dimensional action spaces and complex physical constraints.Detailed curriculum settings are provided in supplementary material.Source: D. Basic Components for RL Training · p. 4
1) Adaptive Sampling:
The training design combines difficulty-aware motion sampling with randomized observation pre-shifts. The pre-shift separates look-ahead policy observations from current-step reward evaluation to encourage smooth interpolation while preserving trajectory fidelity.
1) Adaptive Sampling:
- Difficulty-aware resampling samples more challenging motion instances more frequently, while clipping maintains coverage of all motions.The strategy is intended to improve generalization across varying motion complexities.Source: 1) Adaptive Sampling · p. 4
E. Key Design Components for RL Policy Training
- Global tracking objectives can make recovery behaviors overly aggressive when training encourages exploration through random initial-state perturbations.The policy-training approaches target generalization and high-dynamic performance on full-sized humanoid robots.Source: 1) Observation Pre-shift · p. 4, E. Key Design Components for RL Policy… · p. 4
1) Observation Pre-shift:
- Randomized Observation Pre-shift shifts the reference observation window forward by a sampled temporal offset while keeping tracking reward aligned with the original target at timestep t.The shift is applied during training and disabled at inference; the mismatched references encourage a smooth transition manifold across time.Source: 1) Observation Pre-shift · pp. 4–5
2) Transformer-Based Actor–Critic Network:
The policy uses a Transformer actor–critic and an adversarial motion prior to support stable, realistic whole-body control. The real-world system streams retargeted operator motion and robot pose feedback, and the experiments assess morphology generalization, component effects, and long-horizon teleoperation.
2) Transformer-Based Actor–Critic Network:
- Transformer actor–critic networks attend across historical states, current proprioceptive feedback, and tracking goals to capture spatiotemporal dependencies for long-horizon control.Observation components are embedded as tokens and processed by a Transformer encoder with global self-attention.Source: 2) Transformer-Based Actor–Critic… · p. 5
3) Motion Prior reward:
- Adversarial motion-prior learning regularizes policy behavior toward human-like styles, addressing over-aggressive corrections and excessive joint torques from handcrafted rewards.The composite reward balances stylistic consistency with task-level tracking objectives.Source: 3) Motion Prior reward · p. 5
F. Closed-Loop Real-World Teleoperation System
- The real-world pipeline streams retargeted operator motion and uses high-frequency robot global-pose estimates for closed-loop synchronization.The motion-capture system records human motion at 120 Hz; the policy runs at 50 Hz and joint commands are executed by a 400 Hz PD controller.Source: F. Closed-Loop Real-World… · p. 5, Figure 5 · p. 5
IV. EXPERIMENTS AND RESULTS
- The experiments pose questions about cross-morphology generalization, component contributions to performance and stability, and long-horizon real-world loco-manipulation.These passages state the evaluation questions, not their outcomes.Source: §IV · p. 5
A. Simulation Experiment
In simulation, CLOT is evaluated on held-out motion across two humanoid platforms, against TWIST2 and through component ablations. It reduces tracking error versus TWIST2, while the ablations highlight the roles of observation pre-shift and temporal modeling in stable correction.
A. Simulation Experiment
- Simulation uses MuJoCo, Unitree G1 and PNDbotics Adam Pro, and a 30-minute held-out motion-capture test set spanning walking, running, dancing, squatting, turning, and jumping.Tracking accuracy and control smoothness are assessed with body-, root-, and joint-level errors and torque-based measures.Source: A. Simulation Experiment · p. 5
1) Comparison with Prior Methods:
- More than an order of magnitude lower tracking errors than TWIST2 on Unitree G1, with smoother and more stable joint-level control.The comparison uses TWIST2’s officially released pretrained weights.Source: 1) Comparison with Prior Methods · pp. 5–6
2) Ablation Study:
- With only the Transformer enabled, the policy has larger tracking errors and higher torque statistics than the full system; the ablations also identify motion prior and pre-shift as important components.Observation pre-shift helps avoid over-aggressive, unstable corrections under large trajectory deviations.Source: 2) Ablation Study · p. 6
- Policies with observation pre-shift show lower torque and joint-velocity peaks and smoother, more stable correction under large reference perturbations.The recovery ablation reports lower fluctuation and improved smoothness for AMJT and AMDV.Source: 2) Ablation Study · p. 6, Figure 7 · p. 7
- Under a 2.0-second pre-shift, the Transformer tracks stably, whereas the MLP fails and falls, indicating that pre-shift requires sufficient temporal modeling capacity.The architecture ablation also reports that the MLP fails to converge while the Transformer learns stably.Source: Figure 8 · p. 7
B. Real World Teleoperation
Real-world teleoperation tests cover loco-manipulation tasks and a 30-minute uninterrupted session. CLOT completes the tasks with reported speed and accuracy, while tracking errors remain low and stable throughout the extended run.
B. Real World Teleoperation
- The real-world evaluation uses PNDbotics Adam Pro, a full-sized humanoid, across loco-manipulation, extended-duration continuous teleoperation, and disturbance-robustness experiments.The robot is equipped with dual 6-DoF dexterous hands.Source: B. Real World Teleoperation · p. 7
1) Long-Horizon Loco-Manipulation Performance:
- CLOT completes real-world loco-manipulation tasks with high speed and accuracy, and outperforms an MPC approach on desktop manipulation, which the authors attribute to its expanded workspace.The tasks include tabletop pick-and-place, table-to-shelf bin transfer, and ground-to-table bucket placement.Source: 1) Long-Horizon Loco-Manipulation… · p. 7
2) Long-horizon Teleoperation Stability:
- During a 30-minute uninterrupted loco-manipulation session, global body, root, and joint tracking errors remain consistently low and stable.Tracking errors were recorded at 1 Hz; the authors connect the bounded errors to closed-loop global pose feedback suppressing drift accumulation.Source: 2) Long-horizon Teleoperation Stability · pp. 7–8
3) Robustness Performance under Strong External Disturbances:
Under strong external kicks, the robot recovers balance and resumes motion while joint torques and velocities stay within physical limits. The conclusion presents CLOT as a full-sized-humanoid system and identifies more accessible localization and motion tracking as future work.
3) Robustness Performance under Strong External Disturbances:
- Despite strong external kicks, the robot maintains balance, resumes its trajectory, and keeps joint torques and velocities within physical limits during impact and recovery.The disturbance evaluation is conducted during real-world teleoperation.Source: 3) Robustness Performance under Strong… · p. 8, Figure 11 · p. 8
V. CONCLUSION
- CLOT is validated on a full-sized humanoid, with the conclusion attributing robust, high-fidelity, drift-free deployment to its system components.The conclusion names a data-driven randomization strategy, a 20-hour human motion dataset, and a Transformer-based policy.Source: §V · p. 8
- Lightweight, cost-effective global localization and VR-based human motion tracking are identified as future work to improve system accessibility.The conclusion frames these as planned directions, not capabilities demonstrated in the reported evaluation.Source: §V · p. 9
APPENDIX A RL TRAINING DETAILS
The appendix describes the RL training setup: multi-level tracking and physical-constraint rewards, a shared observation specification, independently sampled domain randomization, and curriculum parameters that become more challenging as tracking improves.
A. Reward Terms and Weights in RL Training
- Reward terms combine multi-level pose and motion tracking with contact and locomotion bonuses, while penalties discourage excessive torques, abrupt actions, instability, limit violations, and unsafe collisions.The positive and penalty terms are grouped in Table V.Source: A. Reward Terms and Weights in RL… · p. 11, Table V · p. 11
B. Observation Setup
- The observation components are specified identically for Unitree G1 and PNDbotics Adam Pro.Their complete specification is summarized in Table VI.Source: B. Observation Setup · p. 11, Table VI · p. 11
C. Domain Randomization Paramters in RL Training
- Domain-randomization parameters are independently sampled from uniform distributions within their specified ranges for PNDbotics Adam Pro.The parameter ranges are summarized in Table VII.Source: C. Domain Randomization Paramters in… · p. 11, Table VII · p. 11
D. Curriculum Learning Paramters in RL Training
- Curriculum training begins with reduced penalties, noise, and randomization, then progressively increases these parameters as tracking performance improves.Parameter-specific adaptation rates and a motion-completion threshold govern updates; the parameters are summarized in Table VIII.Source: D. Curriculum Learning Paramters in RL… · p. 11, Table VIII · p. 11
E. Detailed Neural Network Architecture
The appendix specifies a Transformer actor–critic architecture, fixed PD-controller gains, and the compute platform used for policy training.
E. Detailed Neural Network Architecture
- The policy backbone changes from MLP to Transformer while the discriminator remains an MLP.Table IX reports the architecture parameters.Source: E. Detailed Neural Network Architecture · p. 12, Table IX · p. 12
F. Proportional–Derivative Controller Paramters
- The joint-space PD controller tracks policy-predicted target positions and velocities, with gains fixed across experiments and selected by joint functionality.Table X reports the complete Adam Pro controller parameters.Source: F. Proportional–Derivative Controller… · p. 12, Table X · p. 12
A. Experiment Setup
- Final policy training used MJLab simulation on eight NVIDIA RTX 4090 GPUs for approximately one week.Source: A. Experiment Setup · p. 12
B. Evaluation Metrics
The evaluation metrics cover tracking accuracy at body, root, and joint levels, alongside torque-based control effort and smoothness.
B. Evaluation Metrics
- Global and local mean body-position errors assess tracking of body points, with local positions expressed in the root frame.Source: B. Evaluation Metrics · p. 12
- Root-position error and joint-position error measure tracking accuracy at the base and actuated-joint levels.Source: B. Evaluation Metrics · p. 12
- Mean joint-torque magnitude measures control effort, while temporal torque variability measures smoothness and stability.Smaller mean torque magnitude indicates lower control effort, and smaller torque variability indicates smoother control.Source: B. Evaluation Metrics · p. 12