PaperlayerView exact revision on arXiv

Source-linked AI summary

HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking

Liu Cao, Botian Xu, Ruoqu Chen, Jiajun Liu, Mingzhi Pei, Zihao Wang, Zhongyu Li, Mengdi Xu

arXiv:2610.29903v1cs.RO

TL;DR

Whole-body humanoid teleoperation faces latency from optimization-based retargeting and dense or future-frame reference inputs. HuMiT uses a unified tracking policy driven by a compact target of root motion and four sparse keypoint poses; it reports competitive tracking and low latency, with real-world reactive and loco-manipulation demonstrations.

  • Problem

    Teleoperation latency is driven by optimization-based retargeting and policies that rely on dense reference states or future frames, while many approaches also decouple upper- and lower-body control.

    Source: §I · pp. 1–2

  • Method

    HuMiT uses a unified whole-body policy whose deployment input is root height and velocities plus the current-frame poses of four sparse keypoints.

    Source: §I · p. 2

  • Results

    HuMiT reports competitive tracking versus full-reference methods, ∼20 ms communication-to-inference latency, and 65 ms median end-to-end latency, with real-world catching and loco-manipulation demonstrations.

    Source: §I · p. 2, §VI · p. 6

  • Takeaways & Limitations

    On the tested Unitree G1 tasks, compact-reference tracking supports reactive ball catching and coordinated loco-manipulation.

    Source: §VI · p. 6

    Ground-contact motions such as kneeling or crawling are underrepresented and may have degraded tracking or fail; high-payload loco-manipulation is unsupported because training did not include such loading conditions.

    Source: §VI · p. 7

Abstract

from arXiv · show

Real-time whole-body humanoid teleoperation remains challenging due to the computational latency introduced by optimization-based retargeting pipelines and future-frame conditioning. Existing motion tracking policies typically require dense reference targets including full joint positions, body-part transformations, and even future frames, which tightly couples the controller to expensive upstream modules. In this work, we present HuMiT, a whole-body teleoperation system built on a unified tracking policy that requires only a compact reference target, consisting of root height, root velocity, and the SE(3) poses of four sparse keypoints at the current frame, yet achieves competitive tracking fidelity compared to methods relying on more diverse reference states including complete joint positions, dense body-part transformations, and even future frames. Experiments show that our policy achieves a communication-to-inference latency of $\sim$20 ms measured between headset streaming and policy output and a video-based optical flow latency of 65 ms (median) across different motion types. We validate the system on a Unitree G1 humanoid, showcasing split-second reactive responses such as catching a tossed ball, as well as coordinated loco-manipulation capabilities. Project page: https://xiaohu-art.github.io/Track/.

I. INTRODUCTION

HuMiT addresses teleoperation latency and coordination challenges with a whole-body policy that tracks a compact reference target. It reports competitive tracking, low latency, and real-world reactive and loco-manipulation skills.

I. INTRODUCTION

  • Teleoperation offers a way to collect diverse demonstrations and let human operators intervene in high-risk or likely-to-fail situations.Humanoid morphology suits human-centric workspaces and tools.Source: §I · p. 1
  • Existing systems face latency from optimization-based retargeting and dense or future-frame reference inputs, while upper/lower-body decoupling restricts dynamic whole-body coordination.Hardware-centric exoskeleton and marker-based motion-capture solutions also constrain deployment flexibility.Source: §I · pp. 1–2
  • HuMiT’s compact reference target uses root height and velocities plus current-frame poses of four sparse keypoints, yet achieves competitive tracking against full-reference methods.The target removes optimization-based retargeting from the online control loop and avoids future-frame inputs.Source: §I · p. 2
  • ∼20 ms communication-to-inference latency and 65 ms median video-based optical-flow latency are reported while maintaining robust tracking performance.The system removes optimization-based retargeting and future-frame conditioning from the online loop.Source: §I · p. 2
  • HuMiT is validated on a Unitree G1 for reactive ball catching and coordinated loco-manipulation.Shown skills include carrying a water jug, pushing a wheeled chair while walking, sitting, and reaching overhead.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Prior tracking methods vary in reference richness, while teleoperation systems vary in sensing modality and latency. HuMiT is contrasted with these approaches by using a compact reference without future-frame supervision or distillation.

A. Learning-Based Humanoid Whole-Body Control

  • BeyondMimic [2] uses joint positions and velocities, BFM [3] adds body-part local positions, and SONIC [4] and OmniClone [5] include future frames.These methods share the assumption that dense reference targets are needed for high-fidelity tracking, which remains largely unexamined.Source: A. Learning-Based Humanoid Whole-Body… · p. 2

B. Humanoid Teleoperation Systems

  • Exoskeleton systems – capture the upper body but rely on joysticks or pedals for lower-body control, while marker-based suits are accurate but expensive and workspace-constraining.Marker-based systems also require extensive calibration.Source: B. Humanoid Teleoperation Systems · p. 2
  • Camera-based pose estimation offers flexibility and lower hardware cost, but pose estimation followed by optimization-based retargeting makes low-latency, high-frequency teleoperation difficult.HumanPlus and OmniH2O inherit these latency limitations.Source: B. Humanoid Teleoperation Systems · p. 2
  • ExtremControl [20] reports 50 ms latency with optical motion capture and 60 ms with VR, but its deployable policy is distilled from a teacher using up to 32 future reference steps.Unlike ExtremControl, HuMiT uses single-stage training without distillation or future-frame supervision and replaces global pose anchoring with root height and velocity commands.Source: B. Humanoid Teleoperation Systems · p. 2

C. Motion Retargeting

Classical motion retargeting maps human motion to robot commands through per-frame optimization, which adds latency and can produce unstable references. The formulation separates reference generation from policy tracking and asks how compact the reference can be while preserving high-fidelity whole-body tracking.

C. Motion Retargeting

  • Classical inverse-kinematics or nonlinear-optimization retargeting maps human joint positions to the robot’s kinematic structure but adds per-frame latency and risks numerical instability.Reported instability examples include singularities and local minima, which can yield jittery or infeasible reference commands.Source: C. Motion Retargeting · pp. 2–3
  • HuMiT performs retargeting offline during training and directly uses scaled human keypoints as deployment references, removing retargeting latency and instability from the online loop.The overview also eliminates inverse kinematics and future-frame buffering.Source: C. Motion Retargeting · p. 3, Figure 2 · p. 3

III. PROBLEM FORMULATION

  • The human kinematic state comprises root and body-part transformations in SE(3), with body-part poses expressed relative to the root and keypoints selected from a predefined set.The described setup uses 12 keypoints, including hips, knees, ankles, shoulders, elbows, and wrists.Source: §III · p. 3
  • The teleoperation problem is divided into reference generation, which maps human state to a robot-frame target, and motion tracking, which maps observations and that target to joint-space actions.The design question is how compact the target can be while still enabling high-fidelity whole-body tracking.Source: §III · p. 3

IV. METHOD

HuMiT converts sensed human motion into a compact reference through scale calibration, then uses a whole-body policy to generate robot commands. The policy needs only root motion and four sparse keypoint poses, without joint-level targets, other body-part states, or future frames.

IV. METHOD

  • A VR headset captures human motion, and one-time scale calibration produces the compact reference target for a reinforcement-learning whole-body policy controlling the 29-DoF Unitree G1.The calibration maps human keypoint positions to robot body positions in closed form instead of using optimization-based retargeting.Source: A. System Overview · p. 3
  • The tracking policy observes root height, root linear and angular velocities, and root-relative poses of four keypoints: the bilateral wrists and ankles.It does not require joint-level references, remaining body-part states, or future-frame information.Source: A. System Overview · p. 3

B. From Human Motion to Compact Reference Target

HuMiT derives compact references from consumer-grade sensing and maps human motion into robot-frame targets. Its unified tracking policy uses compact targets at deployment while leveraging full reference states during training, then converts policy outputs into joint targets controlled by a PD controller.

B. From Human Motion to Compact Reference Target

  • A PICO headset, handheld controllers, and leg trackers provide a skeleton stream from which HuMiT extracts four keypoint poses and scales human limb positions to the robot frame.The mappings are closed-form and add less than 1 ms of latency.Source: B. From Human Motion to Compact… · pp. 3–4

C. Whole-Body Motion Tracking Policy

  • Training uses PPO in two stages: offline IK retargeting prepares full robot reference states, then parallel-simulation reinforcement learning trains the policy to track compact references.Full references initialize feasible simulator states, provide dense rewards, and serve as additional critic input; the actor never observes them.Source: C. Whole-Body Motion Tracking Policy · p. 4
  • The policy observes robot proprioception and a compact reference containing root height, root velocities, and sparse keypoint poses, while full references provide dense reward signals and critic input.The compact reference omits joint targets, dense states for other body parts, and future frames.Source: C. Whole-Body Motion Tracking Policy · p. 4
  • The actor outputs joint-space offsets that are scaled around a nominal standing configuration, and a PD controller converts the resulting joint targets into motor torques.The controller uses proportional and derivative gains with measured joint positions and velocities.Source: C. Whole-Body Motion Tracking Policy · p. 4

V. RESULTS

The results section is organized around system latency, whether compact references suffice at deployment, and real-world dynamic whole-body teleoperation.

  • The evaluation asks how HuMiT's operator-to-robot latency compares with systems using optimization-based retargeting and future frames.Source: §V · p. 4
  • The evaluation tests whether compact reference targets suffice as deployment-time policy inputs compared with full reference states.Source: §V · p. 4
  • The evaluation tests real-world dynamic teleoperation, including coordinated loco-manipulation and fast reactive response.Source: §V · p. 4

A. End-to-End Latency (Q1)

HuMiT reduces end-to-end latency relative to deployable comparison systems that buffer future frames or retarget online. Its optical-flow evaluation reports a pooled median latency of 65 ms, while internal communication-to-inference timing is about 20 ms and excludes sensing and actuation.

A. End-to-End Latency (Q1)

  • ∼20 ms communication-to-inference latency covers headset packet arrival through policy output, excluding sensing latency and robot actuation response.Source: A. End-to-End Latency (Q1) · p. 5
  • 65 ms pooled median latency (83 ms mean) across 16 matched motion pairs; HuMiT measures 65 ms versus TWIST2's 120 ms under the external optical-flow protocol.The protocol includes physical execution; the table caption identifies it as the end-to-end teleoperation latency measure.Source: A. End-to-End Latency (Q1) · p. 5, Table II · p. 5
  • SONIC's delay drops from 245 ms to 145 ms when lookahead shortens from 10 to 4 future frames, while HuMiT measures 65 ms without buffering or online retargeting.Source: A. End-to-End Latency (Q1) · p. 5

B. Sufficiency of Compact Reference Targets (Q2)

The section evaluates compact-reference sufficiency through comparisons with richer-reference methods and an internal HuMiT reference-richness ablation.

B. Sufficiency of Compact Reference Targets (Q2)

1) Metrics:

The evaluation uses five complementary metrics to assess tracking accuracy and rollout completion. Baselines span different levels of reference richness, with HuMiT-Full providing a controlled comparison against HuMiT.

1) Metrics:

  • Five metrics assess keypoint geometry, joint accuracy, root-velocity fidelity, and rollout completion across evaluated frames.The evaluation covers T frames, 12 keypoints, and 29 actuated joints.Source: 1) Metrics · p. 5

2) Baselines:

  • Five external baselines span reference richness, from SONIC's 10-frame lookahead to ExtremControl's six-link targets.SONIC-LL uses four future frames; ScaleBFM tracks body-link poses over six future frames.Source: 2) Baselines · p. 5
  • HuMiT-Full observes full references while keeping HuMiT's architecture, reward, randomization, and training procedure identical.The HuMiT versus HuMiT-Full comparison isolates reference richness under matched training conditions.Source: 2) Baselines · pp. 5–6

3) Quantitative Results:

Across three datasets, HuMiT achieves competitive tracking fidelity while using the least reference information and no future frames. Real-world tests on a Unitree G1 demonstrate compact-reference pose tracking, reactive catching, and coordinated loco-manipulation.

3) Quantitative Results:

  • On LAFAN1, HuMiT records the highest external-baseline success rate (52.5%) and lowest joint error (E jpe = 0.035 rad); on BONES-SEED, it has the lowest Eang (0.203 rad/s).On AMASS, HuMiT tracks at Ekp = 313.5 mm with an 87.6% success rate; on BONES-SEED, Elin is 0.090 m/s, within 0.001 m/s of the best. World-frame Ekp favors ScaleBFM.Source: 3) Quantitative Results · p. 6
  • Under identical training conditions, HuMiT and HuMiT-Full are nearly indistinguishable on AMASS and BONES-SEED, while dense references still help on dynamic LAFAN1.The most notable gap between compact and full references appears on LAFAN1.Source: 3) Quantitative Results · p. 6

C. Real-World Teleoperation (Q3)

  • On the Unitree G1, the robot reproduces a wide range of operator-driven whole-body poses using only the compact reference target.This real-time tracking demonstration evaluates how closely the robot mirrors live operator motion.Source: C. Real-World Teleoperation (Q3) · p. 6
  • HuMiT catches the tossed ball in 2 of 10 trials, while SONIC-LL and TWIST2 never succeed; remaining failures reflect arm-workspace mismatch, not delayed response.The task requires the robot to react within the ball's flight time.Source: C. Real-World Teleoperation (Q3) · p. 6
  • Across four loco-manipulation tasks, HuMiT maintains balance without degrading manipulation accuracy, while higher-latency baselines require pauses and operator corrections.Decoupled upper- and lower-body controllers are difficult to use for behaviors requiring tightly synchronized arm and locomotion motions.Source: C. Real-World Teleoperation (Q3) · p. 6

VI. CONCLUSIONS AND LIMITATIONS

HuMiT's compact reference target supports competitive whole-body tracking at low latency, with real-world demonstrations of reactive and coordinated loco-manipulation skills. Its reported limitations concern underrepresented ground-contact motions and untrained high-payload conditions.

VI. CONCLUSIONS AND LIMITATIONS

  • HuMiT achieves competitive tracking fidelity with a compact reference target, ~20 ms communication-to-inference latency, and 65 ms median end-to-end latency.Real-world experiments demonstrate catching a tossed ball and coordinated loco-manipulation.Source: §VI · p. 6
  • Training data is dominated by upright locomotion and upper-body gestures, leaving kneeling and crawling underrepresented and prone to degraded accuracy or failure.Source: §VI · pp. 6–7
  • High-payload loco-manipulation, such as carrying heavy objects while walking, is unsupported because the policy was not trained under those loading conditions.Source: §VI · p. 7
  • Broader training distributions and load-aware policy learning are identified as future directions for addressing these limitations.Source: §VI · p. 7
Loading 2610.29903v1…