PaperlayerView exact revision on arXiv

Source-linked AI summary

GAE: General Action Expert for Real-Time Humanoid Teleoperation

Yuefan Wang, Huaicheng Zhou, Xiao He, Zhijie He, Mingchuan Yang, Huayi Zhang, Li Chai, Jinxin Liu, Donglin Wang

arXiv:2609.34233v1cs.RO

TL;DR

Humanoid teleoperation needs to reproduce diverse whole-body motion while staying synchronized despite end-to-end delay. GAE combines multi-source motion data, two-stage policy training, and latency-conditioned anticipation; experiments report diverse real-world behaviors and improved synchronization with anticipation. The system’s current scope excludes explicit environmental perception.

  • Problem

    Humanoid teleoperation must support diverse, dynamic whole-body behavior while maintaining synchronization despite sensing, communication, processing, and actuation delays.

    Source: §I · p. 2

  • Method

    GAE combines a multi-source motion dataset, generator–executor policy training with curriculum domain randomization, and latency-conditioned anticipation adjusted to pipeline delay.

    Source: §I · p. 2, B. Algorithm Pipeline · p. 4

  • Results

    Real-world experiments show GAE teleoperating diverse whole-body behaviors, with the τ = 4 robot more synchronized than τ = 0 and stable performance across anticipation horizons, including 92.1% success at τ = 5.

    Source: B. Real-world Motion Tracking Results · p. 6, A. Simulation Motion Tracking Results · p. 6

  • Takeaways & Limitations

    GAE demonstrates real-time physical-avatar control across varied behaviors on Unitree G1 and Westlake O1 robots.

    Source: §I · p. 3, B. Real-world Motion Tracking Results · p. 6

    The current controller lacks explicit environmental perception, limiting direct adaptation to terrain and obstacles in scenarios such as stair climbing and obstacle traversal.

    Source: A. Limitations and Future Work · p. 7

Abstract

from arXiv · show

Humanoid avatars extend human physical presence beyond the body, enabling people to participate in social, service, and labor activities through remotely operated robots. This requires teleoperation systems capable of realizing diverse and dynamic whole-body behaviors while maintaining responsive human-robot synchronization. We present General Action Expert(GAE), a unified learning framework for general-purpose, low-latency humanoid whole-body teleoperation. To cover diverse human behaviors, GAE builds a large-scale human motion dataset from heterogeneous sources, including videos, animations, and motion capture, followed by standardization and augmentation. GAE then addresses the noise and embodiment mismatch in human motions with a two-stage training paradigm: a privileged generator policy first tracks human motion references in simulation and rolls out feasible humanoid trajectories; a deployable executor policy then learns to track these generated trajectories under curriculum domain randomization. For responsive human-robot synchronization, GAE introduces a latency-conditioned anticipation mechanism that adaptively compensates for end-to-end delay during real-time teleoperation. Simulation and real-world experiments on Unitree G1 and Westlake O1 robots demonstrate that GAE enables humanoids to smoothly mirror diverse, agile, and expressive human behaviors. Project website: https://wangyf0928.github.io/gae-wlrobotics/

I. INTRODUCTION

Humanoid teleoperation must support diverse whole-body behavior while keeping the robot synchronized with its operator. GAE combines large-scale motion data, two-stage policy learning, and latency-conditioned anticipation to address these needs.

I. INTRODUCTION

  • Teleoperation requires both diverse, agile whole-body behavior and tight operator–robot synchronization, since delays can weaken controllability and embodiment.Even small delays can leave the robot responding to outdated motion, particularly affecting precisely timed motions.Source: §I · p. 2
  • Noisy video and animation motions, limited motion-capture data, and human–robot morphology mismatch make stable, scalable learning for dynamic behavior difficult.Retargeting can add unstable or physically inconsistent references, while deployment domain randomization adds stochastic physical constraints.Source: §I · p. 2
  • GAE builds a standardized, augmented multi-source motion dataset and separates reference-motion adaptation from sim-to-real robustness learning through generator and executor policies.The generator turns adapted human references into feasible humanoid trajectories; the executor learns to track them under curriculum domain randomization while taking raw human motion as input.Source: §I · p. 2
  • GAE uses latency-conditioned anticipation to predict future human motion, adjusting its horizon to pipeline delay and maintaining tighter operator–avatar alignment.The framework is evaluated in simulation and on Unitree G1 and Westlake O1 robots, where it mirrors diverse whole-body behaviors.Source: §I · pp. 2–3

II. RELATED WORK

Prior teleoperation approaches can limit coordinated whole-body control, while robust tracking of open-ended motions and responsiveness to real-world latency remain challenges. GAE formulates control around human-motion goals and explicitly trains for delayed observations.

II. RELATED WORK

  • Some teleoperation systems separate upper-body commands from locomotion, limiting coordinated whole-body behaviors; camera-based approaches can also face occlusion and root-motion errors.Sparse VR inputs only partially observe the body, while motion-capture signals still require a low-latency controller for feasible robot actions.Source: §II · p. 3
  • Learning-based humanoid controllers have broadened from task-specific locomotion toward large-scale motion tracking, but open-ended behavior and explicit latency responsiveness remain challenges.GAE targets a general-purpose whole-body controller that explicitly addresses low-latency teleoperation.Source: §II · p. 3

A. Problem Formulation

  • GAE formulates teleoperation as goal-conditioned reinforcement learning: the policy uses robot proprioception and a human-motion goal to output joint-position targets executed by a low-level PD controller.The policy is optimized with PPO.Source: A. Problem Formulation · p. 3
  • Because latency supplies a delayed motion goal, GAE conditions the policy on that goal and the delay while evaluating tracking reward against current human motion.This trains the policy to match current motion from delayed observations, encouraging internal latency compensation.Source: A. Problem Formulation · p. 4

B. Algorithm Pipeline

GAE prepares morphology-aligned human-motion references, uses a privileged generator to produce feasible humanoid trajectories, and trains a deployable executor for robustness and latency-aware tracking.

B. Algorithm Pipeline

  • GAE combines motion-capture, video-estimated, and animation data in a unified SMPL representation, then augments it through mirroring, stitching, and upper–lower-body recomposition.A humanoid-proportioned SMPL proxy is used to fit collected motions into morphology-aligned generator training references.Source: B. Algorithm Pipeline · p. 4
  • The generator policy tracks fitted human references in clean simulation using privileged information, then its rollouts provide feasible robot trajectories.The generator observation includes proprioceptive and goal-state histories plus a future goal window; recorded rollouts include body and joint states.Source: B. Algorithm Pipeline · p. 4
  • The executor learns from generator rollouts under progressively stronger domain randomization, but receives raw-human-motion targets rather than generated trajectories at deployment.Generated trajectories shape its reward; direct raw-motion input avoids an intermediate online retargeting step.Source: B. Algorithm Pipeline · p. 4
  • The executor's anticipation horizon is encoded by shifting human-motion token position indices, and can be varied in training and adjusted to measured deployment latency.A larger horizon directs the policy to track a farther-future motion state from current observations.Source: B. Algorithm Pipeline · p. 4

C. Training Details

GAE uses shared policy architecture and curriculum randomization, then evaluates tracking across motion difficulty and examines data composition and training strategy through ablations.

C. Training Details

  • Both policies use separate MLP encoders, a mixture-of-experts Transformer backbone with temporal embeddings, and an MLP action head.Each policy contains approximately 852M parameters and is trained in Isaac Lab with parallel environments.Source: C. Training Details · p. 5
  • The policies share penalty and motion-tracking reward components, while the executor uses curriculum domain randomization for sim-to-real transfer.Generator tracking reward uses shape-fitted human motion; executor tracking reward uses generator rollouts.Source: C. Training Details · p. 5

IV. EXPERIMENTS

  • The data ablation reports that more training data improves overall tracking and augmentation provides further gains; the figure also compares generator–executor training with training from scratch.The supplied passage does not state the outcome of that training-strategy comparison.Source: Figure 3 · p. 5
  • Evaluation uses held-out Easy, Medium, and Hard motion sets and reports pose- and velocity-space errors plus sequence success rate.Success requires every tracked keypoint to remain within 50 cm of the reference trajectory; the experiments assess motion tracking, anticipation horizons, real-world teleoperation, and embodiment transfer.Source: §IV · p. 5

A. Simulation Motion Tracking Results

GAE tracks increasingly difficult whole-body motions more robustly than SONIC on the Medium and Hard sets, while its anticipation mechanism trades some tracking precision for latency compensation. Ablations also favor the generator–executor training paradigm over training from scratch.

A. Simulation Motion Tracking Results

  • GAE consistently achieves better overall performance than SONIC on the Medium and Hard sets, with higher sequence-level success rates as motion difficulty increases.On the Easy set, both methods reliably track relatively simple whole-body motions with comparable performance.Source: A. Simulation Motion Tracking Results · pp. 5–6
  • At τ = 5, GAE achieves a 92.1% success rate with 100 ms anticipation, while MKPE rises from 0.045 at τ = 0 to 0.056 at τ = 5.Increasing the horizon moderately degrades tracking precision, but overall performance remains stable; the same trained model is evaluated across horizons without retraining.Source: A. Simulation Motion Tracking Results · p. 6
  • The generator–executor approach achieves higher success rates than training from scratch across all three evaluation sets, especially as motion difficulty increases.The authors attribute the gain to separating noisy human-motion references from randomized physical dynamics.Source: A. Simulation Motion Tracking Results · p. 6

B. Real-world Motion Tracking Results

Real-world demonstrations show GAE synchronizing with operators and supporting diverse whole-body behaviors on G1, with adaptation to O1 after embodiment-specific fine-tuning. The current system lacks explicit environmental perception, limiting its applicability to challenging terrain and obstacle interactions.

B. Real-world Motion Tracking Results

  • With τ = 4, corresponding to 80 ms anticipation, the G1 remains more synchronized with the operator than with τ = 0, which visibly lags behind.The comparison uses two Unitree G1 robots controlled simultaneously by GAE.Source: B. Real-world Motion Tracking Results · p. 6
  • Real-world G1 demonstrations span rapid stepping, kicking, single-leg support, large-amplitude movement, kneeling, crouching, crawling, and tool use with dexterous hands.The behaviors are diverse and continuously varying.Source: B. Real-world Motion Tracking Results · p. 6
  • After embodiment-specific fine-tuning, GAE enables the morphologically and mechanically different Westlake O1 to reproduce diverse human whole-body motions.The reported O1 behaviors include dynamic movements and interactive behaviors.Source: B. Real-world Motion Tracking Results · pp. 6–7

A. Limitations and Future Work

  • Without external observations such as RGB images or LiDAR, GAE cannot directly reason about environmental geometry or proactively adapt to terrain and obstacles.This limits applicability to scenarios such as stair climbing, obstacle traversal, and other contact-rich environmental interactions.Source: A. Limitations and Future Work · p. 7

B. Interesting Findings

GAE’s motion-tracking policy can sustain ongoing movement when the robot’s own state, converted through forward kinematics, is fed back as the next target. The observed behavior suggests learned temporal and dynamical structure, not explicit planning or fully autonomous motion generation.

B. Interesting Findings

  • Feeding the robot’s forward-kinematics state back as its next target creates a self-referential loop that can sustain ongoing dynamic behavior without an external motion reference.The policy ordinarily tracks externally provided human motion targets.Source: B. Interesting Findings · p. 7
  • Walking may settle into persistent locomotion, while dancing may evolve into repeated coherent free-style motion, including cradling sway and wide arm swings.The authors suggest the policy captures motion or dynamics priors that help preserve temporal structure; such behaviors may correspond to stable modes of the closed-loop system.Source: B. Interesting Findings · p. 7, Figure 5 · p. 7
  • The self-sustained behavior is not evidence of explicit motion planning or fully autonomous motion generation.The authors instead interpret it as evidence that a motion-tracking policy may implicitly capture part of how physically meaningful motion evolves.Source: B. Interesting Findings · p. 7

APPENDIX

The appendix describes the reward design and training components used for RL policies, along with domain-randomization settings and training hyperparameters. The supplied material gives no specific table values.

A. Training Details

  • The reward combines regularization and safety penalties with whole-body motion-tracking rewards, weighted to balance accuracy, smoothness, and stability.Table II summarizes the reward terms, but the supplied text does not report their individual values.Source: A. Training Details · p. 9, Table II · p. 9
  • Two auxiliary MLP heads predict future motion targets and robot states, encouraging temporal representation and supervision of state transitions and dynamics.The heads attach to the shared Transformer backbone.Source: A. Training Details · p. 9
  • A five-head critic separately estimates penalty, body-position, linear-velocity, angular-velocity, and rotation returns, then sums them for policy optimization.The value heads share a Transformer backbone and use separate MLPs.Source: A. Training Details · p. 9
  • Training randomizes physical properties, actuation parameters, control latency, and external disturbances to improve robustness and facilitate sim-to-real transfer.Table III summarizes the settings; their specific values are not included in the supplied passage.Source: A. Training Details · p. 9, Table III · p. 9
  • The appendix lists rollout length, PPO coefficients, optimization settings, learning rate, desired KL divergence, and initial actor standard deviation as training hyperparameters.Table IV contains the hyperparameters, without values in the supplied passage.Source: A. Training Details · p. 9, Table IV · p. 9

B. Data Processing Details

The data-processing pipeline standardizes collected human motions and fits them to a humanoid-proportioned skeleton. It augments motion coverage through stitching, which also exposes the controller to interpolated targets.

B. Data Processing Details

  • Heterogeneous motions are standardized with the human SMPL skeleton, then shape-fitted to a humanoid SMPL skeleton whose motions serve as generator-policy references.The figure shows the SMPL body models used for shape fitting.Source: B. Data Processing Details · p. 9, Figure 6 · p. 10
  • Three augmentation strategies—mirroring, stitching, and recomposition—expand the diversity and coverage of the training motions.The supplied passage introduces stitching and recomposition as additions to mirroring.Source: B. Data Processing Details · p. 10

1) Stitch Augmentation:

  • Training on interpolated transitions adapts the policy to sparse-keyframe targets and can yield smoother, more stable control when those targets are used at deployment.Such trajectories can have temporal characteristics different from motion-capture data.Source: B. Data Processing Details · p. 10
  • Stitching interpolates between frames with similar body configurations from separate clips, creating smooth transitions and novel temporal compositions such as walking-to-running.Figure 7 illustrates walking, running, and their transition.Source: 1) Stitch Augmentation · p. 10, Figure 7 · p. 10

2) Recomposition Augmentation:

Recomposition combines body-part motions from separate sequences to create new whole-body coordination demands. These synthesized combinations expand target coverage and provide challenging training examples.

2) Recomposition Augmentation:

  • Recomposition combines motions of different body parts from separate sequences, such as pairing a standing drinking motion with walking to create walking while drinking.Figure 8 illustrates drinking, walking, and drinking while walking.Source: 2) Recomposition Augmentation · p. 10, Figure 8 · p. 10
  • Because the combined regions interact dynamically, the example requires whole-body coordination to maintain balance and satisfy new movement constraints.Walking perturbs upper-body stability, while upper-body changes also affect balance and locomotion.Source: 2) Recomposition Augmentation · p. 10

C. Additional Real-World Demonstrations

Additional demonstrations test GAE in live, VR-based, long-distance, interrupted-command, disturbed, and cross-embodiment settings. The reported outcomes include smooth tracking, recovery, and stable operation across these scenarios.

C. Additional Real-World Demonstrations

Loading 2609.34233v1…