PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSAIC: Bridging the Sim-to-Real Gap in Generalist Humanoid Motion Tracking and Teleoperation with Rapid Residual Adaptation

Zhenguo Sun, Bo-Sheng Huang, Yibo Peng, Xukun Li, Jingyu Ma, Yu Sun, Zhe Li, Haojun Jiang, Biao Gao, Zhenshan Bing, Xinlong Wang, Alois Knoll

arXiv:2602.08594v2cs.RO

TL;DR

Strong simulation tracking does not reliably transfer to sustained hardware teleoperation, where interface and dynamics errors can accumulate. MOSAIC combines a teleoperation-oriented general tracker with small-data residual adaptation; the authors report that this adaptation is the most stable and sample-efficient strategy in their setting.

  • Problem

    Strong simulation tracking does not reliably translate to hardware performance during long-horizon locomotion and sustained teleoperation, where interface- and dynamics-induced errors can compound.

    Source: §I · p. 2

  • Method

    MOSAIC trains a teleoperation-oriented general tracker and distills an interface-specific adaptation policy into it through an additive residual correction module.

    Source: §I · p. 2

  • Results

    Residual adaptation is the most stable and sample-efficient strategy in the authors’ setting, while merging small teleoperation datasets into large-scale training can cause forgetting or gradient dilution.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The system supports both offline motion replay and online teleoperation with a general tracker adapted to interface-specific corrections.

    Source: §I · p. 2

    MOSAIC does not eliminate dependence on reliable low-latency sensing and state estimation.

    Source: §VII · p. 8

Abstract

from arXiv · show

Generalist humanoid motion trackers have recently achieved strong simulation metrics by scaling data and training, yet often remain brittle on hardware during sustained teleoperation due to interface- and dynamics-induced errors. We present MOSAIC, an open-source, full-stack system for humanoid motion tracking and whole-body teleoperation across multiple interfaces. MOSAIC first learns a teleoperation-oriented general motion tracker via RL on a multi-source motion bank with adaptive resampling and rewards that emphasize world-frame motion consistency, which is critical for mobile teleoperation. To bridge the sim-to-real interface gap without sacrificing generality, MOSAIC then performs rapid residual adaptation: an interface-specific policy is trained using minimal interface-specific data, and then distilled into the general tracker through an additive residual module, outperforming naive fine-tuning or continual learning. We validate MOSAIC with systematic ablations, out-of-distribution benchmarking, and real-robot experiments demonstrating robust offline motion replay and online long-horizon teleoperation under realistic latency and noise. Project page: baai-humanoid.github.io/MOSAIC.

I. INTRODUCTION

MOSAIC addresses the gap between strong simulated tracking and reliable hardware teleoperation with a general tracker, small-data residual adaptation, and a modular deployment stack. It brings motion tracking and teleoperation work together in a system intended to preserve broad motion coverage while improving real-robot robustness.

I. INTRODUCTION

  • Interface- and dynamics-induced errors can compound during locomotion, so accurate offline tracking alone does not ensure usable long-horizon teleoperation.MOSAIC combines teleoperation-oriented general tracking with small-data residual adaptation to address this gap.Source: §I · p. 2
  • MOSAIC’s general tracker uses teleoperation-oriented rewards, while residual adaptation injects interface-specific corrections from limited data without sacrificing motion generality.The paper contrasts this with continual fine-tuning, which can underperform through imbalance and generality degradation.Source: §I · p. 2
  • RobotBridge standardizes connections among motion references, policies, backends, and controllers, while the release includes data and training, deployment, and collection pipelines.The stated goals include portability, reproducibility, and fair comparisons.Source: §I · p. 2

II. RELATED WORK

  • Prior work spans generalist tracking, teleoperation, residual adaptation, and large motion priors; MOSAIC focuses on combining these concerns in a deployable system.The related-work discussion emphasizes long-horizon stability and heterogeneous data alongside general tracking.Source: §II · pp. 2–3

III. SYSTEM OVERVIEW

  • MOSAIC uses one learned tracker for offline reference-clip replay and online tracking of live human motion from wearable interfaces.The system overview presents both modes within a unified training-to-deployment pipeline.Source: §III · p. 3

A. Inputs: One-Step Reference

The system uses a shared policy interface across replay and teleoperation, with a multi-source motion corpus and two-level adaptive resampling supporting training. Its overview links training and simulation to real-robot deployment.

A. Inputs: One-Step Reference

  • Both replay and teleoperation provide the policy with only the next reference frame, with live measurements retargeted into robot-space format.The online retargeting pipeline is described as GMR-style [1].Source: A. Inputs: One-Step Reference · p. 3

B. Outputs and Control Execution

  • The policy outputs joint-position targets for a joint-space PD controller, with safety handling and the same control stack in simulation and on the robot.The output description specifies 50 Hz and a 29-dimensional representation.Source: B. Outputs and Control Execution · p. 3

A. Multi-Source Motion Data

  • The corpus contains approximately 64 hours of heterogeneous motion from in-house, public, generated, and interface-specific sources, unified as robot-space data at 50 Hz.Sources include optical and inertial MoCap, AMASS [25], OMOMO [17], and GENMO [18].Source: §IV · p. 3, A. Multi-Source Motion Data · p. 3

IV. MULTI-SOURCE DATA & ADAPTIVE RESAMPLING

  • The system overview connects multi-source training and adaptive resampling to a deployable policy supporting both offline replay and online teleoperation.RobotBridge provides a modular interface for evaluation and deployment across platforms.Source: Figure 2 · p. 4

B. Adaptive resampling

  • Two-level adaptive resampling selects failure-aware time bins within motions and balances difficulty, novelty, and uniform sampling across motions.The sampling scheme is intended to stabilize training across heterogeneous sources and clip lengths while preserving diversity.Source: B. Adaptive resampling · p. 5

A. Motion Tracking Policy Learning

MOSAIC trains general and interface-adaptation tracking policies, then distills their complementary behavior into a residual policy. The experiments examine robust general tracking and adapting with minimal teleoperation data.

A. Motion Tracking Policy Learning

  • The motion-tracking task is formulated as a POMDP and trained with asymmetric PPO, with the critic accessing privileged state during training.The task reward reflects tracking fidelity and teleoperation objectives.Source: A. Motion Tracking Policy Learning · p. 5
  • πGMT uses heterogeneous motion data and world-frame teleoperation rewards, but its local fidelity does not ensure stable long-horizon locomotion or generalization to unseen contact-rich interactions.Incomplete motion coverage and device-induced gait variations are identified as challenges.Source: A. Motion Tracking Policy Learning · p. 5
  • πADAPT learns device-specific noise and gait patterns from a small teleoperation dataset while specializing in localized patterns rather than the broad repertoire of πGMT.It uses the same observations and reward as πGMT.Source: A. Motion Tracking Policy Learning · p. 5

B. Motion-Conditioned Residual Adaptor Distillation

  • Because fine-tuning πGMT risks catastrophic forgetting, MOSAIC adds a residual action refinement while keeping the base policy frozen.Near-zero residual initialization makes early updates conservative and helps avoid sudden degradation of base-policy performance.Source: B. Motion-Conditioned Residual Adaptor… · p. 5
  • Dual-teacher distillation matches the composite student to πGMT actions on general data and πADAPT demonstrations on adaptation data, aiming to preserve general skills while compensating for interface gaps.The student minimizes mean squared error to the respective teacher actions, with regime-specific weighting.Source: B. Motion-Conditioned Residual Adaptor… · p. 5

VI. EXPERIMENTS

  • The experiments ask how to train robust teleoperation-oriented general trackers and adapt them with minimal teleoperation data without sacrificing general tracking robustness.They also investigate why trackers with saturated simulation metrics can underperform in real-world teleoperation.Source: §VI · pp. 5–6

A. Training Setting and Evaluation System

The evaluation measures world-frame tracking fidelity and episode robustness on an out-of-distribution motion benchmark. The experiments compare training choices, including motion-source diversity, to identify designs for precise and robust tracking.

1) Training Setting:

  • Training uses Isaac Lab on eight A100 GPUs, with 30,000 parallel environments per GPU and approximately 48 hours of training.Policies use MLP hidden layers [1024,1024,512,256] with ELU activations.Source: 1) Training Setting · p. 6

2) Evaluation Metrics and Benchmark:

  • The benchmark reports five world-frame tracking errors plus success rate and average episode length, and uses 633 selected Motion-X sequences averaging 19.04 seconds.The sequences are longer than 10 seconds, have maximum root height of at least 0.6 m, and exclude motions requiring environmental support or deemed excessively challenging.Source: 2) Evaluation Metrics and Benchmark · p. 6
  • The reward design combines BeyondMimic tracking objectives with global teleoperation terms and regularizers for contacts, action smoothness, and joint limits or torques.The global terms are inspired by ASAP and KungfuBot.Source: Table III · p. 6

B. Design Principles for Robust General Motion Tracking

1) Ablation on Data Source:

  • On the out-of-distribution test set, increasing motion-source diversity yields consistent gains in tracking fidelity and robustness, indicating better generalization.Fig. 3a compares single- and multi-source data distributions.Source: 1) Ablation on Data Source · p. 6, Figure 3 · p. 7
  • In the high-dynamic comparison, MOSAIC achieves substantial ground clearance at the reference apex, while TWIST and GMT struggle with explosive movements.Fig. 4 presents the methods from left to right as MOSAIC, TWIST, and GMT.Source: Figure 4 · p. 7

2) Ablation on Reward Design:

MOSAIC improves tracking and robustness over representative trackers, while reward and adaptation comparisons examine long-horizon stability. The section identifies teleoperation interface effects—not motion coverage alone—as a key deployment challenge.

2) Ablation on Reward Design:

  • Adding world-frame tracking reduces long-horizon drift and improves stability, with higher success rates and longer episode lengths.The authors attribute gains to better acquisition of dynamic locomotion and an incentive to maintain global trajectories.Source: 2) Ablation on Reward Design · p. 7
  • Pure RL outperforms DAgger-style distillation in tracking fidelity, success rates, and average steps per episode.The reported practical challenge is transferring fine-grained global position and velocity tracking to a deployable student without loss.Source: 2) Ablation on Reward Design · p. 7

4) Comparison with Representative Baseline:

  • MOSAIC improves tracking fidelity and robustness over GMT and TWIST across reported metrics, with especially large gains in global position tracking.On high-dynamic segments, prior methods often stumble or terminate, while MOSAIC maintains stable contacts and smooth state evolution.Source: 4) Comparison with Representative… · p. 7

1) Analysis of the Teleoperation Sim-to-Real Gap:

  • The sim-to-real gap is attributed mainly to interface effects rather than limited motion diversity, including latency, noise, and hardware execution artifacts.VR streaming introduces approximately 400 ms latency, creating a mismatch between operator intent and the robot’s current state.Source: 1) Analysis of the Teleoperation… · pp. 7–8

2) Exploration of Strategies for Rapid Adaptation:

Residual adaptation outperforms fine-tuning and continual learning on held-out real-robot teleoperation while retaining broad tracking performance. Further analyses examine data quantity, motion augmentation, and the scope of the approach.

2) Exploration of Strategies for Rapid Adaptation:

  • On Motion-X-Sub, the Residual Adapter yields the strongest overall tracking and robustness metrics without degrading the general tracker’s broad coverage.The reported result indicates interface-specific corrections can be injected while preserving general tracking.Source: 2) Exploration of Strategies for Rapid… · p. 8
  • On a 10-minute held-out real-robot teleoperation test, the Residual Adapter consistently outperforms fine-tuning and continual learning.Fine-tuning brings marginal position-tracking gains but reduces robustness; continual learning provides negligible improvement, attributed to gradient dilution.Source: 2) Exploration of Strategies for Rapid… · p. 8

3) Further Exploration of the Residual Adapter:

  • Three minutes of adaptation data is insufficient for meaningful improvement; gains emerge at 15 minutes and increase further at 30 minutes.The authors describe modest amounts of high-quality teleoperation data as sufficient to bridge the gap efficiently.Source: 3) Further Exploration of the Residual… · p. 8
  • FLD augmentation improves interpolation within periodic gait patterns but yields only marginal held-out teleoperation gains, with failures on complex OOD motions remaining.In this setting, interface-level adaptation is more effective for deployment robustness than expanding periodic locomotion data.Source: 3) Further Exploration of the Residual… · p. 8

VII. CONCLUSIONS AND LIMITATIONS

  • The conclusions attribute real-robot failures mainly to interface and dynamics gaps, and identify residual adaptation as the most stable and sample-efficient strategy in this setting.The paper contrasts residual correction on a frozen backbone with merging small teleoperation datasets into large-scale training, which can cause forgetting or gradient dilution.Source: §VII · p. 8
  • MOSAIC still depends on reliable low-latency sensing and state estimation, and residual adaptation targets interface shifts rather than every sim-to-real mismatch.The conclusion describes minute-scale stable teleoperation and preserved broad motion generality within this scope.Source: §VII · pp. 8–9

APPENDIX

The appendix describes a heterogeneous motion corpus, its capture and standardization pipelines, and the representation and sampling infrastructure used for training.

A. Dataset Sources

  • The corpus combines public, in-house, and synthetic motions, with approximately 0.5 hours of teleoperation data from each of two interfaces for adaptation and evaluation.Inertial MoCap training data and final teleoperation tests intentionally use different commercial suits, inducing interface-specific latency, noise, and kinematic bias.Source: A. Dataset Sources · p. 11

1) Built-in House Data:

  • Optical capture uses 39 markers at 100 Hz, followed by reconstruction, quality inspection, robot retargeting, and format standardization.The inertial pipeline uses 15 IMUs at 120 Hz and processes AMASS-compatible sequences through retargeting and standardization.Source: 1) Built-in House Data · pp. 11–12

2) Public Motions:

  • AMASS and OMOMO sequences are retargeted and standardized with in-house captures into a consistent robot-space format for training and evaluation.Processing unifies coordinates and units and aligns initial poses to a canonical configuration.Source: 2) Public Motions · p. 12

3) Generated Motions:

  • The synthetic set uses GENMO-generated, planar motions prompted across seven categories and manually curated to retain approximately 800 clips.Prompts require at least 10 seconds; curation removes severe self-collisions, discontinuities, and physically implausible transitions.Source: 3) Generated Motions · p. 13

C. Motion Representation and Sampling Strategy

  • Each motion is stored in robot joint space with body kinematics at a common 50 Hz frame rate, directly matching policy targets and simulator state.The representation is stored in .npz files.Source: C. Motion Representation and Sampling… · p. 13
  • Motions are preloaded into contiguous tensors so batched indexed queries avoid per-step file I/O and support high-throughput sampling.Multi-GPU sharding assigns disjoint motion-file subsets per rank to reduce duplicates and memory overhead.Source: C. Motion Representation and Sampling… · p. 13
  • Two-level adaptive resampling assigns environments to motions and samples within-motion time steps using failure-aware bin probabilities.The command class serves a fixed-horizon future window to the tracking policy.Source: C. Motion Representation and Sampling… · p. 13

1) Fixed-horizon window sampling:

The sampling strategy standardizes policy inputs across variable-length clips and allocates training toward difficult, under-assigned motions and unstable segments.

1) Fixed-horizon window sampling:

  • The policy receives a fixed-horizon reference window, with H = 1 in this setting, keeping its interface consistent across clips of uneven length.Frames are gathered from the current motion and time step, clamped at the clip’s final frame.Source: 1) Fixed-horizon window sampling · p. 13

2) Motion-level sampling: difficulty × novelty × uniform:

  • Motion-level assignment mixes failure-based difficulty, assignment-count novelty, and uniform coverage, with weights ramped up to mitigate early cold-start bias.An optional active pool and periodic remapping improve memory locality and give selected motions repeated training opportunities.Source: 2) Motion-level sampling: difficulty ×… · pp. 13–14

3) Within-motion failure-aware bin resampling (EMA + kernel smoothing):

  • Within-motion adaptive sampling targets failure-concentrated segments, such as sharp turns and contact transitions.The method follows BeyondMimic’s failure-aware bin resampling approach.Source: 3) Within-motion failure-aware bin… · p. 14

D. Observations

The section specifies the tracker’s observations, rewards, training safeguards, and residual adaptor, then describes the hardware control and safety setup.

D. Observations

  • The actor uses five noisy proprioceptive steps, while the critic receives current-frame privileged observations; the torso is the anchor for coordinate transformations.Anchor position and orientation errors are represented relative to the target reference orientation in the robot’s local frame.Source: D. Observations · p. 14, Table S1 · p. 14

E. Rewards

  • Tracking rewards use an exponential error kernel and are combined with regularization penalties in a weighted total reward.The tracking terms cover global and teleoperation-style tracking.Source: E. Rewards · p. 14, Table S2 · p. 15

F. Terminations

G. Domain Randomization

H. PPO Parameters

I. Residual Adaptor

  • The residual adaptor is a lightweight MLP that maps deployable observations to action corrections added to the backbone action each timestep.Its three hidden layers have widths [512, 256, 128], and it shares the backbone’s action space.Source: §I · p. 14, Table S6 · p. 16

J. Real Robot Setup

  • Real-robot tests use a 29-DoF Unitree G1 with a 50 Hz policy and joint-space PD control; gains are set from effective inertia, a 10 Hz natural frequency, and damping ratio 2.0.Source: J. Real Robot Setup · p. 15
  • Joint-specific effort and velocity limits constrain action scaling, with position offsets proportional to 0.25 times the torque limit divided by stiffness.The limits are intended to keep commanded offsets feasible and improve stability during high-dynamic tracking.Source: J. Real Robot Setup · pp. 15–16

K. RobotBridge

RobotBridge provides a common deployment path across robots and simulation, while the section also reports teleoperation latency, actor-architecture comparisons, and FLD training details.

K. RobotBridge

  • RobotBridge separates platform-agnostic policy execution from robot-specific interfaces, allowing the same policy pipeline to run in MuJoCo or on Unitree G1 without code changes.It also supports H1, H1 2, and Adam, though experiments primarily focus on one robot.Source: K. RobotBridge · p. 16

1) Pipeline:

  • The teleoperation pipeline processes VR and inertial-capture streams through retargeting and onboard signal processing, while its layered design permits policy or motion changes without reinitializing the low-level controller.Source: 1) Pipeline · pp. 16–17

2) Delay Analysis:

  • 0.4 s for VR and 0.2 s for inertial motion capture were the average end-to-end latencies over 10 tests; EMA smoothing and interpolation mitigate delay effects.Device-to-workstation transmission and retargeting accounted for 0.267 s and 0.067 s, respectively.Source: Figure S5 · p. 17, 2) Delay Analysis · p. 17

NETWORK ABLATIONS

  • Among MLP, encoder-based, and attention-based fusion, the MLP achieved higher reward and longer episodes; more complex actors added training cost without consistent gains.Source: NETWORK ABLATIONS · p. 17, Figure S6 · p. 18

N. FLD Training

  • FLD is trained self-supervised on roughly two minutes of teleoperated periodic gait data, using temporally normalized cycles and a multi-step reconstruction objective with phase-consistency regularization.Its state includes base velocities and height, projected gravity, and joint positions.Source: N. FLD Training · pp. 17–18

O. Latent-Space Sampling and Motion Synthesis

FLD samples learned gait styles and explicitly propagates phase to synthesize long-horizon periodic locomotion.

OPEN-SOURCE CHECKLIST

The release checklist covers training and deployment code, data, generated motions, and checkpoints, alongside reproducible motion-prompt examples.

OPEN-SOURCE CHECKLIST

  • The planned release includes training scripts and configurations, RobotBridge deployment tools, trained policy and adaptor checkpoints, and reproduction settings.Source: OPEN-SOURCE CHECKLIST · pp. 19–20
  • In-house optical and inertial MoCap plus adaptation data and generated motions will be released in standardized human and G1-retargeted formats with metadata and behavior labels.Source: OPEN-SOURCE CHECKLIST · p. 20
  • Because third-party licensing prevents redistribution of AMASS and OMOMO, the release will provide scripts for retargeting and standardization instead.Source: OPEN-SOURCE CHECKLIST · p. 20

MOTION PROMPT EXAMPLES FOR MOTION GENERATION

  • Motion prompts use an action description with optional style or emotion, specify at least 10 seconds, and place the person on flat ground.Examples span static poses, locomotion, daily activity, martial arts, and dancing.Source: MOTION PROMPT EXAMPLES FOR MOTION… · p. 20

Related papers

Loading 2602.08594v2…