HoloMotion-1 Technical Report

1 Introduction

• In-the-wild video reconstructions provide broad motion diversity, while curated MoCap and in-house sources add cleaner supervision and deployment-relevant coverage.This hybrid regime also brings reconstruction noise, source-domain heterogeneity, and uneven motion quality.Source: §1 · p. 2
• Approximately 40% lower global tracking error than the strongest evaluated baseline is reported for HoloMotion-1.The report attributes this result to integrating heterogeneous motion data, sparse MoE modeling, and deployment-oriented training and inference design.Source: §1 · p. 3
• A sparsely activated MoE Transformer with KV-cache inference preserves model capacity under real-time control constraints, improving inference efficiency by up to 11×.The design addresses the latency bottleneck of dense Transformer inference in closed-loop humanoid control.Source: §1 · pp. 2–3
• Sequence-level PPO trains autoregressive Transformer policies on motion segments, reducing redundant computation on long clips and improving training efficiency by up to 22×.The report evaluates this approach alongside broader generalization and real-robot transfer experiments.Source: §1 · pp. 2–3
• The evaluation covers five unseen motion datasets spanning diverse motion types and capture devices, with direct real-robot transfer and no task-specific fine-tuning.The motivating gap is that existing systems often use limited motion-capture datasets and modest-capacity policies, struggling with unseen styles or degraded sensing.Source: §1 · pp. 2–3

2.1 From Curated Motion Capture to Hybrid Large-Scale Motion Corpora

• Video-reconstructed motion broadens behavioral diversity and scale, while curated MoCap and in-house data complement it with fidelity and deployment relevance.Video reconstruction can introduce artifacts, viewpoint errors, and heterogeneous motion quality.Source: §2.1 · p. 3

2.2 High-Capacity Policies for Real-Time Humanoid Control

• MLPs are efficient for real-time control but limited in explicit sequence modeling; dense MoE and Transformer policies can raise per-step costs beyond latency budgets.Figure 2 illustrates real-robot examples including video-reconstructed dance, contact-rich Kungfu, and teleoperation.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3 Training High-Capacity Sequence Policies

• As model capacity and sequence length grow, existing training paradigms can become computationally expensive on large-scale motion corpora.Improving sequence-policy training efficiency remains a scaling challenge.Source: §2.3 · p. 4

3 HoloMotion-1 Foundation Model

• The control problem is a goal-conditioned POMDP: the robot has a physical state, but the policy receives observations because that state cannot be perfectly measured in the real world.The reference goal may include a short-horizon lookahead of future poses.Source: §3 · p. 5
• Conditioned on processed observations or observation history, the policy selects continuous actions and the environment transitions to a new state.The goal is specified by a reference kinematic state.Source: §3 · p. 5
• A per-step reward measures reference-motion tracking, and learning maximizes expected discounted return under the resulting control formulation.The formulation provides an interface for training on large-scale motion corpora and real-time closed-loop deployment.Source: §3 · p. 5

3.1 Motion Tracking Formulation

• The formulation specifies the observation, reward, termination, domain-randomization, and low-level control components of HoloMotion-1’s motion-tracking task.These components together define the learning problem for the tracking policy.Source: §3.1 · p. 5

3.1.1 Observation

• The actor receives robot-state feedback and reference-motion features, including a 10-step lookahead of future motion targets.The proprioceptive input includes projected gravity, root angular velocity, joint positions and velocities, and the previous action; the lookahead includes reference gravity, root velocity, joint targets, and root height.Source: §3.1.1 · pp. 5–6
• The critic uses clean observations plus privileged simulation information, while the actor is trained on noise-injected observations.The privileged information includes exact anchor-pose differences, heading-aligned reference root states, and precise robot link states; it is unavailable on real robots.Source: §3.1.1 · p. 6

3.1.2 Reward

• A dense reward combines motion-tracking objectives with regularization penalties for control variation, joint acceleration, joint limits, and unstable contacts.Tracking covers root-relative key bodies, root velocity, and five local body points; absolute root-position and orientation rewards are disabled in the main configuration.Source: §3.1.2 · p. 6

3.1.3 Termination

• Episodes terminate at the maximum length or when projected-gravity error, critical-link height error, or pelvis-position drift exceeds its configured limit.The critical links are the pelvis, ankles, and wrists; these termination conditions are intended to prevent unstable rollouts and maintain reference alignment.Source: §3.1.3 · p. 7

3.1.4 Domain Randomization

• Training randomizes observation noise, action delay, initial motion state, contact materials, mass, center-of-mass offsets, and actuator gains.It also applies random pushes and uses rough height-field terrain to expose the policy to disturbances and encourage recovery behaviors.Source: §3.1.4 · pp. 7–8

3.1.5 Action and Low-Level Control

• The policy outputs normalized commands that set joint-position targets around default configurations, which per-joint PD controllers convert into torque commands.This interface lets the learned policy operate in joint-position space while providing a stable low-level actuation layer.Source: §3.1.5 · p. 8

3.2 Sparse MoE Transformer Policy

• The policy uses a causal decoder-only Transformer with sparsely activated MoE layers to map goal-conditioned observations to continuous joint-space control actions.A separate value network estimates expected return during training.Source: §3.2 · p. 9

3.2.1 Tokenization

• Each control-step token combines proprioceptive signals with a fixed look-ahead window of future reference frames, then is normalized and projected into the hidden dimension.The normalization statistics are updated with an exponential moving average during training.Source: §3.2.1 · p. 9

3.2.2 Transformer Backbone

• A decoder-only Transformer processes the most recent C tokens autoregressively, using grouped-query attention and normalization and gating components for efficiency and stability.The backbone also uses a pre-norm residual design, RMSNorm, and rotary positional embeddings.Source: §3.2.2 · p. 9

3.2.3 Mixture-of-Experts

• A reference-routed grouped MoE block combines a shared expert with selected specialized experts, increasing representational capacity while limiting per-token expert computation to k.The design uses a large expert pool but activates only a small number of experts at each control step.Source: §3.2.3 · p. 9

3.2.4 Router

• The router uses reference-motion input alone because low-level robot-state variation can destabilize routing under simulation-to-reality discrepancies.This keeps expert assignments consistent for the same reference motion across simulation and deployment.Source: §3.2.4 · p. 10

3.2.5 Action Distribution

• A lightweight MLP prediction head maps the final Transformer state to a Gaussian action distribution with a learned mean and state-independent log-standard-deviation.The continuous control action is sampled from that distribution.Source: §3.2.5 · p. 10

3.2.6 Auxiliary Losses

• Training-only auxiliary heads supervise base velocity, contact states, and reference and robot key-body positions, while a dead-expert margin regularizer mitigates expert under-utilization.The position losses encourage the pre-MoE representation to encode reference geometry and the robot-specific kinematic realization.Source: §3.2.6 · pp. 10–11

3.2.7 Value Network

• An MLP value network estimates expected return from observations and privileged state information, including noise-free observations and simulation states unavailable at deployment.The privileged signals are used only during training.Source: §3.2.7 · p. 11

3.3 Sequence-Level Policy Optimization

• Standard step-level PPO requires repeatedly storing and recomputing overlapping observation prefixes, creating O(C) computation per sampled step.This becomes a memory and compute bottleneck when scaling to large motion corpora.Source: §3.3 · pp. 11–12
• Sequence-level optimization preserves contiguous rollout segments and computes action log-probabilities for all BT steps in a single batched forward pass, avoiding repeated evaluation of overlapping observation prefixes.This improves training efficiency for long motion sequences.Source: §3.3 · p. 12
• During rollout and real-world deployment, a per-environment KV-cache ring buffer stores context and reduces attention computation from O(C^2) to O(C).The cache is cleared when an episode terminates.Source: §3.3 · p. 12

4 HoloMotion-1 System Pipeline

• The system pipeline spans environment setup, data preparation, distributed training, offline evaluation, policy export, and real-world deployment.It uses explicit intermediate artifacts from prepared motion data through deployment-ready policies and clips.Source: §4 · p. 12, Figure 3 · p. 13
• Motion sources are converted to AMASS-style files, retargeted to the robot, and converted to NPZ trajectories for visual quality inspection before large-scale HDF5 packing.The HDF5 databases support batched motion loading for training and IsaacLab evaluation.Source: §4 · p. 12
• Modular configuration files support distributed PPO training and reuse across datasets, robots, observations, rewards, domain randomization, terrain, and policy architectures.The training stage produces checkpoints and resolved configurations for evaluation and export.Source: §4 · p. 13
• Complementary IsaacLab and MuJoCo evaluation paths support checkpoint or exported-policy testing, while the ROS2 deployment stack runs pretrained or newly trained models on physical robots.The robot runtime supports velocity tracking and motion tracking from offline clips or live teleoperation references.Source: §4 · p. 13

5.1 Experimental Setup

• The system is evaluated on a 29-DoF Unitree G1, with policy training in IsaacLab and benchmark rollouts in MuJoCo.Source: §5.1 · p. 14

5.1.1 Training Corpus

• The hybrid training corpus exceeds 2,000 hours, combining diverse video-reconstructed motions with higher-fidelity curated and deployment-oriented in-house data.MotionMillion supplies video-derived motions; curated MoCap and in-house sources address supervision fidelity and deployment coverage.Source: §5.1.1 · p. 14, Table 4 · p. 14

5.1.2 Training Configuration

• Sequence-level PPO trains policy and value networks on batched trajectory segments from parallel environments to learn from long motion sequences.Source: §5.1.2 · p. 14

5.1.3 Evaluation Datasets

• The evaluation suite spans optical MoCap, RGB-D, VR teleoperation, in-the-wild dance, and inertial teleoperation motions.Together, the datasets cover multiple motion sources and sensing modalities.Source: §5.1.3 · p. 15, Table 5 · p. 15

5.1.4 Evaluation Metrics

• Metrics include global keypoint error, joint-position error, root-velocity error, and rollout success rate; the metrics are macro-averaged across sub-evaluation sets.Mean per-keypoint position error is primary because it reflects both global root accuracy and local articulation fidelity.Source: §5.1.4 · p. 15

5.2 Evaluation Protocol

• The protocol assesses prior-method comparisons, training and inference efficiency, and transfer to real humanoid hardware without task-specific fine-tuning.The released system is evaluated against representative open-source baselines using a consistent MuJoCo suite.Source: §5.2 · pp. 15–16

5.3 Comparison with Prior Methods

• Approximately 40% lower mean per-keypoint position error than Sonic accompanies best performance across evaluation datasets and stable tracking under motion variation and noisy references.Qualitative results also show a wider range of reproduced motion styles and dynamic behaviors than baseline methods.Source: §5.3 · p. 16

5.4 Training and Inference Efficiency

• Sequence-level optimization improves training throughput by approximately 22×, while sparse MoE with KV-cache reduces embedded policy latency by approximately 4×.The training optimization reduces redundant computation from overlapping observation prefixes.Source: §5.4 · p. 17

5.5 Real-World Deployment

• The policy runs at approximately 200–300 Hz on the robot-side platform and transfers to a physical G1 for zero-shot tracking of diverse motions and responsive teleoperation.Demonstrations include dynamic dance, crawling, sitting, and martial arts kicking, without real-world fine-tuning.Source: §5.5 · p. 17

• The training corpus combines large-scale video-reconstructed motions with curated MoCap and in-house sources for diversity, fidelity, and deployment-relevant supervision.Source: §6 · p. 17
• A sparse MoE Transformer, sequence-level policy optimization, and KV-cached inference support long-sequence learning and real-time closed-loop execution as model capacity scales.Source: §6 · p. 17
• Experiments show zero-shot generalization across unseen datasets spanning diverse activities and capture devices, plus direct sim-to-real transfer without task-specific fine-tuning.Source: §6 · p. 17

Imitate Any Pose

• The current Imitate Any Pose stage provides a deployable whole-body policy for offline motion clips and online tracking inputs.It also serves as a scalable training and evaluation baseline.Source: §7 · p. 18

Follow Any Command

• Follow Any Command is the near-term expansion, replacing full reference trajectories with compact inputs such as velocity commands, VR or MoCap, and language instructions.Source: §7 · p. 18

Move on Any Terrain

• Move on Any Terrain targets stairs, slopes, uneven ground, narrow passages, and contact-rich settings, with terrain-aware perception and expanded training.Source: §7 · pp. 18–19

Control Any Embodiment

• Control Any Embodiment targets adaptation across humanoid morphologies using morphology-conditioned representations and robot-specific action decoders.Source: §7 · pp. 18–19