PaperlayerView exact revision on arXiv

Source-linked AI summary

ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation

Xialin He, Sirui Xu, Xinyao Li, Runpei Dong, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui

arXiv:2603.03279v1cs.ROcs.CV

TL;DR

ULTRA addresses the limited physical fidelity, scalability, and autonomy of humanoid control systems that rely on predefined motion references. It combines physics-driven MoCap retargeting with a distilled multimodal controller and reinforcement-learning fine-tuning, then evaluates the resulting system in simulation and on a Unitree G1. The framework supports dense tracking and sparse goal-conditioned behavior from realistic sensing, including egocentric perception, while outperforming tracking-only baselines in the reported evaluations.

  • Problem

    Existing humanoid-control methods face physically inconsistent or limited retargeted data and often rely on predefined references instead of perception and high-level task specifications.

    Source: §I · p. 2

  • Method

    ULTRA uses physics-driven RL retargeting, teacher-student distillation, multimodal availability masking, a latent skill bottleneck, and RL fine-tuning.

    Source: §I · p. 2, C. Additional Details on Student Policy · p. 12

  • Results

    ULTRA improves interaction fidelity, matches tracking performance, remains robust under distribution shift, and boosts success on out-of-distribution goals while transferring to Unitree G1.

    Source: B. Motion Retargeting · p. 6, C. General Motion Tracking · p. 6, §VI · p. 8

  • Takeaways & Limitations

    A single controller can support dense tracking and sparse goal following across sensing modalities, including real-world deployment without test-time reference motions.

    Source: §I · p. 2, §VI · p. 8

    The method can fail under severe point-cloud occlusion that removes critical geometric cues.

    Source: D. Additional Experimental Details · p. 13

Abstract

from arXiv · show

Achieving autonomous and versatile whole-body loco-manipulation remains a central barrier to making humanoids practically useful. Yet existing approaches are fundamentally constrained: retargeted data are often scarce or low-quality; methods struggle to scale to large skill repertoires; and, most importantly, they rely on tracking predefined motion references rather than generating behavior from perception and high-level task specifications. To address these limitations, we propose ULTRA, a unified framework with two key components. First, we introduce a physics-driven neural retargeting algorithm that translates large-scale motion capture to humanoid embodiments while preserving physical plausibility for contact-rich interactions. Second, we learn a unified multimodal controller that supports both dense references and sparse task specifications, under sensing ranging from accurate motion-capture state to noisy egocentric visual inputs. We distill a universal tracking policy into this controller, compress motor skills into a compact latent space, and apply reinforcement learning finetuning to expand coverage and improve robustness under out-of-distribution scenarios. This enables coordinated whole-body behavior from sparse intent without test-time reference motions. We evaluate ULTRA in simulation and on a real Unitree G1 humanoid. Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception, consistently outperforming tracking-only baselines with limited skills.

I. INTRODUCTION

ULTRA addresses the gap between precise reference tracking and autonomous goal-conditioned control by combining physics-aware retargeting with a unified multimodal controller. It targets whole-body loco-manipulation across changing sensing and supervision conditions.

Motivation

  • Existing controllers often separate dense reference tracking from sparse goal conditioning, creating a precision-flexibility trade-off.Dense policies can fail when references are missing or infeasible, while goal-conditioned policies may lack fine-grained coordination.Source: §I · p. 2
  • Kinematic retargeting can produce physically inconsistent demonstrations, while fixed conditioning structures struggle with partial or diverse supervision.These limitations are especially problematic for contact-rich tasks and shifting deployment observability.Source: §I · p. 2

Contributions

  • ULTRA uses physics-driven reinforcement-learning retargeting to scale motion capture to physically feasible humanoid trajectories and support augmentation without per-motion retraining.The retargeting enforces kinematic, dynamic, and contact constraints during simulation-constrained optimization.Source: §I · p. 2
  • A unified multimodal controller is distilled from a privileged tracker to support dense references, sparse goals, and egocentric perception.Availability masking allows one policy to operate when references or modalities are missing.Source: §I · p. 2
  • The system combines scaled motion data, multimodal control, and reinforcement-learning fine-tuning to expand interaction-state coverage and improve robustness to out-of-distribution goals and executions.The paper evaluates this approach in simulation and on a Unitree G1 humanoid.Source: §I · p. 2, B. Humanoid Whole-body Locomotion · p. 3

A. Task Interface

ULTRA formulates whole-body loco-manipulation as goal-conditioned control under partial observation, with inputs ranging from dense references to sparse task goals. Its training pipeline converts human-object demonstrations into physically feasible G1 rollouts before teacher-student learning and deployment.

Task Interface

  • The policy receives an observation and task conditioning, then outputs joint-position targets executed by a PD controller.Success is defined by satisfying the terminal task objective within a tolerance.Source: A. Task Interface · p. 3
  • Partial onboard sensing motivates a goal-conditioned partially observable Markov decision process formulation.The policy acts from observations produced by an observation function rather than the full underlying state.Source: B. Preliminaries · p. 3

Training Pipeline

  • ULTRA trains through four stages: neural retargeting, privileged tracking, multimodal distillation, and deployment under real sensing.The stages connect physically feasible demonstrations to a student policy operating with sparse goals and available modalities.Source: §IV · p. 3, Figure 2 · p. 4

Neural Retargeting

  • Retargeting uses reference-aware observations containing simulator state, reference quantities, and heading-aligned simulation-reference differences.The observation includes proprioception, contact signals, correspondence-defined references, and object state.Source: A. General Motion Tracking for Neural… · pp. 3–4
  • Episodes begin from a default standing pose, transition from initial-frame stabilization to full tracking, and terminate after falls, excessive deviation, or sustained contact mismatch.The retargeting policy uses an idealized simulation controller and no domain randomization during this stage.Source: A. General Motion Tracking for Neural… · p. 4
  • A single retargeting policy supports anisotropic trajectory scaling and object scaling while preserving physical feasibility without retraining.Interaction and contact rewards correct augmentation imperfections while simulator dynamics enforce feasibility.Source: A. General Motion Tracking for Neural… · p. 4

B. Dense Motion Tracking for Teacher Policy

The dense-tracking teacher uses privileged state and retargeted references, while the student learns to reproduce its behavior from heterogeneous, selectively available modalities. Distillation and reinforcement-learning fine-tuning extend the same controller from tracking to sparse-goal and vision-based control.

Teacher Policy

  • The privileged teacher tracks retargeted rollouts with full state and dense reference residuals, while randomized physics and perturbations teach recovery.This teacher provides supervision through the deployment control interface and actuation limits.Source: B. Dense Motion Tracking for Teacher… · p. 4

Multimodal Student

  • The student uses proprioception, goals, object state, egocentric point clouds, and an availability mask to handle missing modalities.A transformer encoder projects modalities into shared tokens, while the mask gates token processing and crossmodal attention.Source: C. Multimodal Student Policy · p. 5

Distillation

  • The training loop gradually shifts rollouts from teacher to student and queries the teacher on visited states in a DAgger-style procedure.At deployment, the student samples its latent from the prior using only available inputs.Source: C. Multimodal Student Policy · p. 5
  • Distillation matches student actions to teacher actions while aligning a privileged posterior with a prior inferred from masked student observations.Auxiliary reconstruction heads encourage the latent representation to retain task-relevant information.Source: C. Multimodal Student Policy · p. 5

Retargeting Comparison

  • The retargeting comparison shows more stable contacts for ULTRA than OmniRetarget in the contact frame.The figure also contrasts final-frame standing foot placement between the methods.Source: Figure 3 · p. 5

RL Fine-tuning

  • RL fine-tuning combines reference-imitation environments with goal-reaching environments that apply state and goal perturbations.A residual shortcut from the full-body goal to the decoder preserves low-level reference information during decoding.Source: C. Multimodal Student Policy · p. 5

Control Modes

  • Modality masking supports high-fidelity tracking, sparse goal-conditioned control, and vision-based manipulation with the same parameters.Local references are unmasked for tracking, long-horizon goals for goal control, and point clouds for vision-based manipulation.Source: C. Multimodal Student Policy · p. 5

V. EXPERIMENTAL RESULTS

ULTRA is evaluated end-to-end for retargeting, dense tracking, sparse-goal control, robustness, and sim-to-real transfer. The experiments test whether one controller can preserve interaction quality while operating under partial sensing and distribution shift.

Evaluation Goals

  • The evaluation asks whether MoCap can be converted into physically consistent rollouts, whether dense tracking matches privileged supervision, and whether sparse-goal control improves with fine-tuning.It also tests latent skill organization and transfer to a physical Unitree G1.Source: §V · p. 5

A. Experimental Setup

ULTRA is evaluated through simulation-based motion tracking and physical-interaction experiments, using augmented data and comparisons against retargeting and control baselines. Results indicate improved tracking and contact quality, including under out-of-distribution conditions.

Evaluation platforms

  • Experiments use IsaacGym for motion-tracking evaluation, with key results validated in MuJoCo and real trials on a Unitree G1.The setup combines simulation training/evaluation with physical-hardware trials.Source: A. Experimental Setup · p. 5

Data augmentation

  • Zero-shot augmentation scales trajectories and manipulated objects while preserving plausible motion, expanding the corpus by approximately 6× without retraining the retargeter.The augmented dataset uses anisotropic trajectory scaling and novel object scales for out-of-distribution evaluation.Source: A. Experimental Setup · p. 6, Figure 4 · p. 6, B. Motion Retargeting · p. 6

Baselines and metrics

Retargeting quality

  • ULTRA outperforms baselines across nearly all physical-interaction metrics, with lower foot skating, reduced penetration, and near-zero contact floating on Largebox and Suitcase.The reported gains are attributed to physics-aware retargeting that preserves stance and hand-object contact.Source: B. Motion Retargeting · p. 6, Table II · p. 7

Motion tracking

  • ULTRA strongly outperforms tracking baselines on humanoid-object tracking, especially for out-of-distribution motions and object scales.Distillation from a privileged teacher improves success and object errors under partial observation compared with direct reinforcement learning.Source: C. General Motion Tracking · pp. 6–7

D. Goal-Conditioned Following

ULTRA evaluates sparse goal-conditioned following across simulated perception settings and real Unitree G1 deployment. RL finetuning improves robustness to out-of-distribution goals, while learned latents organize control modes and motion semantics.

APPENDIX

The appendix organizes supplementary material into demonstrations, retargeting and teacher-policy details, student-policy training, and experimental setups.

  • Section A describes the supplementary demonstration video.Source: APPENDIX · p. 10
  • Section B provides retargeting and teacher-policy details, including observation and reward design.Source: APPENDIX · p. 10
  • Section C covers student-policy distillation and reinforcement-learning finetuning.Source: APPENDIX · p. 10
  • Section D provides experimental details and setups for simulation and real-world experiments.Source: APPENDIX · p. 10

A. Demo Videos

The supplementary webpage highlights ULTRA’s retargeting, tracking, sparse-goal, and egocentric-perception capabilities across simulation and long-horizon object-centric tasks.

  • The retargeting pipeline learns one policy from all data and performs zero-shot retargeting to unseen rescaled trajectories and objects.Source: A. Demo Videos · p. 10
  • The teacher policy transfers reliably across simulators.Source: A. Demo Videos · p. 10
  • The all-in-one ULTRA model faithfully tracks reference motions for object interactions.Source: A. Demo Videos · p. 10
  • ULTRA supports sparse object-goal commands and fine-grained keyboard-based control, demonstrating strong generalization.Source: A. Demo Videos · p. 10
  • ULTRA completes long-horizon object-centric goals using only egocentric perception.Source: A. Demo Videos · p. 10

B. Additional Details on Retargeting and Teacher Policy

The teacher-policy pipeline combines tracking and regularization rewards with observations spanning simulated state, reference targets, residuals, and interaction-graph features.

Rewards

  • The teacher reward combines tracking terms with smoothness and regularization penalties to support sim-to-real transfer.The total reward includes tracking components and smoothness terms.Source: B. Additional Details on Retargeting… · p. 10

Retargeting

Observation

  • Teacher observations concatenate simulated state, reference targets, simulation–reference residuals, and interaction-graph features.The interaction-graph features encode body–object relationships.Source: B. Additional Details on Retargeting… · p. 10

Architecture

Rewards

  • The tracking reward covers body joints, object pose and velocity, palm-to-surface interaction, and contact matching.The listed terms include body tracking, object tracking, palm-to-surface distance, and contact matching.Source: B. Additional Details on Retargeting… · p. 10

C. Additional Details on Student Policy

The student policy fuses multimodal observations into a latent-variable controller, then combines distillation with PPO finetuning and masked goal rewards for robust deployment.

Observation

  • The student observation combines global state, commands, local state, proprioception, task information, and modality masks.The observation space also includes object position, rotation, and point-cloud features.Source: C. Additional Details on Student Policy · pp. 10, 12

Student Policy Architecture

  • ULTRA uses a 64-dimensional latent z and a modality-fusion transformer to represent task-relevant information under noisy or missing modalities.Each modality is encoded into a shared token space before fusion.Source: C. Additional Details on Student Policy · p. 11

Teacher Inputs

Distillation

  • Distillation uses supervised and latent-variable losses, with a 64-dimensional latent space and scheduled optimization.The student rollout uses 4096 parallel environments and horizon length 8.Source: C. Additional Details on Student Policy · p. 12

RL Finetuning

  • PPO finetuning is applied to the deployable prior-only student while preserving part of the distillation updates.The finetuning reward combines dense goal-reaching, progress, and auxiliary regularization terms.Source: C. Additional Details on Student Policy · p. 12
  • Goal-related rewards and success checks are masked when targets are unobserved.The reward table separately identifies dense proximity, clipped progress, and terminal success terms for visible targets.Source: C. Additional Details on Student Policy · p. 13

Perception Robustness

  • Point-cloud domain randomization includes Gaussian noise, dropout, outliers, occlusion, and camera perturbations.The listed settings include 0.02 m Gaussian noise and 0.15 point-dropout probability.Source: C. Additional Details on Student Policy · p. 13

D. Additional Experimental Details.

The method remains vulnerable to severe point-cloud occlusion, marker occlusion in MoCap control, and overly aggressive or infeasible operator goals.

Limitations

  • Severe point-cloud occlusion can remove critical geometric cues and cause failures under out-of-distribution conditions.The limitation concerns deployment with egocentric perception.Source: D. Additional Experimental Details · p. 13
  • MoCap-driven control is sensitive to marker occlusions, which can create jitter and drift in estimated object pose and unstable tracking.This limitation applies to real-world experiments.Source: D. Additional Experimental Details · p. 13
  • Overly aggressive, discontinuous, or physically infeasible goals can degrade performance.The stated examples include large target jumps and goals inconsistent with current contacts.Source: D. Additional Experimental Details · p. 13

Related papers

Loading 2603.03279v1…