PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data

Dianyi Yang, Mengya Liu, Zhumei Wang, Yu Liu, Xiaochen Zhang, Yixuan Li, Le Ma, Yuhang Yang, Weiqi Huang, Wei Liang, Baoxiong Jia, Siyuan Huang

arXiv:2610.35075v1cs.RO

TL;DR

Costly real-world demonstrations and the need for visually faithful scenes and executable whole-body motions constrain generalizable humanoid loco-manipulation. Zero-HLM combines Gaussian-guided scene reconstruction with motion–environment co-adaptation for simulation data generation. Across three tasks, simulation-only training achieves a 20-percentage-point gain over real-data training under matched configurations.

  • Problem

    Generalizable humanoid loco-manipulation needs diverse interaction data, but collecting it on physical robots is costly, and simulation requires faithful scenes and executable whole-body motions.

    Source: §1 · p. 2

  • Method

    Zero-HLM reconstructs visually faithful, physics-ready scenes and uses demonstration-anchored whole-body synthesis with rollout-conditioned correction to generate simulation training data.

    Source: §1 · p. 2

  • Results

    Across three tasks, simulation-only training yields a 20-percentage-point gain over real-data training under matched configurations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Expanding simulation coverage improves real-world generalization across spatial configurations without additional real demonstrations.

    Source: §1 · p. 2, §5 · p. 15

    Zero-HLM-Bench approximates task difficulty under matched configurations but does not reliably predict policy rankings.

    Source: §4.4 · p. 12

Abstract

from arXiv · show

Learning generalizable humanoid loco-manipulation remains bottlenecked by costly real-world demonstrations. Simulation provides a scalable alternative, but requires visually faithful, physics-ready scenes and executable whole-body motions. We present Zero-HLM, a real-to-sim-to-real framework that transforms smartphone captures and simulated source demonstrations into scalable training data for humanoid loco-manipulation. Gaussian-guided agentic real-to-sim reconstruction aligns visual and physical scenes; motion-environment co-adaptation uses rollout feedback to correct whole-body references and interaction geometry. We generate 4,800 physically validated demonstrations for zero-shot transfer across three Unitree G1 tasks. Simulation-trained GR00T N1.7 achieves 66.7% mean full-task success, versus 46.7% with real-data training. On generalized configurations, expanded simulation training raises pooled success from 23% for base-trained real-data policies to 69%, without additional real demonstrations. Project page: https://young-bit.github.io/zero-hlm/

1 Introduction

Zero-HLM addresses costly real-world data collection and the scene-reconstruction and execution challenges that limit simulation-based humanoid loco-manipulation. It combines Gaussian-guided real-to-sim reconstruction with whole-body motion–environment co-adaptation to generate training data and support real-world transfer.

1 Introduction

  • Costly physical demonstrations limit generalizable humanoid behaviors, while scene-scale simulation must preserve visual fidelity, interaction geometry, and dynamics.Humanoid locomotion and changing egocentric viewpoints make scene-level reconstruction more demanding than tabletop settings.Source: §1 · p. 2
  • Unified whole-body data generation must adapt source motions across spatial configurations and account for controller deviations that can disrupt object interactions.Separate locomotion and upper-body commands can limit motion naturalness and support for leg-critical interactions.Source: §1 · p. 2
  • Gaussian-guided agentic reconstruction aligns a scene-level digital twin to the real scene, while hybrid rendering combines Gaussian and simulator renderings.Demonstration-anchored synthesis adapts source motions, and execution-conditioned rollout feedback corrects reference motions and scene configurations.Source: §1 · p. 2
  • 4,800 successful trajectories across three tasks are generated with a 1.8× speedup over real-world collection, and simulation-only training gains 20 percentage points over real-data training under matched configurations.Across five extended generalization settings, pooled real-world success increases from 23% to 69% without additional real demonstrations.Source: §1 · p. 2
  • The benchmark covers three long-horizon tasks and nine skill stages, with comparison dimensions including unified policy, physics-ready reconstruction, few-demo adaptation, and leg-critical interaction.Leg-critical interaction refers to direct leg/foot–object interaction, excluding locomotion and squatting.Source: §1 · p. 2, Table 1 · p. 3

2 Related Work

Prior humanoid loco-manipulation work spans decoupled control, unified whole-body tracking, robot-free interfaces, and real-to-sim reconstruction. Zero-HLM combines Gaussian-grounded scene construction with simulation-based whole-body data generation and correction of tracking deviations.

Humanoid Loco-Manipulation

  • Unified whole-body trackers coordinate reference motions through a shared control interface, whereas decoupled systems separately command lower-body control and arms and hands.Teleoperation provides embodiment-aligned data, but expanding task and scene coverage requires more operator time and robot rollouts.Source: §2 · p. 3
  • Robot-free and egocentric interfaces reduce dependence on a physical robot but still rely on human-recorded interactions for new settings.Vision-language-action and world-action models also depend on diverse interaction data.Source: §2 · p. 3

Real-to-Sim-to-Real for Robot Learning

  • Gaussian-based real-to-sim methods preserve captured appearance, while agentic approaches construct physical scenes through geometry modeling, parameter inference, and simulator configuration.Zero-HLM grounds agentic construction in Gaussian geometry and matched-view renderings to align physical interaction geometry with the visual scene.Source: §2 · p. 3
  • Demonstration-generation methods range from simulation rollouts and point-cloud transformations to task-specific rewards, decoupled control, and unified control focused on single-object transport.Zero-HLM adapts source interaction skills without specialized motion generators and corrects tracking deviations through motion–environment co-adaptation.Source: §2 · p. 3

3 The Zero-HLM Framework

Zero-HLM combines agentic scene reconstruction with motion–environment co-adaptation to produce scalable humanoid loco-manipulation data. Validated executions are rendered as egocentric observations paired with actions for policy learning.

3.1 Gaussian-Guided Agentic Real-to-Sim Reconstruction

  • The hybrid scene combines a Gaussian reconstruction for appearance and spatial reference with an editable mesh scene for collisions and articulation, refined through matched-view inspection and interaction checks.Interaction-critical assets receive separate video-based reconstruction and physical configuration, including collision geometry, joints, mass, friction, and center of mass.Source: §3.1 · p. 4

3.2 Motion–Environment Co-Adaptation for Scalable Data Generation

  • Starting from 15 successful simulation teleoperation demonstrations per task, the method samples configurations and synthesizes continuous whole-body references by replanning locomotion and adapting source interactions.Locomotion planning uses sampled walking speeds, while leg-critical interactions retain demonstrated lower-body motion.Source: §3.2 · pp. 4–5
  • Rollout feedback corrects planar root errors by transforming scene assets and local grasp errors by adjusting interaction geometry or adapting wrist height.Corrected references and environments are re-executed, and only rollouts meeting task-success, stability, and collision criteria are accepted.Source: §3.2 · pp. 5–6

3.3 Visual Observations and Real-World Deployment

  • Validated robot and object states are replayed to render synchronized egocentric views, which are paired with proprioception, task instructions, and recorded actions to train action-chunk policies.Gaussian backgrounds are composited with simulator-rendered robots and objects; predicted motion latents drive the SONIC decoder at deployment.Source: Figure 2 · p. 4, §3.3 · p. 6

4 Experiments

The experiments evaluate the quality, real-world transfer, and generalization of Zero-HLM’s simulation demonstrations, examine sim–real gaps, and establish a shared benchmark for policy families. They report stage-wise and full-task simulation results, with GR00T N1.7 leading in mean full-task success and improving across three tasks as simulation data increases.

4.2.1 Reconstruction Cost and Geometric Accuracy

  • Gaussian guidance reduces symmetric Chamfer distance to LiDAR ground truth from 11.689 to 5.517 cm, while scene construction, physical configuration, and seven interaction assets cost 7h 13m and $22.82.The paper reports that Gaussian guidance outperforms unguided reconstruction and GS-Mesh.Source: Table 4 · p. 9, Table 3 · p. 9, §4.2.1 · p. 9

4.2.2 Generation Success and Collection Time

  • Sim-Ours succeeds on 33%, 34%, and 25% of Tasks A–C and collects 50 successful demonstrations 2.4×, 1.6×, and 1.4× faster than real teleoperation, averaging 1.8×.Sim-Astra succeeds on Tasks A and B at 21.38% and 24.70%, respectively, but produces no successful Task C trajectories.Source: Figure 5 · p. 9, §4.2.2 · p. 9, Figure 6 · p. 10

4.2.3 Data Generation Ablations

  • Removing robot-pose-guided SE(2) correction reduces Task A generation success from 61% to 16%; Gaussian rendering raises real-world full-task success from 3/20 to 7/20.Removing object-level SE(2) correction, Z-axis compensation, or velocity-adaptive transitions yields 49%, 51%, and 55% success; Gaussian rendering also improves picking from 3/20 to 10/20 and walking from 18/20 to 20/20.Source: Table 6 · p. 10, §4.2.3 · p. 10, Table 7 · p. 10

4.3 Direct Sim-to-Real Transfer and Generalization

  • 66.7% mean full-task success across three tasks with Sim-Ours(800) surpasses Real-Only(100) at 46.7%, while Sim-Ours(100) reaches 43.3%.The Sim-Ours(100) and Real-Only(100) mean full-task success rates are comparable; Sim-Ours(800) increases mean success across three tasks.Source: Table 8 · p. 11, §4.3 · p. 11

4.4 Understanding Sim-to-Real Transfer

  • On matched Base configurations, Real-Only(100) achieves 40% real-world versus 31% simulation success, while Sim-Ours(100) achieves 35% and 39%, for sim–real gaps of 9 and 4 percentage points.The reversed policy ranking means similar absolute success rates do not guarantee consistent relative rankings, so the benchmark approximates task difficulty rather than exactly predicting policy rankings.Source: Table 10 · p. 12, §4.4 · p. 12

4.3 Direct Sim-to-Real Transfer and Generalization

  • Sim-Ours(800)∗ raises pooled success across five Expanded settings from 23% for Real-Only(100) to 69%, including 20% to 100% on the object–goal shift.Expanded settings span robot-start shifts, picking and placement locations, and an alternative object–goal pair; Task C uses only Base.Source: §4.3 · p. 11, Table 9 · p. 12

4.4 Understanding Sim-to-Real Transfer

  • Mix-50* achieves 5/5 success in both picking regions with unchanged demonstration count and action supervision, whereas Mix-50 achieves 2/5 upper-region and 4/5 lower-region success.R-25 and S-25 each achieve 0/5 in their respective unseen regions; replacing simulated observations with recorded real images while retaining simulation action labels improves transfer.Source: Table 11 · p. 12, §4.4 · p. 12
  • Adding simulation data to 100 real demonstrations raises full-task success from 9/20 to 11/20 on Task A, 11/20 to 12/20 on Task B, and 8/20 to 9/20 on Task C.On Task A, success rises from 45% to 55%, but both co-training settings remain below Sim-Ours(800); intermediate-stage performance does not improve uniformly.Source: Figure 11 · p. 13, Table 12 · p. 13, §4.4 · p. 13

4.5.3 Evaluation Protocol and Metrics

  • Task B and C evaluations define success through lifting, interaction, and settled placement criteria; Avg. excludes intermediate stages, and sim–real consistency tests the same Task A policies in both settings.The separate consistency comparison uses 100 simulation trials and 20 real-world trials per policy under the Base configuration.Source: §4.5.3 · pp. 14–15

4.5.4 Stage-Wise Benchmark Results

  • 69.7% mean full-task success gives GR00T N1.7 the highest average, while DiT4DiT leads Task B at 80%.The average is computed from full-task success rates across Tasks A–C, excluding intermediate-stage evaluations.Source: Table 13 · p. 15, §4.5.4 · p. 15
  • Simulation-data gains for GR00T N1.7 extend across all three tasks, with larger improvements at smaller dataset sizes and more gradual, non-linear gains at higher counts.The reported trend indicates benefits across distinct whole-body interaction sequences.Source: §4.5.4 · p. 15
  • 90% walking success on Task A did not ensure full-task success: Diffusion Policy achieved only 3% full-task success.The comparison motivates evaluating both intermediate-stage behavior and complete task execution.Source: §4.5.4 · p. 15

5 Conclusions

Zero-HLM uses simulation-generated data to support zero-shot real-world humanoid loco-manipulation and broader generalization across spatial configurations. The conclusion also identifies a remaining visual-domain gap and directions for extending the framework and benchmark.

5 Conclusions

  • Across three long-horizon tasks, simulation-only policies transfer zero-shot to the real world and achieve higher mean full-task success than real-data-trained counterparts.Expanding simulation coverage further improves transfer across spatial configurations without additional real demonstrations.Source: §5 · p. 15
  • The transfer analysis reveals a remaining visual-domain gap, and co-training real and simulation data does not necessarily outperform simulation-only training.The authors identify visual alignment as important for effective co-training.Source: §5 · p. 15
  • Future work will extend evaluation to additional unified whole-body controllers, richer interaction skills, and multiple reconstructed scenes.The authors also plan to examine how tracking accuracy and motion capability affect demonstration quality.Source: §5 · p. 15

Appendix Contents

  • The appendix contents list agentic reconstruction implementation details, experimental details, and additional qualitative results.The experimental material includes reconstruction evaluation, a GPT-6 Astra baseline, and generalization evaluation details.Source: Appendix Contents · p. 19

A Method Implementation Details

The appendix provides implementation details that complement the method, including material on source annotation, selection, reference-motion construction, and reconstruction.

A Method Implementation Details

A.1 Agentic Reconstruction: Implementation Details

The reconstruction appendix traces captured inputs through scene refinement and physical checks, supplementing the main method's reconstruction pipeline.

A.1 Agentic Reconstruction: Implementation Details

  • The appendix directs readers to Section 4.2.1 for reconstruction time and cost.Source: §A.1 · p. 19
  • Figure 14 shows captured RGB views, estimated camera poses, SfM geometry, and Gaussian reconstruction as inputs to the workflow.Source: Figure 14 · p. 19
  • Mesh refinement compares mesh and Gaussian renderings from matched viewpoints, while cabinet-opening and object-release tests check articulation and supporting contacts in IsaacLab.The physical checks complement visual comparison because visual agreement alone does not verify interaction support.Source: §A.1 · p. 19, Figure 14 · p. 19

B Experimental Details

The experimental appendix supplements the results with diagnostics, baseline details, and additional results, while the system setup and evaluation configurations appear in Section 4.1.

B Experimental Details

B.1 Agentic Reconstruction Evaluation

The evaluation compares reconstruction variants with a LiDAR scan in table and cabinet interaction regions. Gaussian guidance improves geometric accuracy and local scene details.

B.1 Agentic Reconstruction Evaluation

  • The comparison measures symmetric Chamfer distance against a handheld Q9000 LiDAR scan within table and cabinet interaction regions.The distance averages the two directional nearest-neighbor distances between sampled mesh and scan points.Source: §B.1 · p. 20
  • 5.517 cm CD for the full reconstruction beats GS-Mesh at 5.894 cm, while removing Gaussian guidance raises error to 11.689 cm.The measurements use symmetric Chamfer distance in table and cabinet interaction regions.Source: §B.1 · p. 20
  • Figure 15 compares LiDAR ground truth, GS-Mesh collision examples, and reconstructions with and without Gaussian guidance.The qualitative comparison includes mesh–Gaussian detail views.Source: Figure 15 · p. 20, §B.1 · p. 20

B.2 GPT-6 Astra Baseline

The Sim-Astra baseline generates successful trajectories for Tasks A and B but none for Task C. Its trajectories often contain pauses and repeated attempts, and policies trained on them show similar hesitation during deployment.

B.2 GPT-6 Astra Baseline

  • GPT-6 Astra receives the same scenes, assets, task configurations, and source demonstrations as Zero-HLM, plus full simulator state.It can autonomously synthesize complete reference motions or issue incremental closed-loop commands using planning, inverse-kinematics, and whole-body-control interfaces.Source: §B.2 · p. 21, Figure 16 · p. 21
  • 100 successful trajectories per task were generated for Tasks A and B, but none for Task C.Results are from the stabilized pipeline after human-assisted debugging and task alignment.Source: §B.2 · p. 21
  • Generated trajectories often exceed one minute versus approximately 30 seconds for real-world demonstrations, with pauses and repeated attempts.Policies trained on these data show similar hesitation in real-world deployment, suggesting temporal irregularities may carry over.Source: §B.2 · p. 21

B.3 Generalization Evaluation Details

The generalization evaluation compares real-data and simulation-data policies across Expanded configurations. Pooled success is higher for Sim-Ours(800)∗ than for Real-Only(100).

B.3 Generalization Evaluation Details

  • Each configuration has five real-world trials, and the comparison is between Real-Only(100) and Sim-Ours(800)∗.Entries report successes out of five; the Changed percentages exclude Base.Source: §B.3 · p. 22
  • The object–goal evaluation uses a separate set of 800 simulation demonstrations, split evenly between its two goal types.Robot-start and tabletop-location evaluations use the same set of 800 simulation demonstrations.Source: §B.3 · p. 22
  • The banana varies across eight Expanded locations with the basket fixed, while the basket varies across two locations with the banana fixed.The alternative object–goal pair is bitter melon and a white plate.Source: §B.3 · p. 22
  • 69/100 successes for Sim-Ours(800)∗ versus 23/100 for Real-Only(100) across five pooled Expanded settings.The five settings contribute 35, 10, 40, 10, and 5 trials, respectively.Source: §B.3 · p. 22

C Additional Qualitative Results

The qualitative results show generated simulation demonstrations across Tasks A–C, pairing simulated rollouts with egocentric observations used for policy training.

C Additional Qualitative Results

  • Figure 17 covers generated simulation demonstrations for Tasks A–C.The examples include locomotion, manipulation, and leg-assisted interactions.Source: C Additional Qualitative Results · p. 23, Figure 17 · p. 23
  • Each example pairs a third-person simulated rollout with its corresponding rendered egocentric observation.The egocentric renderings are the observations used for policy training.Source: C Additional Qualitative Results · p. 23
  • The depicted interactions span locomotion, manipulation, and leg-assisted behaviors.The demonstrations illustrate diverse whole-body skills in simulation.Source: Figure 17 · p. 23

Related papers

Loading 2610.35075v1…