PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

Yingdong Hu, Haodong Zhu, Boyuan Zheng, Yihang Hu, Tong Zhang, Zunhao Chen, Junming Zhao, Ruiqian Nai, Yang Gao

arXiv:2606.22174v1cs.RO

TL;DR

Whole-body humanoid loco-manipulation requires coordinating the full kinematic chain, while existing systems often decouple upper and lower bodies. OpenHLM studies a three-phase recipe spanning teleoperation, VLA design, and heterogeneous co-training. It reports stronger long-horizon performance than two humanoid VLA baselines with less than half the demonstration time.

  • Problem

    Many humanoid systems use separate upper- and lower-body controllers, leaving whole-body coordination and humanoid loco-manipulation underexplored.

    Source: §1 · p. 1, §3.2 · p. 6

  • Method

    The paper conducts controlled experiments across whole-body teleoperation, VLA design, and heterogeneous co-training to develop OpenHLM.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    OpenHLM outperforms GR00T N1.6 and Ψ0 [8] on a long-horizon task spanning the humanoid’s wide vertical workspace, using less than half their demonstration time.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Joint-based whole-body teleoperation and heterogeneous co-training extend the policy to new objects and instructions without additional whole-body teleoperation on those targets.

    Source: §1 · p. 2

    The experiments use only the Unitree G1, so whether the conclusions generalize to other humanoids or more diverse scenes remains uncertain.

    Source: §6 · p. 11

Abstract

from arXiv · show

Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and $\Psi_0$) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].

1 Introduction

The introduction frames whole-body humanoid loco-manipulation as a coordination problem and studies a recipe for controlling the humanoid’s full action space. OpenHLM combines whole-body teleoperation, VLA adaptation, and heterogeneous co-training, with reported gains on training and held-out tasks and a long-horizon baseline comparison.

1 Introduction

  • Existing systems commonly separate upper- and lower-body controllers, limiting whole-body coordination and leaving the lower body as a mobile base.The paper motivates a high-level VLA that maps language and pixels to whole-body commands, tracked by a low-level controller.Source: §1 · p. 1
  • The paper organizes controlled, one-variable-at-a-time experiments around whole-body teleoperation, VLA design, and heterogeneous co-training.The study builds toward a recipe for whole-body humanoid loco-manipulation.Source: §1 · p. 2
  • A joint-level whole-body interface outperforms alternatives such as VR 3-point control, while pretrained VLAs transfer to humanoid whole-body actions despite the embodiment gap.Held-out action MSE is a poor proxy for real-world task progress.Source: §1 · p. 2
  • OpenHLM outperforms GR00T N1.6 and Ψ0 [8] on a long-horizon task spanning the humanoid’s wide vertical workspace, using less than half their demonstration time.It reaches 89% average task progress on 8 training tasks; co-training raises progress on 4 held-out tasks from 33% to 87%, near the 12-task oracle at 94%.Source: §1 · p. 2
  • The study presents OpenHLM as an open-source recipe for VLAs that jointly control the humanoid’s full action space.The stated release includes teleoperation and data-collection code, training and deployment code, model checkpoints, and demonstration data.Source: §1 · pp. 2–3
  • Co-training with stationary teleoperation or HuMI [1] extends the policy to new objects and instructions without additional whole-body teleoperation on those targets.The paper characterizes heterogeneous co-training as a way to extend coverage using cheaper data sources.Source: §1 · pp. 2–3

2 Design Goals and Task Suite

The section defines three design goals for humanoid loco-manipulation and introduces HLM-12, a 12-task benchmark organized into four capability families. It also specifies a real-world evaluation protocol and how task coverage grows across the study.

Design goals

  • A whole-body-native policy commands every humanoid joint, while language steerability and cheap-data extensibility target multi-task use and reduced reliance on costly whole-body teleoperation.The three desiderata are whole-body native control, language-steerable and data-efficient learning per task, and extension through cheap data.Source: §2 · p. 3

The HLM-12 Benchmark

  • HLM-12 contains 12 language-conditioned tasks across four capability families, from locomotion with object placement to manipulation by non-arm body parts and environmentally constrained motion.The families are pick-and-place with locomotion, whole-body workspace extension, using body parts as manipulators, and loco-manipulation under environmental constraint.Source: §2 · p. 3

Evaluation protocol

  • Each policy-task pair is evaluated in five real-world rollouts with shared scene configurations, and each rollout receives a task-progress fraction from 0 to 1.Progress scoring gives partial credit for sub-stages, and reported means include standard errors.Source: §2 · p. 4
  • Evaluation expands from a four-task subset to eight tasks after VLA design, then to 12 tasks for co-training, with an extra long-horizon task for baseline comparison.The added tasks measure whether cheaper data extends the policy beyond whole-body teleoperation coverage.Source: §2 · p. 4

3 Building a Whole-Body Loco-Manipulation System: A Roadmap

The roadmap builds a whole-body humanoid loco-manipulation system through controller and teleoperation design, VLA adaptation, and heterogeneous co-training. Its results support joint-based whole-body teleoperation and show that cheaper data can extend the policy to held-out tasks.

3.1 Low-Level Controller & Teleoperation

  • Joint-based whole-body teleoperation alone completes all three tested tasks, achieving 80%–87% task progress at 10–12 footsteps per rollout; decoupled control cannot perform Bottle Disposal.On Cola Placement, decoupled control averaged 42.3 footsteps, 3.5× joint-based, while VR 3-point reached 40% progress with a 67.8-second rollout.Source: §3.1 · p. 5, Table 1 · p. 5
  • The joint-based interface exposes 32 action dimensions, while VR 3-point exposes 24 and decoupled control 21; the alternatives restrict expressible whole-body actions.The table reports mean progress, rollout duration, and footsteps across five rollouts, and marks tasks that methods cannot perform by construction.Source: §3.1 · pp. 4–5, Table 1 · p. 5
  • Whole-body demonstrations retargeted online to robot joint space outperform native SMPL-based recording, despite comparable operator experience, motion quality, and throughput.The paper attributes the difference to the need for the policy to learn coordination across all joints in the SMPL representation.Source: §3.1 · pp. 5–6, Figure 3 · p. 5
  • A 0.2 s future-frame preview balances locomotion and manipulation, reaching 67% task progress at 35.2 s demonstration duration; 0.6 s preview yields 13% progress.The 0.2 s setting is used for high-quality demonstrations on subsequent tasks.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2 Whole-Body VLA Policy Design

  • At 40 whole-body demonstrations per task, task progress reaches roughly 90%, with the largest scaling gain occurring between 10 and 20 demonstrations.A skilled operator spends about 1.5 hours per medium-difficulty task at the 40-demonstration budget.Source: §3.2 · p. 8
  • The 8-task whole-body VLA reaches 89% average task progress using a π0.5-initialized backbone, weight-surgery action projection, proprioception, absolute joint targets, and multi-step inference.Robot pretraining reaches 91% average progress, versus 60% for PaliGemma and 42% from random initialization; one-step alternatives underperform the 10-step baseline by roughly 20 task-progress points.Source: §3.2 · pp. 7–8

3.3 Heterogeneous Co-Training

  • Stationary co-training raises held-out task progress from 33% to 87% without regressing the 8 training tasks, while HuMI supplies new semantics but not new motions at the current data budget.Stationary data also supports the new vessel-tilt motion needed for Pouring; HuMI matches it on motion-reuse tasks but fails to acquire that motion.Source: §3.3 · p. 9

4 System-Level Comparison on Long-Horizon Tasks

The system-level comparison evaluates OpenHLM with HuMI co-training against two humanoid VLA baselines on a long-horizon task. OpenHLM achieves higher task progress with less than half their operator time.

System-Level Comparison on Long-Horizon Tasks

  • GR00T N1.6 combines a Cosmos-2B vision-language model with a DiT action head, while Ψ0 uses an egocentric-video-pretrained backbone and a humanoid-demonstration-trained action expert.Both baselines use decoupled control; their official checkpoints and recommended fine-tuning protocols are used.Source: §4 · pp. 9–10
  • OpenHLM receives full loco-manipulation teleop on 6 of 20 object pairs and HuMI manipulation demonstrations on the remaining 14; the oracle uses full teleop on all 20.Both baselines receive full loco-manipulation teleop on all 20 pairs; each system uses the same total number of demonstrations.Source: §4 · p. 10
  • 87.5% task progress for OpenHLM (HuMI co-training) exceeds Ψ0 (48.8%) and GR00T N1.6 (57.5%), nearing the teleop oracle (97.5%).OpenHLM uses 1.14 hours of operator time, versus 2.70–2.73 hours for the baselines.Source: §4 · p. 10

5 Related Work

Related work covers whole-body teleoperation, humanoid VLA adaptation, and robot-free data for co-training. The paper positions its study around how these design choices interact in humanoid loco-manipulation.

Related Work

  • Whole-body teleoperation commands all degrees of freedom jointly, while decoupled control restricts lower-body actions to coarse motion commands.Prior work leaves open how teleoperation interfaces interact with VLA-driven policies.Source: §5 · pp. 10–11
  • Humanoid VLA efforts add humanoid data during pretraining or mid-training, but the paper reports that its π0.5-initialized adapted VLA outperforms both compared systems.The authors identify backbone adaptation, teleoperation format, and cheap-data mixing as design questions.Source: §5 · p. 11
  • HuMI extends UMI with wearable body trackers and inverse-kinematics retargeting to full-body humanoid joints; the broader space of robot-free data tools remains open.EgoHumanoid pairs human and teleoperation demonstrations on the same full tasks, whereas this work uses cheap data for new objects and language prompts.Source: §5 · p. 11

6 Discussion, Conclusion and Limitations

The paper presents OpenHLM as a recipe combining joint-based whole-body teleoperation, a π0.5-initialized humanoid-adapted VLA, and heterogeneous co-training. It reports stronger performance than two humanoid VLA baselines on a long-horizon task, while identifying limits in platform scope, robot-free data, and mechanistic understanding.

6 Discussion, Conclusion and Limitations

  • OpenHLM combines joint-based whole-body teleoperation, a π0.5-initialized humanoid-adapted VLA, and heterogeneous co-training; it beats two humanoid VLA baselines with less than half their demonstration time.The comparison is on a challenging long-horizon task spanning the humanoid’s large vertical workspace.Source: §6 · p. 11
  • Experiments use only the Unitree G1, leaving it uncertain whether the conclusions generalize to other humanoids or more diverse scenes.Source: §6 · p. 11
  • HuMI better grounds new objects and language prompts than novel motion patterns; visual and action gaps or limited data may contribute to this boundary.Source: §6 · p. 12
  • The study does not explore loco-manipulation-specific VLA architectures, and validation action MSE does not reliably predict on-robot performance.One-step action generation underperforms despite lower MSE, and these findings remain mechanistically underexplained.Source: §6 · p. 12

Core Contributors

The contributor descriptions identify primary leadership areas while noting that roles overlap and do not rank individual effort. The listed leads span controller and VLA design, software and evaluation, co-training, and hardware and task construction.

Core Contributors

  • The paper cautions that leadership areas overlap and are not disjoint partitions or a ranking of individual effort.Source: Core Contributors · p. 12
  • Haodong Zhu led teleoperation and evaluation software and codebase organization; Boyuan Zheng led co-training data collection and baseline implementation.Source: Core Contributors · p. 12
  • Yihang Hu led hardware design and assembly, task-scene construction, process design, and teleoperation data collection.Source: Core Contributors · p. 12
  • Tong Zhang led low-level controller design and testing, while Yingdong Hu led the manuscript outline and VLA design; both also contributed across controller, VLA, hardware, or teleoperation software.Source: Core Contributors · p. 12

A.1 The 12 Tasks

HLM-12 comprises 12 tasks with language prompts, detailed processes, evaluation rubrics, and stressed capabilities. The appendix also specifies real-world evaluation, robot hardware, and the VR teleoperation setup.

A.1 The 12 Tasks

  • The HLM-12 task suite covers varied whole-body behaviors, including object placement, shelf transfers, foot-operated pedals, and bimanual jar opening.The appendix organizes each task around a prompt, process, rubric, and stressed capability.Source: §A.1 · pp. 17, 18, 21

A.2 Overall Evaluation Protocol

  • Each policy-task pair receives five real-world rollouts using matched scene configurations, and task progress fractions capture partial completion rather than only success or failure.The protocol reports means with standard errors.Source: §A.2 · pp. 22–23

B.1 Humanoid Robot

B.2 Whole-Body Teleoperation Hardware

  • On-robot teleoperation uses a PICO4U HMD, two handheld controllers, and two leg trackers; the HMD streams head and wrist camera views, while controller triggers operate the grippers.Source: §B.2 · p. 23

B.3 Teleoperation-Free Data Collection Hardware

The appendix describes HuMI hardware for teleoperation-free demonstrations, four teleoperation action interfaces, and VLA design choices for humanoid control. These details specify what data and representations the policy pipeline uses.

B.3 Teleoperation-Free Data Collection Hardware

  • HuMI records wrist-view RGB, gripper widths, and synchronized tracker poses from handheld grippers and trackers attached to the pelvis and feet.An online IK preview helps collectors adjust demonstrations to keep robot trajectories feasible.Source: §B.3 · p. 24

C.1 Low-Level Controller & Teleoperation

  • The four teleoperation interfaces share visual observations and proprioceptive state, but expose different body-action spaces that shape demonstrations and the VLA’s action representation.Source: §C.1 · p. 24
  • Joint-based whole-body teleoperation records a 32-D action covering root motion and all 29 robot joints, unlike sparse interfaces that give indirect control of the lower body.The interface directly exposes arms, waist, legs, and root motion.Source: §C.1 · pp. 24–25

C.2 Whole-Body VLA Policy Design

  • The VLA design ablations cover action and proprioception interfaces, pretrained versus non-pretrained initializations, and one-step action-generation methods.Source: §C.2 · pp. 25–26
  • The default action ordering preserves π0.5’s pretrained bimanual slots and appends humanoid dimensions; proprioception supplies body pose that head and wrist cameras do not clearly observe.Source: §C.2 · p. 26

C.3 Heterogeneous Co-Training

The section describes cheaper stationary demonstrations that preserve manipulation and posture adjustments while omitting walking, and HuMI demonstrations converted into the same humanoid state-action format. OpenHLM training uses the stated hardware, duration, augmentation, and inference settings.

C.3 Heterogeneous Co-Training

  • Stationary teleoperation keeps the robot in place while collecting manipulation, posture adjustments, and in-place turns, avoiding walking portions of demonstrations.It uses the same robot and joint-based whole-body interface as full loco-manipulation demonstrations.Source: §C.3 · pp. 26–27
  • HuMI records stationary manipulation without the humanoid, then offline IK converts tracker trajectories into whole-body joint targets that can mix with teleoperation data.HuMI actions are shifted to approximate the 0.2 s future-frame preview latency; wrist images remain noticeably wider than robot-mounted views despite rectification.Source: §C.3 · p. 27

C.4 Hyperparameters

  • Training OpenHLM on 4 A800 GPUs takes approximately 24 hours, with random crop and rotation for head-camera images and brightness, contrast, and saturation changes for all images.At inference, the VLA generates 50-step chunks at 30 Hz and executes up to 25 actions before the next inference.Source: §C.4 · p. 27

D.1 Task Details & Evaluation Protocol

The long-horizon fruit-arrangement task combines two-handed picking and placement across three heights with walking and return to the start. Evaluation averages task progress across 10 of 20 fruit pairs; the GR00T and Ψ0 baselines are fine-tuned on 120 decoupled-control demonstrations.

D.1 Task Details & Evaluation Protocol

  • The fruit-arrangement task requires picking two designated fruits with opposite hands from 40 cm and 70 cm surfaces, placing them in shelf containers at 95 cm, and returning to the start.The robot walks to appropriate positions before manipulation, spanning a wide range of the G1's whole-body reachable workspace.Source: §D.1 · p. 28
  • The fruit set yields 20 ordered pairs and 120 demonstrations, while OpenHLM with HuMI uses 36 demos for six pairs and replaces 84 demos involving held-out lemon and tomato with HuMI data.Each pair receives six demonstrations.Source: §D.1 · p. 28
  • Evaluation averages task progress over 10 of 20 fruit pairs, with distractor-fruit layouts recorded to keep comparisons fair.Each fruit appears twice as fruit 1 and twice as fruit 2 among the selected pairs.Source: §D.1 · p. 28, Table 4 · p. 28

D.2 Baselines & Data Collection

  • GR00T N1.6 is fine-tuned for 50k steps on the 120 decoupled-control demonstrations, with gripper-width commands replacing its hand block and wrist cameras added to its visual input.The baseline retains relative arm commands and the decoupled body-command interface.Source: §D.2 · p. 29
  • Ψ0 is adapted to the same 120 demonstrations and decoupled-control stack, masking unused hand dimensions and fine-tuning its pretrained components for 50k steps.Its 36-D tensor layout is retained, but training uses only action dimensions present in the controller interface.Source: §D.2 · p. 29

E.1 Per-Task Results on the HLM-12 Benchmark

On HLM-12, co-training preserves training-task performance and brings both methods near oracle on held-out motion-reuse tasks, while stationary co-training—but not HuMI—matches the oracle on the new-motion pouring task. More HuMI demonstrations improve progress on the motion-reuse tasks, with diminishing gains after 10 demonstrations.

E.1 Per-Task Results on the HLM-12 Benchmark

  • On held-out motion-reuse Tasks 9–11, both co-training methods reach near-oracle performance, but on new-motion Task 12 (Pouring), stationary co-training matches the oracle while HuMI fails.Training-task performance remains similar across methods, with no regression from co-training.Source: Figure 12 · p. 30, §E.1 · p. 30

E.2 Scaling HuMI Demonstrations

  • 84% task progress is reached with 40 HuMI demos per task, versus 42% with 5 demos, on held-out motion-reuse Tasks 9–11.Progress is 67% at 10 demos and 76% at 20; the largest increase is 25 points from 5 to 10 demos, with smaller gains thereafter.Source: Figure 13 · p. 30, §E.2 · p. 30
  • The scaling experiments cover motion-reuse tasks where HuMI succeeds; extending HuMI data to support motion transfer such as Pouring remains future work.The authors identify scaling HuMI data for new motions as a direction they plan to explore.Source: §E.2 · p. 30

Related papers

Loading 2606.22174v1…