PaperlayerView exact revision on arXiv

Source-linked AI summary

SP-HLM: Sparse-Pose-Driven Motion Generation and Whole-Body Control for Humanoid Loco-Manipulation

Zhaohui Yang, Quanbao Lin, Chengqi Shi, Zhenglang Weng, Jicheng Lin, Boyu Chen, Tengjie Zhu, Guanzhu Ren, Jiahao Xu, Yang Feng, Xiaokang Yang, Yao Mu

arXiv:2610.30044v1cs.ROcs.AI

TL;DR

Humanoid loco-manipulation interfaces can burden high-level policies or leave whole-body coordination underspecified. SP-HLM combines five explicit task-critical poses with a learned motion prior, using SP-VLA, SP-MGM, and SP-GMT to predict targets, reconstruct motion, and control the body. The framework reports improved tracking, coordination, and loco-manipulation outcomes over existing approaches in simulation and on a real robot.

  • Problem

    Existing interfaces use a single action representation for task intent and whole-body coordination, which may burden the high-level policy or leave one aspect underspecified.

    Source: §I · pp. 1–2

  • Method

    SP-HLM combines five explicit task-critical body poses with a learned latent motion prior, with SP-VLA predicting targets, SP-MGM reconstructing full-body motion, and SP-GMT integrating both for control.

    Source: §I · p. 2

  • Results

    SP-HLM achieves 52.75% overall success across eight simulated tasks, outperforming token-space and joint-space baselines, and is validated on four real-world tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Experiments across motion tracking, motion generation, and loco-manipulation in simulation and the real world demonstrate the effectiveness of the interface and complete framework.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Humanoid loco-manipulation requires an effective interface between high-level task policies and whole-body control. Existing interfaces typically rely on a single action representation for both task intent and whole-body coordination. This may burden the high-level policy or leave one aspect underspecified. We present SP-HLM, a framework built on a sparse-pose-driven interface. The interface combines five task-critical body poses with a learned latent motion prior to represent spatial intent and whole-body coordination in complementary forms. SP-VLA, a vision-language-action model, first predicts task-relevant sparse poses and finger joint positions from multimodal observations and language instructions using asynchronous action-chunk inference. Given these sparse targets, SP-MGM, a motion generation model trained through two-stage distillation, reconstructs coordinated full-body motion in real time. A pretrained motion encoder maps this motion to a compact latent representation. SP-GMT, a general motion tracker, then integrates the explicit sparse targets with this implicit motion prior for whole-body control. Experiments in simulation and on a real humanoid robot demonstrate improved motion tracking, whole-body coordination, and loco-manipulation success over existing approaches. Additional results and videos are available on our \href{https://sp-hlm.github.io/}{\textcolor{blue}{project page}}.

I. INTRODUCTION

SP-HLM pairs explicit sparse task-space targets with a learned whole-body motion prior, and its complete system outperforms evaluated baselines across simulated tasks while being validated on a real robot. The introduction positions this interface against motion-generation and control methods that leave task-relevant whole-body intent comparatively less explored.

I. INTRODUCTION

  • Existing interfaces trade off high-level policy burden against underspecified coordination: joint-space actions require coordination modeling, while sparse task-space targets do not fully specify whole-body motion.Token-space actions encode reusable coordination patterns, but task-critical spatial targets may remain implicit.Source: §I · pp. 1–2
  • The proposed interface makes five task-critical poses explicit and represents whole-body coordination with a learned latent motion prior.This preserves direct task-space control without requiring the high-level policy to specify full-body coordination.Source: §I · p. 2
  • A pretrained motion VAE encodes full-body motion into a compact latent that supplies coordination information absent from sparse targets and supports efficient policy training.SP-GMT conditions jointly on the sparse task-space targets and this latent representation.Source: §I · p. 2
  • SP-HLM achieves 52.75% overall success across eight simulated tasks, outperforming token-space and joint-space baselines, and is validated on four real-world tasks.SP-MGM also improves sparse-pose adherence over its motion-generation baseline while achieving a 1.45× higher generation rate.Source: §I · p. 2
  • SP-VLA predicts sparse poses and finger targets from multimodal inputs, SP-MGM reconstructs coordinated motion, and SP-GMT combines the targets and motion prior for control.SP-VLA uses asynchronous action-chunk inference; SP-MGM uses two-stage distillation.Source: §I · p. 2

A. General Humanoid Motion Generation and Control

  • Prior motion-generation and tracking work increasingly supports hierarchical control, but largely uses dense references; the interface conveying task-relevant whole-body intent remains comparatively less explored.Related work also describes scalable motion tracking and reusable behavioral representations.Source: A. General Humanoid Motion Generation… · p. 2

B. Motion Interfaces for Humanoid Loco-Manipulation

SP-HLM combines explicit task-space targets with an implicit whole-body motion prior, connecting high-level loco-manipulation policies to whole-body control. Its pipeline predicts five-point poses and hand targets, reconstructs full-body motion, and integrates both command forms in a tracking policy.

B. Motion Interfaces for Humanoid Loco-Manipulation

  • Existing interfaces either require high-level policies to reason over detailed body configurations or use compact learned representations that make task-relevant spatial constraints less explicit.SP-HLM combines explicit task-space targets with a learned whole-body motion prior.Source: B. Motion Interfaces for Humanoid… · pp. 2–3

A. SP-HLM Overview

  • The interface constrains the pelvis, both wrists, and both ankles in world-frame position and orientation, while a motion prior supplies coordinated full-body motion consistent with the task geometry.An explicit–implicit tracking policy combines the sparse-pose constraints with the latent prior.Source: A. SP-HLM Overview · p. 3
  • SP-VLA predicts five-point pose sequences and finger-joint targets; SP-MGM generates a conditioned full-body trajectory, whose encoded latent and pelvis-frame sparse targets feed SP-GMT joint-position commands.Finger targets go directly to the hand PD controllers.Source: A. SP-HLM Overview · p. 3

B. SP-GMT: Explicit–Implicit General Motion Tracking

  • SP-GMT combines explicit sparse-pose constraints with a latent encoding whole-body coordination to support precise, dynamically feasible whole-body control.The sparse poses specify task-critical spatial constraints; the motion latent encodes coordination.Source: B. SP-GMT: Explicit–Implicit General… · p. 3

1) Yaw-Invariant Latent Motion Prior:

A yaw-aligned motion VAE provides a compact latent prior, and SP-GMT integrates that latent with explicit sparse-pose targets. Training and deployment obtain the motion and pose commands from different sources.

1) Yaw-Invariant Latent Motion Prior:

  • A motion VAE pretrained on approximately 190 hours of motion data learns a compact representation of coordinated whole-body motion.Its input representation removes global horizontal translation and heading while retaining articulated pose, pelvis tilt, and local root dynamics.Source: 1) Yaw-Invariant Latent Motion Prior · p. 3
  • Yaw-aligned preprocessing makes the motion representation invariant to global placement while preserving articulated pose, pelvis tilt, and local root dynamics.After pretraining, the frozen encoder’s posterior mean supplies the deterministic latent for the tracking policy and helps improve training efficiency.Source: 1) Yaw-Invariant Latent Motion Prior · p. 3

2) Explicit–Implicit Integration:

  • SP-GMT’s implicit branch encodes full-body motion, while its explicit branch transforms world-frame five-point poses into the current pelvis frame.The policy also receives proprioceptive state and the previous action, then predicts body joint-position targets.Source: 2) Explicit–Implicit Integration · p. 3
  • SP-GMT training uses five-point poses and full-body motion from reference data, whereas deployment obtains those commands online from SP-VLA and SP-MGM, respectively.The sparse targets and latent motion command are integrated with the robot’s current state for control.Source: 2) Explicit–Implicit Integration · p. 3

3) Tracking Policy Optimization:

SP-GMT is trained as a goal-conditioned reinforcement-learning tracker, with reward terms for full-body and five-point pose tracking. Training also emphasizes challenging motion segments and randomizes deployment conditions.

3) Tracking Policy Optimization:

  • SP-GMT uses goal-conditioned reinforcement learning with PPO and an asymmetric actor–critic whose critic receives privileged full-body state during training.Source: 3) Tracking Policy Optimization · p. 3
  • The tracking reward combines full-body pose and velocity tracking with a regularizer, while a sparse-pose term rewards global position and orientation tracking of five-point poses.Source: 3) Tracking Policy Optimization · pp. 3–4
  • Failure-aware and uniform motion sampling emphasize challenging segments while retaining coverage, and randomization spans dynamics, contacts, perturbations, delays, and command filtering.Source: 3) Tracking Policy Optimization · p. 4

C. SP-MGM: Real-Time Motion Generation from Sparse Pose Constraints

SP-MGM reconstructs full-body motion from five-point targets using a compressed pretrained generator and two-stage distillation. Stage I trains the student on teacher predictions and paired ground-truth motions.

C. SP-MGM: Real-Time Motion Generation from Sparse Pose Constraints

  • SP-MGM compresses a pretrained diffusion motion generator by reducing network depth and denoising steps, then distills it to improve sparse-pose adherence while maintaining smooth, coordinated motion.Source: C. SP-MGM: Real-Time Motion Generation… · p. 4

1) Offline Supervised Fine-Tuning (Stage I):

  • Stage I jointly trains the compressed student on frozen-teacher predictions and paired ground-truth motions, using a subsampled student denoising schedule matched to teacher timesteps.Source: 1) Offline Supervised Fine-Tuning… · p. 4
  • The Stage I objective balances teacher supervision with reconstruction fidelity; the motion loss covers root and joint kinematics, foot contacts, and forward-kinematics consistency.Source: 1) Offline Supervised Fine-Tuning… · p. 4

2) Rollout-Aware Distillation (Stage II):

Stage II adds teacher correction on student-visited states under five-point conditioning and regularizes temporal variation. The SP-VLA front end predicts sparse poses and hand commands asynchronously, while SP-HLM rollouts illustrate task progression across simulated tasks.

2) Rollout-Aware Distillation (Stage II):

  • Stage II addresses inference-time error accumulation and distribution shift by applying frozen-teacher supervision to student-visited states under five-point conditioning.Source: 2) Rollout-Aware Distillation (Stage II) · p. 4
  • The rollout-aware objective regularizes joint motion, pelvis translation, and orientation with finite differences, yielding temporally smooth motion from a compressed network with fewer denoising steps.Source: 2) Rollout-Aware Distillation (Stage II) · p. 4

D. SP-VLA: Asynchronous Sparse-Pose and Hand-Command Prediction

IV. EXPERIMENTS

The experiments are organized around three questions: motion-tracking performance, the speed and constraint adherence of two-stage distillation, and task success and real-world transfer of the sparse-pose interface.

IV. EXPERIMENTS

  • RQ1 asks whether explicit–implicit tracking improves motion tracking and how it compares with state-of-the-art general motion trackers.Source: §IV · p. 5
  • RQ2 asks whether two-stage distillation accelerates motion generation while preserving accurate adherence to sparse end-effector constraints.Source: §IV · p. 5
  • RQ3 asks whether the sparse-pose interface improves success over joint-based and token-based interfaces across diverse tasks and how the complete system transfers to real-world deployment.Source: §IV · pp. 5–6

A. Motion Tracking Performance

SP-GMT delivers strong global tracking across two unseen test datasets despite substantially less motion data and training compute than major baselines. Ablations show that sparse spatial targets and a learned latent reference contribute complementary tracking information.

A. Motion Tracking Performance

  • The benchmarks contain 500 motions each and are evaluated over ten seeded runs with perturbations to reference trajectories.BONES-SEED Diverse-500 is included in SONIC’s training data.Source: A. Motion Tracking Performance · p. 6

1) Comparison with Motion-Tracking Baselines:

2) Ablation Study of SP-GMT:

  • Removing either the latent reference or five-point reference worsens tracking in different ways, while replacing the latent with joint-space reference increases global tracking error.The combined references yield the lowest global MPJPE and EE error on both benchmarks while remaining competitive in local tracking.Source: 2) Ablation Study of SP-GMT · p. 6
  • Removing the robot pelvis pose causes only a moderate performance drop, suggesting limited dependence on precise pelvis localization.Source: 2) Ablation Study of SP-GMT · p. 6

B. Motion Generation Quality and Efficiency

Across eight simulated loco-manipulation tasks, SP-HLM achieves the highest overall success rate among evaluated methods, while SP-MGM improves sparse-pose generation accuracy and speed. Its ablations indicate that the second distillation stage and temporal regularization improve adherence and smoothness.

1) Comparison with Motion-Generator Baseline:

2) Ablation study of SP-MGM:

  • Full SP-MGM performs best across metrics; Stage I alone raises P95 EE error from 3.212 to 12.772 cm, while removing Ltemp raises EE and body-jerk P95 by 27.8% and 37.0%.Stage II improves sparse-pose adherence and temporal smoothness; temporal regularization suppresses high-frequency jitter while slightly reducing EE errors.Source: 2) Ablation study of SP-MGM · p. 6

1) Experimental Setup:

  • Evaluation spans eight simulated tasks, including six in-house tasks with 120 SONIC demonstrations each and two HumanoidArena tasks; four everyday real-world tasks use 80 demonstrations each and a ZED 2i camera.The simulated tasks cover workspace extension, body-part interaction, and locomotion-conditioned manipulation.Source: 1) Experimental Setup · p. 6

2) Simulated Loco-Manipulation:

  • SP-HLM achieves the highest overall success rate among evaluated methods across eight simulated tasks; directly predicting the latent drops success to 23.25%.The comparison includes token-space and joint-space interfaces; the direct-latent variant predicts the latent alongside sparse poses and finger targets.Source: 2) Simulated Loco-Manipulation · p. 6, 1) Experimental Setup · p. 6

3) Real-World Loco-Manipulation:

Real-world evaluation reports success across four loco-manipulation tasks, while the conclusion identifies hand-level dexterity as outside the study’s scope.

3) Real-World Loco-Manipulation:

  • Success exceeded 50% for object placement and shelf transfer, and reached 25% for cabinet opening and 40% for trash-bin opening.The evaluation used 20 trials per real-world task.Source: 3) Real-World Loco-Manipulation · pp. 6–7
  • Fig. 4 presents representative execution keyframes from four real-world humanoid loco-manipulation tasks.The figure shows representative rollouts rather than additional reported success-rate values.Source: Figure 4 · p. 7

V. CONCLUSIONS

  • SP-HLM integrates explicit sparse-pose constraints with an implicit full-body motion prior for whole-body control.The conclusion describes SP-VLA, SP-MGM, and SP-GMT as the components realizing this interface.Source: §V · p. 7
  • Experiments in simulation and the real world cover motion tracking, motion generation, and loco-manipulation.The paper concludes that these experiments demonstrate the effectiveness of the interface and framework.Source: §V · p. 8
  • The study focuses on whole-body coordination; hand control uses finger joint-position targets, and dexterous manipulation is not explored.The authors identify whole-body dexterous manipulation as future work.Source: §V · p. 8

VI. APPENDIX

The appendix gives implementation details for SP-GMT rewards, two-stage SP-MGM distillation, and SP-VLA action-chunk training and inference.

A. SP-GMT Training Details

  • SP-GMT adopts BeyondMimic’s reward formulation, adding global position and orientation tracking rewards for wrists and ankles and penalties for undesired foot motion.Contact and self-collision thresholds are 1 N; the air-time term is non-positive and disabled for randomly sampled static segments.Source: A. SP-GMT Training Details · p. 9, Table V · p. 9

B. SP-MGM Training Details

  • SP-MGM distills a pretrained KIMODO generator in two stages, using paired-motion supervision followed by rollout-aware distillation on student-visited states.The second stage relabels intermediate rollout states with the frozen teacher and applies temporal regularization for motion smoothness.Source: B. SP-MGM Training Details · p. 9, Table VI · p. 9

C. SP-VLA Training and Inference Details

  • SP-VLA is fine-tuned for 160,000 steps and predicts 40-step action chunks using a 59-D representation.Inference executes the first 34 actions, retains the final 6 as the next clean prefix, and uses four flow-matching inference steps.Source: C. SP-VLA Training and Inference Details · p. 9
Loading 2610.30044v1…