PaperlayerView exact revision on arXiv

Source-linked AI summary

Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating

Haozhuo Zhang, Qiang Zhang, Jian Tang, Mingzhe Ni, Michele Caprio, Angelo Cangelosi, Wei Pan

arXiv:2610.08320v1cs.ROcs.GR

TL;DR

Long-horizon whole-body loco-manipulation requires humanoids to transport multiple objects continuously, but existing approaches face training challenges as task chains grow. Humanoid Horizon combines parallel stage training, dynamic handoff starts, and reward gating in a shared policy. It exceeds 80% per-stage success on two-object transport, with graceful degradation as horizons lengthen relative to prior baselines.

  • Problem

    Long-horizon whole-body loco-manipulation requires continuous multi-object transport without resets, while prior approaches often focus on short-horizon single-object skills.

    Source: §1 · p. 2

  • Method

    Humanoid Horizon uses one policy with parallel stage optimization, rollout-derived downstream starts, and reward gating to protect previously placed objects.

    Source: §1 · p. 2

  • Results

    Per-stage success exceeds 80% on two-object transport, and performance degrades gracefully as horizons grow relative to prior baselines.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The framework supports continuous multi-object rearrangement with one policy across diverse scenes and maintains behavior as task horizons lengthen.

    Source: §1 · p. 3

    The Succ1-to-Succ2 drop is larger than later stage drops because later cycles start from varied post-task configurations in clutter, complicating reorientation and navigation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Cluttered indoor environments, where large and heavy objects are scattered across diverse surfaces, require humanoid robots to sequentially navigate, grasp, transport, and accurately place each item at its target location within a single uninterrupted episode. This long-horizon, whole-body loco-manipulation task remains a significant challenge for current methods. Previous approaches often suffer from two main issues: easy-reward bias, where training overemphasizes early transport stages at the expense of later ones, and catastrophic forgetting, where focusing on later stages leads to a decline in earlier-stage performance. In this work, we introduce Humanoid Horizon, a unified policy framework designed to overcome these limitations through three interrelated mechanisms. The Parallel Training Strategy organizes $N$ scenes into $S$ concurrent stage streams governed by a shared policy, ensuring all transport stages receive continuous gradient updates and removing the bottleneck of sequential optimization. The Dynamic Starting Mechanism updates each environment's initial state with terminal states from upstream rollouts, gradually broadening transition coverage and enhancing robustness at stage boundaries. Reward Gating sets the reward to zero for the rest of the episode in later-stage streams when the immediately preceding object is displaced beyond a set threshold, so the shared policy learns not to disturb a just-placed object and earlier placements are preserved throughout the episode. Collectively, these strategies achieve per-stage success rates exceeding 80\% on the two-object LHM-Humanoid benchmark (350 training scenes, 66 held-out scenes). As the number of sequentially transported objects grows beyond two, success declines with the horizon, but the degradation is graceful relative to the sharp drop seen in all baselines.

1 Introduction

Humanoid Horizon targets continuous multi-object humanoid loco-manipulation without resets, addressing training failures in long task chains. Its coupled training mechanisms achieve strong two-object results, while performance degrades gracefully as horizons grow.

1 Introduction

  • Long-horizon whole-body loco-manipulation remains a gap: humanoids must continuously transport multiple large objects without environment resets.Source: §1 · p. 2
  • Longer task chains expose easy-reward bias toward early stages and catastrophic forgetting of first-object transport after training shifts to later stages.The paper describes LHM-Humanoid as a concrete instance of this forgetting behavior.Source: §1 · p. 2
  • Humanoid Horizon uses one policy for continuous multi-object loco-manipulation without resets, auxiliary models, or handcrafted task-specific modules.Parallel Training jointly optimizes stages; Dynamic Starting expands transition coverage using upstream terminal states; Reward Gating zeros later-stage rewards when the preceding object is displaced beyond tolerance.Source: §1 · p. 2
  • Per-stage success exceeds 80% in two-object sequential transport, and success degrades gracefully with horizon length relative to the sharp drop in prior baselines.Evaluation uses 350 training/evaluation scenes and 66 unseen generalization scenes.Source: §1 · p. 3
  • The framework’s contributions include jointly optimizing stages, broadening handoff coverage, and preserving earlier placements during later transport.Source: §1 · p. 3

2 RELATED WORK

Prior work spans physically based motion synthesis, human-scene interaction, and long-horizon manipulation, but humanoid multi-object loco-manipulation remains limited. Humanoid Horizon addresses scalability and stage transitions with a unified policy and Dynamic Starting.

2.1 Physically-Based Motion Synthesis and Human-Scene Interaction

  • Most motion-interaction systems remain short-horizon and single-object; HumanVLA handles heavier objects but retains fixed initial states, while LHM-Humanoid’s sequential training and task-specific designs limit scalability.Source: §2.1 · p. 3

2.2 Long-Horizon Loco-Manipulation

  • Autonomous long-horizon whole-body humanoid loco-manipulation remains nascent, with reliable humanoid state estimation and transfer still challenging.Representative attempts include LessMimic and LHM-Humanoid.Source: §2.2 · p. 3
  • Dynamic Starting differs from reverse curricula, reset-free RL, and demonstration curricula by starting each stage from the preceding stage’s produced terminal state without extra policies, demonstrations, or reward.Source: §2.2 · p. 4

3 Method

Humanoid Horizon trains one policy for uninterrupted multi-object transport by jointly optimizing stages, adapting downstream starts to upstream handoffs, and guarding completed placements. It also distills the teacher into a perception-conditioned VLA student.

3.1 Task Formulation

  • The task requires one policy to navigate, grasp, lift, transport, and place ordered objects, then continue to the next without resetting across diverse scenes.Scenes vary in layout, object configuration, and target surfaces.Source: §3.1 · p. 4

3.2 Parallel Training Strategy

  • Parallel Training jointly optimizes all transport stages in one shared policy, keeping persistent gradients across stages instead of serial optimization.Stage streams use separate state buffers; later-stage streams also train transition behaviors such as release, reorientation, and obstacle-aware maneuvering.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Dynamic Starting Mechanism

  • Dynamic Starting replaces fixed proxy starts with upstream rollout terminal states, broadening handoff coverage through repeated epoch-wise buffer updates.This aligns downstream initial states with natural transitions without extra supervision or handcrafted sampling.Source: §3.3 · p. 5

3.4 Reward Gating for Physical Consistency

  • Success-All requires every object to remain at its goal throughout the episode, so per-stage success alone does not capture disturbances to earlier placements.Source: §3.4 · p. 6
  • Reward Gating zeros the remaining reward in a later-stage stream when the immediately preceding object moves beyond tolerance, encouraging preservation of prior placements.The shared policy is tested through stages in order, with each stage guarding its immediate predecessor.Source: §3.4 · p. 6

3.5 Extension: Vision-Language-Action Policy Distillation

  • DAgger distills the RL teacher into RGB- or depth-conditioned VLA students that receive per-stage language instructions and retain long-horizon multi-object behaviors.The students execute online without privileged state information.Source: §3.5 · p. 6

4 Experiments

Experiments evaluate Humanoid Horizon on two-object transport, held-out scenes, VLA distillation, and five-object episodes. The full method outperforms the strongest prior baseline on the reported two-object metrics, while success declines more gradually as the horizon grows.

4.1 Dataset and Evaluation Metrics

  • The benchmark uses 350 cluttered scenes for two consecutive fetch-carry-place cycles and 66 unseen tasks for generalization.The dataset spans four room categories and 79 object categories, including 25 movable targets; a cycle succeeds when the object root is within 0.2 m of its goal.Source: §4.1 · pp. 6–7

4.2 Baseline clarifications

  • Baselines follow their original pipelines; all except the VLA comparison share observation and action interfaces.The reported comparisons therefore primarily reflect learning-strategy differences rather than mismatched interfaces.Source: §4.2 · p. 7

4.3 Main results on 350 tasks

  • 96.6%/81.3%/80.4% on Succ1/Succ2/SuccAll, versus LHM-Humanoid’s 87.4%/72.6%/71.1% on 350 tasks.The full method combines Parallel Training, Dynamic Starting, and Reward Gating.Source: Table 1 · p. 7, §4.3 · pp. 7–8
  • Ablations show removing Dynamic Starting preserves Succ1 at 92.7% but reduces SuccAll to 67.7% because fixed proxy starts mismatch evolving handoff states; removing Reward Gating lowers SuccAll to 69.6%.Source: §4.3 · p. 7

4.4 Results on 66 unseen tasks

  • 97.1%/79.3%/78.0% on Succ1/Succ2/SuccAll across 66 unseen scenes; LHM-Humanoid reaches 61.6% SuccAll.Removing Dynamic Starting reduces SuccAll to 63.5%, while removing Reward Gating yields 89.1%/72.3%/66.4% on Succ1/Succ2/SuccAll.Source: Table 2 · p. 8, §4.4 · p. 8

4.5 VLA extension results

  • Distilled RGB and depth Humanoid Horizon variants outperform all distilled baselines on most metrics in the VLA extension.All methods use the same DAgger pipeline; distillation replaces privileged state inputs with partial egocentric vision, and weaker teachers suffer larger drops.Source: Table 3 · p. 8, §4.5 · p. 8

4.6 Extension to more than two objects sequential transport

  • For five-object transport, Humanoid Horizon’s success falls from 82.2% at stage 3 to 66.5% at stage 5, with 60.2% SuccAll.The same policy remains ahead at every stage; SuccAll falls to 49.8% without Dynamic Starting and 38.9% without Reward Gating.Source: Table 4 · p. 9, §4.6 · p. 9

5 Failure Analysis

The larger first-to-second-stage success gap reflects harder starting conditions after the first cycle. Frozen-action failures occur when grasp completion fails, with equal scene budgets a likely contributor to under-optimized difficult cases.

5 Failure Analysis

  • The Succ1-to-Succ2 gap exceeds later per-stage drops because later cycles start from arbitrary post-task poses amid clutter and nearby obstacles.This makes reorientation and obstacle-aware navigation harder than starting from a fixed canonical pose on open floor.Source: §5 · p. 9
  • Frozen-action failures occur when the humanoid reaches the target but does not complete a stable grasp.Uniform per-scene training budgets may leave harder clutter and object configurations under-explored.Source: §5 · p. 9

6 Conclusion and Future Work

Humanoid Horizon combines parallel training, dynamic starting, and reward gating for long-horizon whole-body loco-manipulation. The same policy extends to five-object transport, with success degrading more gracefully than competing methods as the horizon grows.

6 Conclusion and Future Work

  • Parallel Training maintains optimization pressure across stages, Dynamic Starting uses realistic handoff-state distributions, and Reward Gating preserves completed placements during later transport.Together, the mechanisms mitigate easy-reward bias and catastrophic forgetting.Source: §6 · p. 9
  • The same policy extends to five-object sequential transport without an architecture change, although success declines as the horizon lengthens.The degradation is graceful relative to the sharp drop of competing methods.Source: §6 · pp. 9–10
  • Future work targets the Succ1-to-Succ2 navigation gap, frozen-action grasping failures, horizons beyond five objects, and physical humanoid deployment.The authors plan to investigate improved path planning and difficulty-aware training.Source: §6 · p. 10

A.1 Humanoid Embodiment

The humanoid embodiment has 15 rigid bodies and 28 controlled DoFs, with policy actions mapped to PD position targets. The appendix also specifies the teacher, AMP discriminator, and VLA student observation spaces.

A.1 Humanoid Embodiment

  • The humanoid has 15 rigid bodies and 28 DoFs, with normalized policy actions mapped to PD position targets at 30 Hz control frequency.Joint-limit-derived per-DoF scale and offset constants define the affine mapping.Source: §A.1 · p. 13, §A.2 · p. 13

A.2 Observation and Action Spaces

  • The teacher receives a 250-dimensional privileged state vector plus a 200-point object representation, while the AMP discriminator uses ten 131-dimensional motion frames.The VLA student instead receives body proprioception, an RGB or depth image, a language embedding, and the previous action.Source: §A.2 · p. 13

A.3 Network Architectures

The appendix describes shared-backbone teacher and multimodal VLA student architectures, points to teacher training hyperparameters, and specifies the components of the total training loss.

A.3 Network Architectures

  • The teacher shares a backbone across actor, critic, and discriminator heads; the VLA student combines image, text, body-proprioception, and previous-action inputs to output deterministic actions.The teacher uses MLP heads and a learnable-variance Gaussian action distribution; the student uses an EfficientNet-B0 image backbone and residual actor blocks.Source: §A.3 · pp. 13–14

A.4 Reinforcement Learning Hyperparameters

  • Table 5 lists the teacher reinforcement-learning training hyperparameters.The supplied text identifies the table’s scope but does not state its parameter values.Source: §A.4 · p. 14, Table 5 · p. 14

A.5 Loss Functions and Weights

  • The total training loss combines actor, critic, action-bound, and AMP discriminator losses.The actor uses PPO’s clipped surrogate objective, the critic uses normalized-value MSE, and the discriminator includes prediction loss, regularization, gradient penalty, and weight decay; Table 6 lists loss weights.Source: §A.5 · pp. 14–15, Table 6 · p. 15

A.6 Reward Design

The appendix specifies the shaped transport reward, DAgger training loop, simulation settings, and episode termination conditions.

A.6 Reward Design

  • The transport reward has eight bounded terms, with approach, grasp, and height rewards clamped near the goal to avoid interfering with placement.The approach and grasp thresholds are δr = 0.5 m and δo = 0.3 m; near the goal, dobj→goal < δo triggers the clamp.Source: Table 7 · p. 15, §A.6 · p. 15
  • The reward design uses a 1.5 m/s target speed and sets floor-to-floor transport height reward to 1.0.The carry height is max(zinit, zgoal) + 0.1 m.Source: §A.6 · p. 15

A.7 DAgger Distillation Hyperparameters

  • DAgger alternates student and teacher actions, stores teacher-labeled observations, and trains the student on replay-buffer batches with exponentially decaying β.The student minimizes MSE on sampled batches; β shifts rollouts from teacher-driven toward student-driven.Source: §A.7 · p. 16

A.8 Simulation and Physics

  • Experiments use IsaacGym with GPU-accelerated PhysX on two A100 80 GB GPUs, with TGS and four position iterations.The setup also specifies contact, friction, environment-spacing, and axis settings.Source: §A.8 · p. 16

A.9 Termination Conditions

  • Episodes end on a qualifying body-part contact-and-height fall condition or at 300 single-object or 600 two-object steps.Fall detection excludes hands, lower arms, and feet, and requires contact force > 0.1 N and height < 0.15 m simultaneously.Source: §A.9 · p. 16

B Training Curves across Seeds

Three random-seed runs converge to similar training success, while logged training and sequential evaluation metrics differ because they measure success differently.

B Training Curves across Seeds

  • SuccAll converges to approximately 87–89% across three seeds, with standard deviation ≤0.3%.The authors report stable convergence across independent initializations.Source: Figure 3 · p. 17, B Training Curves across Seeds · p. 17
  • The initial rapid rise during epochs 0–2,000 reflects adaptation from the pretrained single-object checkpoint, followed by steady improvement.The authors attribute subsequent improvement to the interplay of dynamic starting and reward gating.Source: Figure 3 · p. 17
  • Training SuccAll is 87.6% versus evaluation SuccAll 80.4%, because training averages independent stage-stream success while evaluation compounds sequential-stage errors.Training Succ1 is 92.5% versus evaluation Succ1 96.6%; training uses a 0.1 m placement threshold, compared with 0.2 m at evaluation.Source: B Training Curves across Seeds · p. 17

C Evaluation Seed Variance

This section reports evaluation seed-variance materials, broader-impact considerations, and checklist statements about experimental support and release.

C Evaluation Seed Variance

  • Table 10 reports three-seed standard deviations of success rates for Humanoid Horizon and LHM-Humanoid in percentage points.The VLA block is the distilled student, with RGB and depth listed separately in Table 3.Source: Table 10 · p. 18, C Evaluation Seed Variance · p. 18

D Broader Impact

  • The authors identify assistive robotics and warehouse automation as potential benefits, while noting dual-use risks and the need for additional safety mechanisms before real-world deployment.They specify force limits and human-in-the-loop supervision as examples; the work uses no personal data, human subjects, or crowdsourcing.Source: D Broader Impact · p. 18

NeurIPS Paper Checklist

  • The checklist states that experiments and ablations in Section 4 support the abstract and introduction claims, and that dataset and code will be released upon acceptance.It also points to the appendix for implementation details and reports results as mean ± standard deviation across three seeds.Source: NeurIPS Paper Checklist · pp. 19–21
  • The checklist records that limitations are discussed in Section 5 and Section 6.It identifies those sections as Failure Analysis and Future Work, respectively.Source: NeurIPS Paper Checklist · p. 19
Loading 2610.08320v1…