PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Multi-Phase Robot Loco-Manipulation via Phase-Conditioned Policy Optimization

Simone Manni, Annika Wollschläger, Dario Bellicoso, Abel Gawel, Marco Hutter, Simon Le Cleac'h, Jan Brüdigam

arXiv:2610.36270v1cs.RO

TL;DR

Long-horizon loco-manipulation remains challenging because it combines exploration with distinct physical phases, while demonstration-based methods bring data and platform constraints. The paper extends PPO with phase-guided learning and hierarchical control, demonstrating multi-stage skills on hardware across two platforms.

  • Problem

    Long-horizon loco-manipulation requires coordinating distinct behaviors, and existing approaches face exploration challenges or rely on demonstrations that add processing overhead and platform specificity.

    Source: §I · pp. 1–2

  • Method

    The method extends PPO with phase-conditioned value estimation, phase-specific task rewards, and multi-phase training, using a shared hierarchical control interface across platforms.

    Source: §I · p. 2

  • Results

    The framework demonstrates multi-stage loco-manipulation skills on hardware, including Spot tire rolling and Unitree G1 box lifting and carrying.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    The work shows that task phases can be incorporated directly into policy learning for long-horizon loco-manipulation without human guidance.

    Source: §I · p. 2

    Scaling remains constrained by the need to manually segment tasks into phases and tune rewards, with box-shape changes requiring reset-keyframe updates.

    Source: §V · p. 7

Abstract

from arXiv · show

Complex whole-body loco-manipulation requires executing diverse maneuvers, such as crouching, grasping, and locomotion, within a single long-horizon task. While performing simple pick-and-place behaviors is well-established, scaling to complex, long-horizon tasks remains difficult. Currently, this challenge is addressed by relying on human motion data and correspondingly humanoid platforms, which limits platform independence and introduces substantial data processing overhead. Instead, we extend proximal policy optimization (PPO) to train a single policy for multi-phase tasks across diverse robotic hardware, leveraging prior task structure to shape behavior. Our method adapts PPO using a phase-conditioned critic, phase-gated reward functions, and a multi-phase reset curriculum to guide exploration. To support varied hardware, we employ a decoupled hierarchical control architecture that separates lower-level balancing dynamics from high-level phase execution. We demonstrate multi-stage loco-manipulation skills on hardware on both the Unitree G1 and a Boston Dynamics Spot equipped with a manipulator arm, and validate our design choices through ablations. Our approach enables the acquisition of complex loco-manipulation behaviors purely through environment interaction while directly incorporating high-level task structure into policy learning.

I. INTRODUCTION

Long-horizon loco-manipulation demands coordinated behavioral phases, but existing approaches either struggle with exploration and reward tuning or rely on demonstrations that limit platform independence. The paper uses task phases to guide a single PPO policy and evaluates it across robotic platforms and tasks.

I. INTRODUCTION

  • Lifting and carrying require coordinating walking, crouching, and rising, while standard reward tuning can lead to unwanted local minima such as kicking or dragging the box.Source: §I · p. 2
  • Demonstration-driven methods incur data-processing overhead and platform-specificity, while newer autonomous methods focus on table-to-table transport rather than ground-level transitions.Source: §I · p. 2
  • Task stages offer a route to guide exploration without demonstrations, and a single policy can manage inter-phase transitions without external orchestration.Source: §I · p. 2
  • The proposed PPO variant segments tasks with a state machine, trains the actor to predict phase, and conditions the critic on the active phase.Source: §I · p. 2
  • A common control interface supports experiments across Spot and Unitree G1, with targeted ablations testing the framework’s design choices.Source: §I · p. 2

II. RELATED WORK

The paper builds on hierarchical loco-manipulation control and demonstration-free whole-body learning, while focusing its contribution on a unified high-level policy for long-horizon behavior. It contrasts this approach with skill chaining and motion-tracking methods.

A. ReLiC for Hierarchical Loco-Manipulation

  • ReLiC separates low-level locomotion and posture tracking from high-level task behavior; the paper adopts this interface, adapting a low-level controller for G1 and using ReLiC’s controller on Spot.Source: A. ReLiC for Hierarchical… · p. 2

B. Humanoid Whole-Body Control

  • The humanoid policy follows a demonstration-free single-policy approach but narrows its objectives, using feasible command regions and keyframe-based exploration rather than ULC’s broader tracking objectives and curriculum.Source: B. Humanoid Whole-Body Control · pp. 2–3

C. Reinforcement Learning for Long-Horizon and Multi-Stage Tasks

Prior long-horizon methods often chain sub-policies, a strategy that can make transitions brittle in dynamic loco-manipulation. The paper instead uses phase structure to train one unified policy with phase-specific value estimation.

C. Reinforcement Learning for Long-Horizon and Multi-Stage Tasks

  • Dynamic contacts and momentum make stable boundaries between chained sub-policies difficult, whereas the paper directly optimizes one policy over multiple phases.Source: C. Reinforcement Learning for… · p. 3
  • Like reward-machine approaches, the method uses phase-dependent rewards, but trains a unified policy with a phase-conditioned value network instead of separate policies.Source: C. Reinforcement Learning for… · p. 3

III. METHOD

  • The method section presents the phase-conditioned policy optimization approach after formulating the RL problem and describing the low-level controllers.Source: §III · p. 3

A. Reinforcement Learning Problem

The task is formulated as a POMDP and controlled through a hierarchy that separates whole-body command tracking from high-level task planning. The low-level policy uses feasible, smoothly updated commands and posture initializations to support varied movement.

A. Reinforcement Learning Problem

B. Low-Level Whole-Body Policy

  • The low-level policy tracks high-level velocity and orientation commands while observing, but not directly controlling, the arm joints; its 15 leg and waist DoFs track a five-dimensional command.Source: B. Low-Level Whole-Body Policy · pp. 3–4

A. Reinforcement Learning Problem

  • The phase-conditioned PPO framework uses a state machine, phase prediction, phase-conditioned value estimation, and phase-gated rewards to structure multi-stage learning.Source: Figure 3 · p. 4

B. Low-Level Whole-Body Policy

  • Inverse-kinematics keyframes initialize valid postures, while feasible command bounds and residual delta updates constrain targets and provide command smoothness without heavy action-rate penalties.Source: B. Low-Level Whole-Body Policy · p. 4

C. Phase-Conditioned Policy Optimization

The method extends PPO by using task phases to structure long-horizon learning. Its components guide phase progression, behavior, value estimation, and exploration while training one policy.

C. Phase-Conditioned Policy Optimization

  • The framework extends PPO by using task structure and phase transitions to guide training across long-horizon tasks.The approach builds on local success conditions, such as the box clearing the ground, to trigger transitions.Source: C. Phase-Conditioned Policy Optimization · p. 4
  • The environment state machine assigns states to behavioral phases using task conditions, with local success conditions guiding transitions and hysteresis reducing premature regression.At test time, the actor executes the sequence without observing or being constrained by phase labels.Source: C. Phase-Conditioned Policy Optimization · p. 5
  • Phase-gated rewards activate or deactivate terms by phase, while transition incentives reward forward progress and penalize backward regression.For example, approach rewards give way to crouching rewards for lowering base height while maintaining stability.Source: C. Phase-Conditioned Policy Optimization · p. 5
  • A phase-prediction head trains the actor's shared representation to encode task-relevant structure, despite the actor not observing the active phase.The auxiliary head uses cross-entropy loss against the environment's phase label.Source: C. Phase-Conditioned Policy Optimization · p. 5
  • The phase-conditioned critic combines a global value head for overall task progress with phase-specific residual heads for local reward nuances.An L2 penalty on residual-head weights keeps the primary learning burden on the global head.Source: C. Phase-Conditioned Policy Optimization · p. 5
  • Multi-phase initialization samples episodes from randomized task phases, and curated keyframes help complex maneuvers learn gripping and holding behaviors.A curriculum progressively reduces mid-episode resets until all episodes start in the first phase.Source: C. Phase-Conditioned Policy Optimization · p. 5

IV. EXPERIMENTS

The experiments include hardware evaluations on multiple robot platforms and ablations of the proposed design choices.

IV. EXPERIMENTS

  • Hardware experiments across multiple robot platforms and ablations evaluate the proposed approach and its design choices.Source: §IV · p. 5

A. Hardware Deployments

The evaluation spans a Spot quadruped and a Unitree G1 humanoid performing pushing, lifting, carrying, and tire-maneuvering tasks. The G1 lift-and-carry task combines posture changes with sustained, contact-rich interaction.

A. Hardware Deployments

  • The Spot maneuvers a tire from the ground to a target, while G1 tasks include pushing one or two boxes, lifting a box, and lifting and carrying it.Source: A. Hardware Deployments · p. 5
  • G1 box lifting and carrying requires repeated transitions between upright and crouching postures, challenging both the framework and physical hardware.Source: A. Hardware Deployments · p. 5
  • The lift-and-carry task involves continuous forearm contact, creating a heavily under-constrained, contact-rich regime.Table II is described as summarizing the evaluated tasks' behavioral phases and phase counts.Source: A. Hardware Deployments · p. 6
  • Policies are trained in GPU-accelerated mjlab simulation, with object mass, size, and friction randomized to bridge the sim-to-real gap.G1 policies are exported in ONNX and deployed through custom ROS2; Spot uses a separate custom control framework.Source: A. Hardware Deployments · p. 6

1) Motion-Capture-Free Execution:

The G1 push-box and lift-box tasks run without motion capture using onboard state estimation and ArUco detection. Bimanual lifting and carrying required adaptations for actuator delay, contact slippage, and excessive gripping force.

1) Motion-Capture-Free Execution:

  • The G1 push-box and lift-box tasks execute without motion capture using onboard state estimation and ArUco marker detection.The policy showed zero-shot robustness despite training without explicit missing-data or marker-dropout modeling.Source: 1) Motion-Capture-Free Execution · p. 6

2) Sim-to-Real Transfer for Bimanual Grasping:

  • Randomized actuator delays up to 5 physics steps (25 ms at 200 Hz) during training addressed hardware arm oscillations that destabilized the grasp.The delays let the policy compensate for unmodeled dynamics and hardware latencies.Source: 2) Sim-to-Real Transfer for Bimanual… · p. 6
  • A friction curriculum starts metal-box contacts at sliding friction up to 3.0 and anneals to final values of 1.5–2.0 to address micro-slippage during extended holds.The curriculum responds to slippage associated with MuJoCo's soft contact model.Source: 2) Sim-to-Real Transfer for Bimanual… · p. 6
  • Clamping commanded-to-current joint-position deviation acts as an implicit torque limit, preventing excessive gripping force and supporting steady box holds up to 5 minutes.The selected offset limit was 7 N · m, computed via position-gain Kp, balancing secure clamping and thermal safety.Source: 2) Sim-to-Real Transfer for Bimanual… · p. 6
  • The G1 lift-box deployment pairs onboard state estimation with a first-person camera view tracking the ArUco marker.Source: Figure 4 · p. 6

B. Ablation Analysis

The ablation analysis compares framework components on Spot tire rolling and G1 two-box pushing. The G1 task tests multi-phase learning while remaining simpler to isolate than box carrying.

B. Ablation Analysis

  • The ablation study tests Spot tire rolling and G1 two-box pushing by removing framework components and tracking success throughout training.The authors chose these tasks to isolate architectural contributions from sensitive reward tuning.Source: B. Ablation Analysis · p. 6
  • G1 two-box pushing provides a rigorous multi-phase test while remaining more straightforward than complex box carrying.This choice helps separate architectural performance from sensitive reward tuning.Source: B. Ablation Analysis · p. 6
  • The analysis aims to isolate each component’s contribution by systematically removing it and tracking task success during training.Source: B. Ablation Analysis · p. 6

1) Auxiliary Phase Prediction:

The ablations examine phase-prediction supervision, multi-phase episode initialization, and value-network topology. They report substantial success-rate drops without phase prediction, collapse on the more complex task without multi-phase resets, and subtler gains from residual value architecture.

1) Auxiliary Phase Prediction:

  • Removing auxiliary phase-prediction supervision drops success from 89% ± 2% to 44% ± 9% on Spot and from 92% ± 2% to 14% ± 6% on G1.The ablation also reduces convergence speed; phase predictions track the task’s temporal structure across 1,000 successful Spot tire-rolling episodes, with minor discrepancies at transitions.Source: 1) Auxiliary Phase Prediction · p. 6, Figure 5 · p. 7

2) Multi-Phase Episode Initialization:

  • Without multi-phase resets, G1 two-box pushing collapses after mastering the first box, while the simpler task succeeds with high variance.Training only from the first phase leaves the policy exposed to only the task’s initial segment; multi-phase initialization is reported as critical for stable learning and faster convergence across extended horizons.Source: 2) Multi-Phase Episode Initialization · pp. 6–7

3) Value Network Topology:

  • The residual value architecture achieves higher success and lower training variance than MLP and fully decoupled-head baselines, though benchmark differences are subtle.The authors select residuals as a middle ground for phase-specific value changes while retaining task-wide representation and bounded discontinuity at phase boundaries.Source: 3) Value Network Topology · p. 7

V. CONCLUSION

The paper concludes that its PPO extension addresses long-horizon multi-phase loco-manipulation and is effective on hardware. It also identifies scaling, manual task design, and simulator fidelity as limitations.

V. CONCLUSION

  • The PPO extension is demonstrated on hardware for Spot tire rolling and G1 box lift-and-carry.Source: §V · p. 7
  • Scaling remains challenging as more behavior windows and longer horizons strain policy capacity and phase-specific value estimation.The method also requires manual phase segmentation and reward fine-tuning; changing box shape requires updating reset keyframes, and simulator contact limits complicate transfer to hardware.Source: §V · p. 7
  • Future directions include phase-guided off-policy learning, multi-robot coordination, and automatic phase discovery from offline data for offline-to-online learning.Source: §V · p. 7

Related papers

Loading 2610.36270v1…