PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-to-Real Learning for Humanoid Box Loco-Manipulation

Jeremy Dao, Helei Duan, Alan Fern

arXiv:2310.03191v1cs.RO

TL;DR

Humanoid box loco-manipulation requires coordinating object handling with whole-body balance, and successful sim-to-real RL demonstrations for bipedal humanoids were lacking. The paper trains separate pickup and locomotion skills with box-aware rewards and a relative action space, then combines them for Digit. The learned policies transfer to hardware, including pickups of boxes from 1 to 8 kg.

  • Problem

    Humanoid loco-manipulation requires full-body coordination to move and manipulate objects while maintaining balance, and the authors report no prior successful sim-to-real RL demonstration on a bipedal humanoid.

    Source: §I · p. 1

  • Method

    The authors learn separate box pickup and locomotion skills using interaction-focused rewards and a relative action space, then combine them into a Digit loco-manipulation system.

    Source: §I · p. 1

  • Results

    All trained policies transferred successfully to hardware, with pickups of different-sized boxes from 1 to 8 kg, including boxes on the ground or a table.

    Source: §VI · p. 6

  • Takeaways & Limitations

    Separately learned policies can be combined to perform full box-moving episodes on Digit, from walking to the box through setting it down.

    Source: §VII · p. 6

    The demonstrated system relies on a human operator and ArUco markers; removing that reliance is left for future work.

    Source: §VII · p. 6

Abstract

from arXiv · show

In this work we propose a learning-based approach to box loco-manipulation for a humanoid robot. This is a particularly challenging problem due to the need for whole-body coordination in order to lift boxes of varying weight, position, and orientation while maintaining balance. To address this challenge, we present a sim-to-real reinforcement learning approach for training general box pickup and carrying skills for the bipedal robot Digit. Our reward functions are designed to produce the desired interactions with the box while also valuing balance and gait quality. We combine the learned skills into a full system for box loco-manipulation to achieve the task of moving boxes from one table to another with a variety of sizes, weights, and initial configurations. In addition to quantitative simulation results, we demonstrate successful sim-to-real transfer on the humanoid r

I. INTRODUCTION

The paper addresses the gap in learned loco-manipulation for bipedal humanoids, where moving objects requires full-body coordination and balance. It proposes reward-designed skills and a relative action space, then combines separately learned policies in a sim-to-real system for Digit.

I. INTRODUCTION

  • The work targets sim-to-real reinforcement learning for humanoid loco-manipulation, a setting the authors say had not yet been demonstrated successfully on a bipedal humanoid.The task requires full-body coordination while manipulating and moving objects and maintaining balance.Source: §I · p. 1
  • Reward functions specify hand contact timing, box-moving forces, and hand-motion regulation to learn smooth, gentle motions that still achieve pickup.Source: §I · p. 1
  • For box pickup, a relative action space adds policy PD setpoint commands to current joint positions, yielding faster learning and less forceful box interaction.Source: §I · p. 1
  • The system combines individually learned standing, walking, and pickup skills and demonstrates sim-to-real loco-manipulation on Digit.The authors describe it as the first learning controller to achieve a loco-manipulation task on real-world hardware, to their knowledge.Source: §I · p. 1

A. Loco-Manipulation for Humanoids

Prior humanoid loco-manipulation work included model-based lifting and trajectory planning, while learning approaches often relied on demonstrations or separate locomotion and manipulation modules. This paper's related work contrasts those approaches with learned, whole-body control aimed at hardware execution.

A. Loco-Manipulation for Humanoids

  • Model-based humanoid box-lifting methods plan whole-body motion and compensate for box forces, but their conservative stability constraints yield pickups taking over 20 seconds.Source: A. Loco-Manipulation for Humanoids · p. 2
  • Learning-based humanoid manipulation has often used trajectory tracking or demonstrations, which can introduce motion-tracking or human-robot alignment errors.Source: A. Loco-Manipulation for Humanoids · p. 2

B. RL for Loco-Manipulation

  • For legged systems with attached arms, prior approaches either separated locomotion and manipulation modules or learned unified control; this work separates skills while retaining full-joint control.The authors contrast their full-joint policies with Ma et al. (2022), whose learned policy controls locomotion while an MPC planner controls the arm.Source: B. RL for Loco-Manipulation · p. 2
  • Xie et al. (2023) targets the same humanoid box pickup and locomotion task, whereas this work seeks to reduce trajectory-tracking reliance and emphasize hardware execution.Source: B. RL for Loco-Manipulation · p. 2

III. SYSTEM OVERVIEW

The system decomposes box loco-manipulation into five behavior-specific policies with restricted transitions. Policies share a recurrent learning framework and are trained in randomized simulation with PPO.

III. SYSTEM OVERVIEW

  • The task uses five distinct policies: WALK, STAND, PICKUP, WALKWITHBOX, and STANDWITHBOX.Source: §III · p. 2
  • Transitions are restricted: the robot goes through STAND rather than moving directly from WALK to PICKUP.The allowed transitions define the target behavior workflow.Source: §III · p. 2, Figure 2 · p. 2
  • Policies use a shared LSTM framework with policy-specific commands, producing PD setpoints at 50 Hz for a 2 kHz controller.The network has two 128-dimensional recurrent hidden layers; its input includes a 67-dimensional robot-state vector.Source: §III · p. 2
  • Training uses MuJoCo, clipped-objective PPO, dynamics randomization, and noisy true simulation state rather than a state estimator.Randomized parameters include body mass, joint damping, center-of-mass location, and friction.Source: §III · p. 2

A. Box Pickup Policy

The box-pickup policy uses phased hand motion and rewards that coordinate box interaction with stable standing and smooth movement. Training varies box properties and starting conditions to support pickup across configurations.

A. Box Pickup Policy

  • The pickup command includes box dimensions, mass, starting pose, target position, and phase indicators; after pickup, box pose is approximated by the average hand pose.The box pose is explicitly known only at behavior onset, avoiding real-time box-pose estimation during the behavior.Source: A. Box Pickup Policy · p. 3
  • Adding policy outputs to current motor positions, rather than a static neutral offset, resulted in faster pickup-policy learning.The authors attribute this likely to pickup motions having greater profile variance from neutral than locomotion motions.Source: A. Box Pickup Policy · p. 3
  • Domain randomization perturbed the policy’s box-mass estimate by ±0.5 kg, dimensions by ±5 cm, and starting location by ±5 cm; boxes weighed 1–10 kg and measured 20–45 cm per dimension.Starting boxes could be on the ground or up to 1.3 m in the air, and target height was above the starting location. Pickup phases used hand-selected contact and lift times, tcontact = 100 and tlift = 175.Source: A. Box Pickup Policy · pp. 3–4
  • The reward combines hand-trajectory tracking, correctly timed box interaction, stable standing, and motion regulation for smoothness and sim-to-real transfer.The contact reward is activated only at the end of the contact phase, helping prevent early contact.Source: A. Box Pickup Policy · pp. 3–4
  • The hand trajectory is linearly interpolated through timed waypoints, approaching beside the box, making contact, and moving the box to its target.The hands are 10 cm from the box at 1.5 seconds, reach its transverse faces at 2 seconds, and reach the final waypoint at 3.5 seconds.Source: A. Box Pickup Policy · pp. 3–4, Figure 3
  • Training assigns a −0.1 sparse penalty for self-collision or either hand exceeding 1.0 m/s, and terminates for torso height below 40 cm, pitch above 35°, foot contact loss, table contact, or the box on the ground.Source: A. Box Pickup Policy · p. 4
  • Episodes also terminate if the hands miss box contact after the contact countdown or the box remains on the table after the pickup countdown, each with a 0.5-second grace period.Source: A. Box Pickup Policy · p. 4

B. Locomotion Policies

The locomotion policies adapt standard walking and standing learning setups for carrying a box. Their commands and reward terms account for box properties and carrying behavior.

B. Locomotion Policies

  • The regular locomotion reward is augmented with hand-position terms that keep the hands in front of the torso and equidistant from each other.Box locomotion adds further box-related reward terms to the regular locomotion setup.Source: B. Locomotion Policies · pp. 4–5
  • Box locomotion commands include box dimensions, mass, and desired carrying height, alongside randomized walking velocities and turning rates.The box-height command ranges from 1.0 to 1.3 m; commanded forward velocity ranges from −0.5 to 1.0 m/s.Source: B. Locomotion Policies · p. 4
  • Box locomotion episodes start with the box in the hands and terminate if contact is lost; training also randomizes the box mass and dimensions by ±0.5 kg and ±5 cm.Source: B. Locomotion Policies · p. 5

C. Standing Policies

The standing policies learn transitions from walking to static standing, with a separate box-standing variant that accounts for the carried box. Both prioritize stable standing, while box standing additionally maintains box contact.

C. Standing Policies

  • The normal standing policy receives only the robot state, with no command input, because its goal is to stand still.Source: C. Standing Policies · p. 5
  • The box-standing policy uses the normal standing setup with minor reward modifications and commands for box dimensions, mass, and desired height.Training starts from randomized walking states and randomizes the box-height command.Source: C. Standing Policies · p. 5
  • Box standing adds the box-interaction reward and terminates if either hand loses box contact; its box mass and dimension inputs are randomized by ±0.5 kg and ±5 cm.Source: C. Standing Policies · p. 5

D. Skill Transitions

The system connects individually trained skills through transitions that each policy must handle. Earlier policies generate initial states for training later skills, and action interpolation smooths handoffs.

D. Skill Transitions

  • The policies are connected through allowed transitions, with each skill trained from initial states generated by an earlier policy.Source: D. Skill Transitions · p. 5
  • Regular walking generates low-speed states for standing-policy training, and standing then generates initial states for box pickup.Source: D. Skill Transitions · p. 5
  • Actions are linearly interpolated between policies over 10 timesteps during both training and execution to smooth transitions.Initial-state generation also saves the last applied action with the robot state.Source: D. Skill Transitions · p. 5

V. SIMULATION RESULTS

The learned policies show strong performance under randomized simulation conditions, and the pickup policy generalizes beyond its training distribution. Ablations show that hand-trajectory rewards and relative action commands improve pickup learning efficiency.

Overall Policy Performance

  • Across 10K evaluation episodes, all five policies achieve high success rates and relatively small box-position errors, with robustness to randomized scenarios and transition start states.Success requires the robot not to fall and, for box behaviors, to retain the box; error is measured on successful episodes.Source: §V · p. 5, Table IV · p. 6

Limits of PickUp Policy

  • The PickUp policy successfully handles boxes much larger and heavier than those in training, based on tests varying one box parameter at a time with others held at their training-distribution means.The tests disable domain randomization and increment the parameter under test until pickup fails.Source: §V · p. 5, Table V · p. 6

PickUp Ablation

  • Removing hand-trajectory tracking still permits successful pickup but approximately doubles the samples needed to reach the same reward.Without trajectory tracking, each hand target remains at the center of a box face in its target position throughout training.Source: §V · p. 6
  • Using a fixed-offset action space learns significantly slower than adding policy outputs to current joint positions, with slowdown comparable to removing hand-trajectory tracking.The authors report no similar sample-efficiency loss for standing and walking policies.Source: §V · p. 6

VI. HARDWARE RESULTS

All trained policies transferred successfully to Digit, including transitions between skills. Hardware pickup worked across boxes of different masses and sizes and from both floor and table starting positions, although execution required operator support and policy-specific trim adjustments.

Hardware Results

  • All trained policies, including transitions between skills, transferred successfully to hardware; pickups worked for boxes weighing 1–8 kg and starting on the ground or a table.The tested boxes also varied in size.Source: §VI · p. 6
  • Hardware execution used a human operator for skill transitions, navigation, and walking commands, while box size and mass inputs were hardcoded.Box pose was obtained using Digit’s RGBD camera and an ArUco marker on the box.Source: §VI · p. 6
  • Policies leaned in hardware and required policy-specific orientation trims; the selected trims did not work for the heaviest boxes, where increased trim was needed.The authors suspect force or contact inconsistencies between simulation and hardware may contribute to the issue.Source: §VI · p. 6

VII. CONCLUSION

The paper presents a learned box loco-manipulation system for Digit, with separate policies for five behaviors. It reports high simulation performance across varied boxes and successful sim-to-real transfer for full episodes, while identifying autonomy and transfer quality as future work.

VII. CONCLUSION

  • The system divides box loco-manipulation into five behaviors, each learned as a separate policy.The behaviors support full episodes of walking to a box, picking it up, carrying it to a target, and setting it down.Source: §VII · p. 6
  • In simulation, the policies achieved high performance across boxes with varying masses, sizes, and starting locations.Source: §VII · p. 6
  • The authors demonstrated successful sim-to-real transfer for full loco-manipulation episodes using learned policies, which they describe as the first such demonstration to their knowledge.Source: §VII · p. 6
  • Future work targets removing reliance on a human operator and ArUco markers, and improving sim-to-real transfer through domain randomization and hardware data.Source: §VII · p. 6

Related papers

Loading 2310.03191v1…