PaperlayerView exact revision on arXiv

Source-linked AI summary

HULK: Learning Whole-Body Forceful Loco-Manipulation for Humanoids

An Dang, Arturo Flores Alvarez, Yu-Ming Chen, Conor Mc Gartoll, Helen Sun, Aaron Ames, Nima Fazeli, Manikantan Nambi

arXiv:2610.08970v1cs.ROcs.LG

TL;DR

Humanoid loco-manipulation must maintain balance under sustained loads that shift the robot’s weight. HULK uses MPC reward guidance and a training-time capture-point barrier in learned whole-body control. On hardware, it achieves the highest completion rate and maximum completed load on three of four tasks, tying AMO on payload carry.

  • Problem

    Sustained payloads shift a humanoid’s weight and challenge balance, while forceful whole-body interaction remains comparatively underexplored beyond wrist-only or horizontal push–pull control.

    Source: §I · pp. 1–2

  • Method

    MPC reward guidance supplies loaded-dynamics signals, and a capture-point control barrier augments the wrist-force teacher during training while retaining network-only inference.

    Source: §I · p. 2

  • Results

    HULK achieves the highest completion rate and maximum completed load on three of four hardware tasks, ties AMO on payload carry, and its CP-CBF teacher has lower loaded translational errors and smaller DCM excursions than Ours.

    Source: B. Simulation Results · p. 6, C. Real-world Results · p. 7

  • Takeaways & Limitations

    On the tested hardware tasks, HULK handles cart push/pull, payload carry, and carrying objects supported against the torso or held at the side.

    Source: C. Real-world Results · p. 7

    Grasp acquisition, regrasping, and rapid contact changes remain outside the demonstrated scope, which starts from established grasps.

    Source: §V · p. 7

Abstract

from arXiv · show

Humanoid loco-manipulation of large, heavy objects demands forceful interaction across the entire body. However, such payloads shift a humanoid's center of mass and impose sustained loads across the upper body, challenging balance and command tracking. We present HULK, a whole-body control framework for forceful loco-manipulation. Using model predictive control (MPC) to guide reinforcement learning with predictions of the loaded dynamics, we train two teachers: one tracks arm motions under wrist forces, and the other locomotes while holding large objects against the body. A capture-point control barrier function augments the wrist-force teacher during training to improve balance under load. We distill both teachers into a single policy. Evaluation spans simulation and the Unitree G1. In simulation, the teacher with the barrier function achieves the lowest forward and lateral velocity tracking errors at 10 kg per arm among evaluated controllers and reduces aggregate divergent component of motion (DCM) excursion magnitude by 35.7% relative to MPC-guided reinforcement learning alone. Our wrist-force teacher withstands torso push disturbances of up to 130 N.

I. INTRODUCTION

Forceful humanoid loco-manipulation requires balance under sustained upper-body loads, a comparatively underexplored challenge beyond wrist-only or horizontal push–pull force control. HULK combines MPC reward guidance with training-time capture-point balance constraints for learned whole-body control.

I. INTRODUCTION

  • Persistent loads shift the robot’s weight and compete for balance-related actuation, making forceful whole-body work relevant to tasks such as carrying heavy objects while walking.Unlike transient disturbances, these loads persist throughout the task.Source: §I · pp. 1–2

II. RELATED WORKS

  • Forceful interaction remains comparatively underexplored because existing approaches address compliance, wrist-force resistance, or horizontal push–pull moments rather than sustained whole-body loading.The cited approaches do not model whole-body object interaction during training.Source: §I · p. 2, §II · pp. 2–3

I. INTRODUCTION

  • HULK extends learned whole-body control to forceful interactions across upper-body contacts, using MPC reward guidance and training-time capture-point constraints while retaining network-only inference.The approach trains specialist teachers and distills them into a student policy.Source: §I · p. 2

II. RELATED WORKS

  • Prior MPC-guided learning uses wrist-applied interaction forces, whereas HULK extends the controller to height, torso-tilt, and arm commands and models payloads and whole-body grasps.The related method does not address contact against the torso.Source: §II · p. 3

III. METHODOLOGY

HULK trains a whole-body controller for payload-driven loco-manipulation using two specialist teachers and a distilled student. MPC plans provide reward guidance for reinforcement learning rather than being executed directly.

III. METHODOLOGY

  • The framework targets one whole-body controller that follows planar velocity, base height, torso orientation, and arm commands while loads act on the upper body.Two specialist teachers are distilled into the deployed student policy.Source: §III · p. 3

A. Problem Formulation

  • The control problem is formulated as a partially observable Markov decision process in which a parameterized policy maximizes expected discounted return.The formulation specifies state, observation, action, reward, transition, initial-state distribution, and discount factor.Source: A. Problem Formulation · p. 3

B. MPC Reward Guidance

  • MPC generates interaction-force, center-of-mass, and pelvis-rotation plans that guide the teachers’ rewards; the policy tracks those plans instead of executing them.This gives the policy a model-based signal for payload-induced changes to the reduced-order state.Source: B. MPC Reward Guidance · pp. 3–4

C. Centroidal Dynamics

HULK uses a reduced centroidal model that includes upper-body interaction forces and payload effects. Its footstep references also account for sustained external forces when planning touchdown locations.

C. Centroidal Dynamics

  • The reduced-order MPC state collects center-of-mass position, base orientation, and centroidal linear and angular momentum, while its inputs include foot and wrist wrenches.Payload force and moment enter the dynamics as interaction wrenches rather than as decision variables.Source: C. Centroidal Dynamics · p. 4

D. Force-Aware References

  • The force-aware Raibert reference offsets touchdown locations to account for sustained external forces applied above the ground, with the offset scaled by a gain.The force contribution is related to its application height and body weight.Source: D. Force-Aware References · pp. 4–5

E. Teacher Training

HULK trains two teachers for different interactions: one tracks static arm poses under wrist forces, while the other locomotes while maintaining grasps on varied objects.

E. Teacher Training

  • The wrist-force teacher holds sampled static arm poses while resisting independent wrist forces up to ±40 N horizontally and −50 to +5 N vertically.Forces are held for 3–5 seconds and ramped through a per-environment magnitude curriculum.Source: E. Teacher Training · p. 5
  • The whole-body-grasp teacher learns to locomote while maintaining object grasps, using 130,000 offline-generated grasps across multiple geometries.Each environment begins with an object and compatible grasp already established; training warm-starts from the wrist-force teacher.Source: E. Teacher Training · p. 5

F. Capture-Point Balance Guidance via CBF-RL

HULK adds a capture-point barrier to the wrist-force teacher during training, correcting commands and penalizing residuals while leaving the exported policy unfiltered. The section also describes mode-conditioned DAgger distillation and experiments targeting loaded tracking, disturbance recovery, and transfer to forceful hardware tasks.

F. Capture-Point Balance Guidance via CBF-RL

  • The capture-point control barrier corrects wrist-force teacher commands during training and penalizes residuals, while the exported policy runs unfiltered.The guidance is incorporated through CBF-RL.Source: F. Capture-Point Balance Guidance via… · p. 5
  • The force-aware capture point is measured against a support region that includes contacting soles and predicted touchdown footprints, with h_cp marking the admissible excursion margin.The figure depicts a recovery step into a separate admissible footprint and marks h_cp = 0 as the boundary.Source: F. Capture-Point Balance Guidance via… · p. 5, Figure 4 · p. 5

G. Student Distillation

  • A mode-conditioned student is trained with DAgger on its own visited states, using the selected frozen teacher’s action as the target.The student uses the teachers’ observation space.Source: G. Student Distillation · p. 5

IV. EXPERIMENTS

  • The experiments assess commanded-motion tracking under sustained load, disturbance recovery, and transfer to forceful loco-manipulation tasks.The real-world tasks include cart push/pull, payload carry, fridge carry, and canister carry, all while walking.Source: §IV · p. 5, Figure 5 · p. 6

A. Experimental Setup

  • Simulation comparisons use identical commands and perturbation schedules, with results reported as mean ± sample standard deviation across five evaluation-seed aggregates per fixed policy.The simulation comparisons include FALCON, Thor, and AMO; the HULK rows evaluate wrist-force teachers, while hardware evaluates the distilled student zero-shot.Source: A. Experimental Setup · p. 6
  • Carrying tests apply 10 kg per arm after a 2 s loading ramp and 2 s settling period; separate one-second torso pushes span 50–130 N in three directions.Push recovery is measured by each episode’s minimum post-push DCM margin; less negative margins indicate smaller worst departures outside the support envelope.Source: A. Experimental Setup · p. 6
  • Wrist-force rejection tests apply randomized 3D forces of 0, 20, or 40 N per wrist while the robot tracks arm and velocity commands.Upper-body error is RMS tracking error over common arm joints; composite root error sums horizontal velocity-error magnitude and absolute yaw-rate error over time.Source: A. Experimental Setup · p. 6
  • Hardware trials compare controllers at common commands and object placement; success requires task completion without falling, protective stop, or dropping the object.Payload carry additionally requires staying within a prescribed path-deviation threshold, and fixed-load success rates use task-specific loads.Source: A. Experimental Setup · p. 6

B. Simulation Results

The CP-CBF teacher improves loaded translational tracking and DCM excursion relative to the wrist-force teacher, while hardware results show strong performance across cart and torso-supported carrying tasks. The discussion identifies limits in runtime safety and in the demonstrated contact and motion scope.

B. Simulation Results

  • The CP-CBF teacher combines lower loaded translational errors with smaller DCM excursions than Ours, though AMO has the lowest mean turn-rate error.The smaller DCM excursions suggest improved recovery quality that push survival alone may hide; negative margins indicate recovery can leave the support envelope.Source: B. Simulation Results · p. 6, Table I · p. 7
  • At the strongest tested wrist force, both HULK teachers perform competitively with FALCON in arm and composite root tracking, with a mean arm-error difference of 0.02 rad.Arm errors remain nearly constant as force increases, while FALCON retains the lowest means.Source: B. Simulation Results · p. 6, Table II · p. 7
  • CP-CBF has higher root error than the HULK baselines at both nonzero force levels.The teachers train on sampled poses held under load with MPC and CP-CBF guidance emphasizing balance; FALCON uses a torque-aware force curriculum.Source: B. Simulation Results · p. 7, Table II · p. 7

C. Real-world Results

  • HULK has the highest observed completion rate and maximum completed load on three of four hardware tasks, and ties AMO on payload carry.In cart push/pull, HULK completes 5/6 trials versus 3/6 for FALCON at the common load and pulls up to 300 kg versus 200 kg in the separate load sweep.Source: C. Real-world Results · p. 7
  • For torso-supported loads, HULK completes 4/4 trials with both the fridge and canister, while all baselines complete 0/4 canister trials.Maximum completed loads are 7.0 kg for the fridge and 6.1 kg for the canister, versus baseline maxima of 6.0 and 2.6 kg, respectively.Source: C. Real-world Results · p. 7

V. DISCUSSION AND LIMITATIONS

  • Force application point determines its moment about the CoM, coupling arm configuration to the support needed to sustain a load.Transfer from wrist forces to forearm loading suggests learned compensation can extend beyond the trained contact location, but does not establish generalization to arbitrary contact arrangements.Source: §V · p. 7
  • The exported policy is unfiltered, so the training-time barrier provides no runtime safety guarantee.The model and training regime also leave grasp acquisition, regrasping, rapid contact changes, and transitions between stable configurations outside the demonstrated scope.Source: §V · p. 7

APPENDIX

The appendix derives the capture point under a constant-height LIPM and uses its relation to the support polygon to motivate a bounded-distance balance criterion.

A. Capture Point Derivation

  • The derivation assumes a constant-height LIPM, with cxy the robot CoM's horizontal position and p the ground-plane center of pressure.The CoM height z is held constant, and ω0 is defined as the natural frequency of the LIPM.Source: A. Capture Point Derivation · p. 10
  • With fixed p, the LIPM solution separates into a decaying term and a divergent term; setting the divergent coefficient to zero yields the capture point.Under the constant-height LIPM, the resulting instantaneous capture point is also the planar divergent component of motion (DCM).Source: A. Capture Point Derivation · p. 10
  • When the capture point lies inside the support polygon, an admissible CoP can be driven toward it; outside, no CoP under the current support configuration can coincide with it.The criterion describes stopping without stepping under LIPM assumptions; during dynamic walking, a capture point outside the current polygon alone does not imply a fall.Source: A. Capture Point Derivation · p. 10
  • Bounding the distance between the capture point and feasible support region keeps the robot near states where ground-reaction forces can modulate CoP toward the capture point.This motivates using that distance as a safety function to guide the wrist-force teacher during training.Source: A. Capture Point Derivation · p. 10

B. CBF-RL Training and Reward Guidance

The training procedure combines a locally approximated barrier constraint with a reward penalty, then removes both filter and auxiliary reward for deployment.

B. CBF-RL Training and Reward Guidance

  • The safe set is defined by states with hcp(s) ≥ 0; under a valid CBF constraint and feasible modeled dynamics, trajectories starting there remain there.The stated condition is ḣcp + κCBF hcp ≥ 0, with κCBF > 0.Source: B. CBF-RL Training and Reward Guidance · p. 10
  • The training filter locally approximates the barrier condition and uses a CBF-QP to find the closest lower-body joint target satisfying it.An admissible proposal passes unchanged; a violating proposal with nonzero sensitivity is projected to the constraint boundary in closed form.Source: B. CBF-RL Training and Reward Guidance · p. 10
  • The auxiliary reward penalizes both the proposed barrier violation and the correction, encouraging the policy to need less intervention.The simulator executes the filtered target while optimization uses the policy's original sampled action; the filter and auxiliary reward are removed at deployment.Source: B. CBF-RL Training and Reward Guidance · pp. 10–11
  • The approximate, capped training filter and unfiltered deployed policy do not inherit the ideal invariance guarantee.The sensitivity approximation differentiates only the CoM-position contribution to the capture point and uses lower-body joints.Source: B. CBF-RL Training and Reward Guidance · p. 11

C. Wrist-Force Teacher MPC Optimization

The wrist-force MPC incorporates prescribed wrist loads into predictions of the robot’s momentum, while the grasp MPC allocates contact forces to balance an object and guide locomotion. The grasp formulation also tracks planned grasp wrenches and falls back to measured reactions when its plan is invalid.

C. Wrist-Force Teacher MPC Optimization

  • The wrist-force formulation specializes the dynamics to two foot wrenches and two wrist forces, with wrist perturbations modifying the predicted linear and angular momenta.The unhatted rates use the same robot state and foot wrenches without wrist perturbations.Source: C. Wrist-Force Teacher MPC Optimization · p. 11
  • The wrist-force MPC treats prescribed wrist forces as external loading, without explicitly modeling payload mass distribution or inertial dynamics.Forces act in the world frame, and wrist application points come from forward kinematics of the retargeted arm command.Source: C. Wrist-Force Teacher MPC Optimization · p. 11

D. Grasp Teacher MPC Optimization

  • The grasp teacher balances each assigned object under a quasi-static model, then supplies the resulting reaction to the locomotion MPC.Object equilibrium is enforced through active torso-and-arm grasp contacts.Source: D. Grasp Teacher MPC Optimization · p. 11
  • Grasp-force guidance tracks planned torso and arm wrenches; if contact, finite-wrench, or solver checks fail, measured reactions supply the locomotion MPC load.The tracking reward uses a 25 N scale for grasp-wrench error.Source: D. Grasp Teacher MPC Optimization · p. 11

E. Wrist-Force Teacher Training

Wrist-force teacher training varies arm-pose coverage and applied-force magnitude through episode-based curricula. A fixed subset remains force-free to preserve unloaded training exposure.

E. Wrist-Force Teacher Training

  • Arm targets are sampled as individual frames from retargeted motions, blended from the measured pose, then held before joint-limited targets are resampled.The motion bank contains 48,233 frames across 252 motions and retains 14 arm joint angles.Source: E. Wrist-Force Teacher Training · pp. 11–12
  • Pose coverage and force magnitude each adapt at reset according to completed episode length, increasing after longer episodes and decreasing after shorter ones.Coverage changes by 0.01 around 200-step and 40-step thresholds; force magnitude changes by 0.02 around 200 steps.Source: E. Wrist-Force Teacher Training · p. 12
  • A fixed 20% of environments remain force-free, preserving unloaded exposure and discouraging reliance on external forces for balance and command tracking.These environments do not advance the force curriculum.Source: E. Wrist-Force Teacher Training · p. 12

F. Grasp Teacher Training

The grasp teacher trains from authored object–grasp configurations and adjusts payload bounds based on episode outcomes. Its training setup separates grasp-teacher adaptation from convergence evidence and specifies distinct teacher MDPs and perturbation settings.

F. Grasp Teacher Training

  • Each environment is assigned an object–grasp pair, with the authored arm posture and object transform restored together at reset.Grasp geometry is preserved by disabling motion-bank coverage.Source: F. Grasp Teacher Training · p. 12, Figure 6 · p. 12
  • A grasp’s payload bound starts at 0.5 kg, rises after a full 20 s episode without terminal failure, and falls after terminal failure.Mass remains fixed during an episode, and the bound follows the identity of the grasp and object across resets.Source: F. Grasp Teacher Training · p. 12

G. Teacher Training Curves

  • The plotted grasp-teacher return is early adaptation rather than convergence, based on one seed-42 run with 501 new updates.The wrist-force run spans 15,000 updates; both curves report rolling returns over 100 completed episodes.Source: G. Teacher Training Curves · p. 12, Figure 7 · p. 13

H. Teacher MDPs

  • The teachers use separate MDPs and shared actor observations; their rewards combine command tracking, MPC guidance, and regularization, with object retention added for grasping.Teacher critics receive privileged simulation observations.Source: H. Teacher MDPs · p. 12, Table IV · p. 13
  • Training randomizes robot properties and delays actions, while wrist-force training adds periodic base perturbations and sustained pushes up to 150 N.Grasp training disables those perturbations and instead samples loads from multiple objects; grasp episodes can terminate on object drop.Source: H. Teacher MDPs · pp. 12–13

I. Reward Formulation

The appendix specifies how teacher rewards and training are partitioned across body regions, with shared trajectories but separate reward streams for lower- and upper-body policy pairs.

I. Reward Formulation

  • The reward is formulated separately for lower and upper body partitions as a time-scaled sum of weighted terms.The appendix defines positive-part and indicator notation for reward terms.Source: §I · p. 13

J. Teacher Dual-Policy Training

  • Teacher actors use stacked noisy observations, whereas each critic receives 140 current-state privileged observations without added noise.Actors do not directly observe object pose or contact forces.Source: Table V · p. 13

I. Reward Formulation

  • Reward conventions gate motion terms by commanded speed and define gait signals for contact, touchdown, swing height, velocity, and force.Lengths and times are expressed in meters and seconds, and angles are wrapped to [−π, π).Source: §I · p. 13

J. Teacher Dual-Policy Training

  • Each teacher has separate lower- and upper-body actor–critic pairs that share trajectories but train on separate reward streams.Their Gaussian actions concatenate into 15 lower-body and 14 upper-body joint offsets.Source: J. Teacher Dual-Policy Training · pp. 13–14
  • The body partitions share transitions but use separate rewards and updates under PPO, with GAE λ = 0.95 and a target KL divergence of 0.01.The adaptive learning-rate schedule targets KL divergence separately for each body partition.Source: Table VI · p. 14, J. Teacher Dual-Policy Training · p. 14
  • The appendix organizes teacher rewards into tracking and gait terms, plus posture, regularization, and grasp terms.The tables distinguish upper-body terms from other reward terms and indicate absent terms with a dash.Source: Table VII · p. 15, Table VIII · p. 16

Related papers

Loading 2610.08970v1…