Source-linked AI summary
Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion
Zipeng Fu, Xuxin Cheng, Deepak Pathak
TL;DR
Legged manipulators need coordinated arm–leg control, but hierarchical pipelines require substantial engineering and can produce non-smooth motion. The paper learns a unified reinforcement-learning policy using Advantage Mixing and Regularized Online Adaptation, evaluates it on a low-cost untethered platform, and reports dynamic coordinated behaviors across tasks. The approach also shows improved out-of-distribution adaptation and whole-body coordination, while general-purpose object interaction remains open.
Problem
Hierarchical arm–leg control for legged manipulators suffers from coordination gaps, error propagation, non-smooth motion, and difficult high-degree-of-freedom training.
Method
The paper learns one reinforcement-learning policy for leg and arm control, using Advantage Mixing and Regularized Online Adaptation for training and sim-to-real transfer.
Results
The unified policy demonstrates dynamic, agile, and coordinated behaviors across teleoperation, vision-guided, and open-loop mobile-manipulation tasks on a 6K USD untethered platform.
Takeaways & Limitations
Unified whole-body control coordinates leg and arm actions, with legs expanding manipulation workspace and arms helping balance under larger perturbations.
Takeaways & Limitations
General-purpose object interaction involving occlusion and soft objects remains a challenging open research direction.
Abstract
from arXiv · showhide
An attached arm can significantly increase the applicability of legged robots to several mobile manipulation tasks that are not possible for the wheeled or tracked counterparts. The standard hierarchical control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion. However, this is ineffective. It requires immense engineering to support coordination between the arm and legs, and error can propagate across modules causing non-smooth unnatural motions. It is also biological implausible given evidence for strong motor synergies across limbs. In this work, we propose to learn a unified policy for whole-body control of a legged manipulator using reinforcement learning. We propose Regularized Online Adaptation to bridge the Sim2Real gap for high-DoF control, and Advantage Mixing exploiting the causal dependency in the action space to overcome local minima during training the whole-body system. We also present a simple design for a low-cost legged manipulator, and find that our unified policy can demonstrate dynamic and agile behaviors across several task setups. Videos are at https://maniploco.github.io
1 Introduction
The paper motivates unified whole-body control for legged manipulators because hierarchical arm–leg control demands substantial engineering and can produce poorly coordinated motion. It introduces a unified learned policy, adaptation and training mechanisms, and a low-cost platform evaluated on dynamic mobile-manipulation tasks.
- Legged manipulators extend robots beyond locomotion, enabling everyday tasks that require manipulation.
- Hierarchical arm–leg controllers require substantial engineering and can suffer coordination failures, error propagation, and slow, non-smooth motions.
- Whole-body learning is challenging because the system has 19 degrees of freedom, continuous high-frequency dynamics, and conflicting manipulation–locomotion objectives.
- The proposed unified policy coordinates quadruped legs and the manipulator to reach end-effector poses while commanding locomotion.
- Regularized Online Adaptation addresses simulation-to-real transfer, while exploiting action-space causal structure stabilizes and accelerates learning.
- A fully untethered Go1-based platform costing 6K USD demonstrates coordinated behaviors for teleoperation, vision-guided control, and open-loop tasks.
2 Method: A Unified Policy for Coordinated Manipulation and Locomotion
The method learns one policy to coordinate leg and arm control for whole-body manipulation and locomotion, using advantage mixing and regularized online adaptation to improve training and transfer.
- Unified policy: A single neural network receives base, leg, arm, command, and previous-action information, then outputs target positions for the arm and legs.Joint-space position control converts target arm positions to torques through PD controllers.
- Unified policy: The policy maximizes discounted return from separate manipulation and locomotion rewards, including squared leg-joint energy consumption.The energy term encourages lower average consumption and lower variance across leg joints.
- Advantage Mixing: Advantage Mixing uses manipulation and locomotion advantages with a curriculum parameter β that increases linearly from 0 to 1.It initially attributes manipulation returns mainly to arm actions and locomotion returns mainly to leg actions, then gradually mixes the objectives.
- Regularized Online Adaptation: Regularized Online Adaptation jointly trains a privileged-information encoder and an observation-history adaptation module, regularizing their environment latents while removing the two-phase teacher-student pipeline.The regularization strength λ increases from 0 to 1 through a fixed linear scheme.
- Evaluation: The unified policy achieves the best performance at the same energy consumption compared with separate legs-arm and uncoordinated policies.The comparison uses the command ranges specified in Table 2.
- Deployment: The system supports joystick teleoperation, RGB-tracking closed-loop control, and open-loop playback of human demonstrations during deployment.The unified policy and adaptation module execute jointly onboard.
3 Experimental Results
Experiments evaluate the unified policy, Advantage Mixing, Regularized Online Adaptation, and real-world whole-body behaviors on a low-cost untethered legged manipulator. The method improves coordination, robustness, pick-up performance, and dynamic locomotion-manipulation behavior across the tested setups.
- 3.2 Simulation Experiments: The unified policy outperforms separate and uncoordinated policies because arm and leg actions jointly use manipulation and locomotion rewards with aggregate observations.Baselines struggle to maintain small base acceleration, producing larger velocity-following and end-effector pose errors.
- 3.2 Simulation Experiments: Advantage Mixing speeds learning by addressing credit assignment: without it, the policy learns end-effector following first while neglecting locomotion.The resulting task-focused-then-merged training process acts as a curriculum-like mechanism.
- 3.2 Simulation Experiments: Regularized Online Adaptation is more robust than RMA and Domain Randomization on out-of-distribution environment parameters, reducing EE Error by 20%.Regularization yields smaller latent imitation error, while Domain Randomization can trivially stand in difficult environments.
- 3.3 Real-World Experiments: Real-world tests show coordinated base rotation and arm motion, successful pick-up behavior on easy and hard tasks, and agile arm movement while walking on uneven grass.Compared with MPC+IK, the method’s failures are mainly attributed to visual target-position mismatch, whereas MPC+IK also fails from IK singularity or self-collision.
4 Related Work
Prior legged-locomotion methods achieve strong terrain traversal but require meticulous controller design and manual tuning. Prior mobile-manipulation systems commonly separate leg and arm control, limiting coordination and dynamic movement demonstrations.
- Legged Locomotion: Traditional model-based legged control succeeds on difficult terrains but requires meticulous controller design and extensive manual tuning.Adding an arm increases the difficulty of this design process because of its extra weight and movement.
- Mobile Manipulation: Prior quadruped mobile-manipulation work combines an RL leg controller with a model-based arm controller, leaving the modules separate and demonstrating no dynamic movements.The related approach tracks desired arm end-effector positions while using RL to maintain balance.
5 Discussion and Limitations
The authors present preliminary object-interaction results while identifying general-purpose interaction and vision-based control as challenging directions for future work.
- General-purpose object interaction involving occlusion and soft objects remains a challenging open research direction.
- Future extensions include vision-based policies using egocentric cameras mounted on the torso and arm.
- The authors also propose using front legs to climb obstacles and extend reach when the arm alone cannot reach a tabletop object.
A Experiment Videos
Real-world videos show the legs and arm coordinating dynamically, with leg motion increasing arm reach and stability.
- The legs and arm function in coordination during real-world experiments.The authors direct readers to compiled result videos documenting the behaviors.
- The legs bend and stretch to increase the arm’s reach and attain stability.
B Regularized Online Adaptation Details
Regularized Online Adaptation alternates between privileged-information guidance and policy adaptation using replayed transitions, with a curriculum that increases regularization over training.
- Regularized Online Adaptation uses a privileged information encoder, adaptation module, unified policy, and replay buffer.
- The procedure periodically computes adaptation representations and otherwise uses privileged representations while collecting transitions from parallel environments.
- Every 20 iterations, the adaptation module is updated by matching its representation to the privileged encoder’s detached representation.The algorithm sets H to 20.
- On other iterations, the policy and privileged encoder optimize the advantage-mixing reinforcement-learning objective with an L2 regularization term.
C Simulation Details
The simulation connects customized quadruped and arm models in IsaacGym, samples end-effector commands in a base-independent frame, and terminates episodes under stability or time constraints.
- Simulation Details: Customized URDF models rigidly connect a Unitree quadruped and Interbotix arm for parallel IsaacGym simulation.Fractal noise generates rough terrain with two octaves, lacunarity 2.0, gain 0.25, frequency 10Hz, and amplitude 0.15m.
- Simulation Details: End-effector position commands are generated by sampling spherical coordinates and transforming them into world coordinates relative to the robot base.The transformation uses base orientation and position.
- Simulation Details: Setting the base height to 0.53 and base roll and pitch to zero makes end-effector commands independent of base height, roll, and pitch.This design is intended to encourage smooth arm motion and whole-body coordination.
- Simulation Details: Episodes last at most 1000 steps and terminate early when robot height or body roll and pitch exceed command-dependent stability thresholds.The thresholds include 0.28m minimum height and ±0.2 radians for relevant roll or pitch conditions.
- Simulation Details: The policy runs at 50Hz while simulation runs at 200Hz, with separate leg and arm stiffness, damping, default positions, and target-position ranges.Leg stiffness and damping are 50 and 1; arm stiffness and damping are 5 and 0.5.
D Training Details
Training uses a neural policy with separate leg and arm output heads, trained over a large PPO sample budget. Advantage Mixing is included to support simultaneous walking and grasping.
- Policy Architecture: The policy uses a multilayer perceptron whose outputs split into separate leg and arm heads.The first hidden layer has 128 dimensions, followed by two 128-dimensional hidden layers per head.
- Training Configuration: Training runs for 10,000 iterations, totaling 2 billion samples and 200,000 gradient updates.PPO hyperparameters are listed in the supplementary training table.
- Training Configuration: Advantage Mixing helps the unified policy learn walking and grasping together, whereas its absence leaves velocity error consistently high.Without mixing, the policy remains at a local minimum focused on end-effector commands.
E Advantage Mixing Details
Advantage Mixing modifies the unified policy objective by combining manipulation and locomotion advantages. This helps avoid a training failure where the policy follows end-effector commands without learning to walk.
- Objective: The policy objective is defined for a diagonal-Gaussian policy using sampled transition batch D.The objective is written with respect to the policy parameters θπ.
- Objective: Advantage Mixing combines manipulation and locomotion advantages in the training objective.The mixed objective weights the locomotion advantage with β alongside the manipulation advantage.
- Training Effect: Without Advantage Mixing, the unified policy fails to learn walking and remains at a local minimum that follows end-effector commands.Initial exploration in leg action space destabilizes the base and harms manipulation, discouraging further leg exploration.
F Real-World Setup and Experiment Details
The real-world platform combines a quadruped, a mounted robot arm, onboard sensing, and fixed-weight policy deployment. Experiments include vision-guided tracking and pickup-task comparisons.
- Hardware Setup: The platform uses a Unitree Go1 quadruped with 12 actuated DoFs and a mounted 6-DoF Interbotix WidowX 250s arm.A parallel gripper and RealSense D435 camera are included, with both systems powered by the quadruped battery.
- Deployment: The unified policy and adaptation module run onboard at 50Hz with fixed weights on a Raspberry Pi 4.The arm software runs on an Nvidia TX2, with UDP communication between the computers.
- Vision Tracking: Vision-guided tracking uses the average pose of two AprilTags as the target pose.The camera and tag offset parameters are documented in the supplementary camera-parameter table.
- Pickup Evaluation: The pickup comparison evaluates the proposed method against MPC+IK over 10 real-world trials using average time to completion.The table defines p_end as the sampled ground goal position and TTC as average completion time.
- Gripper Control: The gripper is controlled outside the policy, either by joystick during teleoperation or by a scripted proximity rule during vision tracking.In vision tracking, the gripper closes after remaining near the AprilTag-specified target for one second.