PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Whole-Body Control for Legged Loco-Manipulation

Minghuan Liu, Zixuan Chen, Xuxin Cheng, Yandong Ji, Ri-Zhao Qiu, Ruihan Yang, Xiaolong Wang

arXiv:2403.16967v5cs.ROcs.CVcs.LG

TL;DR

Legged loco-manipulation asks how robots with arms can autonomously manipulate objects using visual observations while coordinating their legs and arms. VBC combines low-level whole-body control with high-level visual planning, and the authors report successful real-world pickup across diverse objects and heights. The system’s stated limitations include visual tracking loss, gripper design, depth estimation, and compounding error.

  • Problem

    The paper studies how legged robots with arms can perform mobile manipulation autonomously from visual observations while using leg–arm coordination to extend workspace.

    Source: §1 · p. 2

  • Method

    VBC pairs a low-level whole-body policy tracking body-velocity and end-effector commands with a high-level policy that proposes them from visual inputs, trained in simulation and transferred to real robots.

    Source: §1 · p. 2

  • Results

    Real-world pickup experiments reported high success rates, retrying behavior, and generalization across 14 objects and ground, box, and table heights.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported system adapts to objects at varying heights through whole-body behavior, including ground objects that static-height methods cannot reach without arm–leg coordination.

    Source: §2 · p. 3

    The authors identify visual tracking loss, inadequate gripper design, imprecise depth estimation, and compounding error as system and task limitations.

    Source: §5 · p. 8

Abstract

from arXiv · show

We study the problem of mobile manipulation using legged robots equipped with an arm, namely legged loco-manipulation. The robot legs, while usually utilized for mobility, offer an opportunity to amplify the manipulation capabilities by conducting whole-body control. That is, the robot can control the legs and the arm at the same time to extend its workspace. We propose a framework that can conduct the whole-body control autonomously with visual observations. Our approach, namely Visual Whole-Body Control(VBC), is composed of a low-level policy using all degrees of freedom to track the body velocities along with the end-effector position, and a high-level policy proposing the velocities and end-effector position based on visual inputs. We train both levels of policies in simulation and perform Sim2Real transfer for real robot deployment. We perform extensive experiments and show significant improvements over baselines in picking up diverse objects in different configurations (heights, locations, orientations) and environments.

1 Introduction

The paper studies autonomous visual loco-manipulation with legged robots, coordinating legs and arms to extend manipulation beyond the arm’s reach. It presents a hierarchical control framework and reports real-world pickup across varied objects and heights.

1 Introduction

  • Coordinating the legs and arm expands the robot’s workspace; the arm alone can hardly reach objects on the ground.The paper frames autonomous control from visual observations as a central challenge, given the need for precise manipulation, stability, and robustness in diverse environments.Source: §1 · p. 2
  • Visual Whole-Body Control (VBC) uses a high-level policy to propose end-effector poses and body velocities, tracked by a low-level policy.The low-level policy is trained with reinforcement learning; a privileged task teacher is then distilled into a depth-image-based student through online imitation learning, with all training in simulation before Sim2Real deployment.Source: §1 · p. 2
  • High success rates and emergent retrying and generalization were reported for 14 objects across ground, box, and table heights.The experiments used a Unitree B1 quadruped with a Unitree Z1 arm.Source: §1 · p. 2

2 Related Work

Prior legged loco-manipulation work includes optimization, reinforcement learning, and autonomous systems, but the paper identifies limits in human involvement, visual autonomy, and whole-body adaptation. VBC is presented as an autonomous, vision-based system that adapts to object heights through whole-body behavior.

2 Related Work

  • Earlier loco-manipulation methods [8] [9] required human involvement and teleoperation, while autonomous quadruped systems [10] [11] [12] used static-height locomotion and lacked whole-body adaptation to objects at different heights.Source: §2 · p. 2
  • Trajectory optimization, leg-based pushing and lifting, and hierarchical pushing each address parts of the problem, with cited approaches requiring restricted settings or omitting visual inputs or an arm [14, 15, 16, 17].The cited motion-planning approach [14] requires dense scene descriptions, while reports limited surroundings and simple objects.Source: §2 · p. 3
  • VBC is described as autonomous and vision-based, adapting to varying object heights through whole-body behavior and exhibiting retrying when pickup fails.The paper contrasts this with static-height methods [10] [11] [12], which cannot reach ground objects without arm–leg coordination, and reports indoor and outdoor operation without external limits.Source: §2 · p. 3

3 Visual Whole-Body Control

VBC pairs a whole-body goal-reaching controller with a task-planning policy that supplies commands from visual and proprioceptive observations. Its training and deployment pipeline combines reinforcement learning, privileged teacher learning, imitation, and real-robot sensing.

3 Visual Whole-Body Control

  • VBC combines a low-level goal-reaching policy with a high-level task planner; static-height methods cannot reach ground objects without arm–leg coordination.The planner uses segmented depth images and proprioception to provide commands, while the low-level policy tracks them.Source: §3 · p. 3, Figure 2 · p. 3

3.2 High-Level Policy for Task Planning

  • The high-level policy plans task-relevant commands for the low-level controller, using visual observations to guide autonomous loco-manipulation.The framework separates this task-specific planning role from the low-level controller’s goal-reaching role.Source: §3 · p. 3

3.1 Low-Level Policy for Whole-Body Goal-Reaching

  • The low-level policy uses reinforcement learning to track end-effector poses and body velocities across terrains, with sampled goals in a height-invariant frame.It controls the quadruped’s 12 leg joints; arm target joint angles are derived from end-effector pose commands using inverse kinematics.Source: §3.1 · pp. 3–4

3.2.1 Privileged Teacher Policy

  • A privileged teacher is trained with reinforcement learning and task-specific rewards; it outputs target end-effector positions and robot velocity commands for the low-level policy.Its observations include object shape features and pose; training randomizes terrain friction, robot properties, controller gains, and pickup configurations.Source: §3.2.1 · pp. 4–5

3.2.2 Visuomotor Student Policy

  • The visuomotor student uses segmented depth images and proprioception, and is distilled from the state-based teacher with DAgger.The method uses teacher-corrected student transitions and depth-image augmentations to bridge simulation and reality.Source: §3.2.2 · p. 5

3.3 Real World Deployment

  • Real deployment uses a B1 quadruped with a Z1 arm and two cameras; TrackingSAM propagates object masks from trial-start annotations during pickup trials.The B1 onboard computer runs low-level control, while an additional computer handles high-level visual planning; masks are tracked at about 10 fps.Source: §3.3 · pp. 5–6

4 Experiments

The experiments evaluate VBC on simulated and real-world object pickup across varied object categories and heights. VBC outperforms the reported baselines, with real-world tests also showing autonomous retries and generalization to unseen shapes.

4 Experiments

  • The real-world evaluation uses 14 objects across three height configurations, while simulation tests 33 YCB objects grouped into seven shape categories.Real objects are placed at visible random locations and orientations; almost all are unseen in simulation.Source: §4 · p. 6

4.1 Simulation Results

  • VBC’s student policy uses visual observations and is tested against alternative student policies across heights; the non-hierarchical baseline is omitted because it did not reach meaningful results.Figure 6 reports mean success rates and the maximum and minimum across seven categories.Source: Figure 6 · p. 6, §4.1 · p. 7

4.2 Real-World Experiments

  • Real-world pickup trials allow up to five attempts per reset, and the reported emergent retrying behavior improves performance.Figure 7 averages success time over five trials for each object and height.Source: Figure 7 · p. 7, §4.2 · p. 7

4.1 Simulation Results

  • VBC achieves the best performance in 4/7 simulated object categories and significantly improves on the floating-base baseline at almost every tested height.With 3D shape features, performance improves on irregular objects but worsens on regular ones compared with VBC without shape features.Source: Figure 6 · p. 6, §4.1 · p. 7, Table 1 · p. 7

4.2 Real-World Experiments

  • In real-world tests, VBC surpasses the teleoperated stationary-leg baseline at all height settings, while the baseline fails at 0.0m and 0.3m.VBC is generally better even at 0.5m, where the baseline is equivalent to a static arm; it also generalizes to unseen shapes.Source: Figure 8 · p. 8

5 Conclusions and Limitations

The paper concludes that VBC is a fully autonomous quadruped loco-manipulation system trained through a hierarchical pipeline. It identifies visual tracking loss, gripper design, depth estimation, and compounding error as limitations, alongside related work and background on learning and whole-body control.

5 Conclusions and Limitations

  • VBC is a fully autonomous quadruped loco-manipulation system with a hierarchical training pipeline that trains low-level goal-reaching and high-level task-planning policies using reinforcement and imitation learning.The authors list visual tracking loss, inadequate gripper design, imprecise depth estimation, and compounding error as system limitations.Source: §5 · p. 8

A Extended Related Works

  • Prior legged-locomotion work includes model-based controllers and learning-based policies trained in simulation for transfer to real robots.The related-work discussion also notes wheeled mobile-manipulation advances while observing that most such platforms cannot operate on challenging wild terrain.Source: A Extended Related Works · p. 15

B.1 Reinforcement Learning

  • Reinforcement learning is framed as policy learning in a discounted infinite-horizon Markov decision process to maximize accumulated reward.The MDP specifies state and action spaces, dynamics, an initial-state distribution, reward, and discount factor.Source: §B.1 · p. 15

B.2 Proximal Policy Optimization

  • PPO maximizes a surrogate objective that constrains the size of policy updates.The objective uses the probability ratio between current and previous policies and the advantage function.Source: §B.2 · p. 16

B.3 Imitation Learning

  • The paper uses online DAgger imitation learning, requesting expert actions online and training a student to minimize action distance on states it encounters.In practice, the action-distance loss is mean squared error.Source: §B.3 · p. 16

B.4 Whole-Body Control

  • Whole-body control combines locomotion and manipulation in one framework; here, a learned policy generates signals for a quadruped and 6-DoF arm with underlying PD controllers.The quadruped follows external commands such as base velocity, while the arm follows an end-effector pose command through inverse kinematics.Source: §B.4 · p. 16

C.1 Low-Level Training

Low-level training teaches the policy to track body-velocity and end-effector goals while producing stable whole-body behavior. Goal sampling, randomized environments, and online adaptation support robust transfer from simulation to real deployment.

C.1 Low-Level Training

  • The low-level reward combines command following, energy use, survival, and gait-phase tracking to encourage command tracking, smooth motion, continued operation, and steady walking.The phase rewards include swing-phase tracking for trotting.Source: §C.1 · pp. 16–17
  • Goal poses use a height-, roll-, and pitch-invariant frame, so the sampled end-effector targets depend on horizontal movement and yaw rather than body height.This design encourages leg and body-height adjustment to reach targets beyond the arm’s workspace.Source: §C.1 · p. 17
  • The end-effector command is interpolated from its current position to a sampled target, which is resampled at fixed intervals or when a command would cause self-collision or ground collision.The robot trains to track the velocity commands and end-effector pose.Source: §C.1 · p. 17, Figure 9 · p. 17
  • Regularized Online Adaptation trains an encoder on privileged environment information, then trains an adaptation module to estimate its latent representation from recent low-level observation history.The adaptation module starts training after the policy and encoder converge; the loss includes a gradient-stop operator and a Lagrangian multiplier.Source: §C.1 · pp. 17–18, Figure 10 · p. 18
  • Training randomizes terrain, friction, robot mass, and center of mass; the low-level policy uses an MLP and an adaptation module that processes history rollouts.The policy’s privileged encoder models environment physics during training, while the adaptation module is used at test time.Source: §C.1 · p. 18

C.2 High-Level Training

The high-level policy is trained with staged task rewards, auxiliary shaping, domain randomization, and transfer-oriented techniques. The section also reports improved goal-reaching precision with IK-based end-effector control and close simulation-to-real behavior.

C.2 High-Level Training

  • High-level training uses separate approaching, task-progress, and completion rewards, plus auxiliary rewards for smoother actions and object-directed body and gripper behavior.Only one stage reward is active at a time; auxiliary rewards also discourage large commands and joint acceleration and help avoid losing object tracking.Source: §C.2 · pp. 18–19
  • Training techniques include action delay, command clipping and reward curricula, occasional object repositioning for retrying, and stopping base velocity while the gripper closes.Object position and pose are randomly changed with 10% probability; the command-penalty reward is added after pickup learning, and stopping during gripper closure supports stop-then-pick behavior.Source: §C.2 · pp. 19–20

D Summary of Domain Randomization

  • During simulated training, low-level randomization aimed to develop robust walking that adapts to different surroundings and body poses for end-effector goals, while high-level teacher randomization targeted manipulation generalization.The teacher additionally randomized 6-8 low-level policy calls per high-level call (0.12-0.16s, or 6.25-8.33Hz), arm-controller Kp and Kd parameters, and table heights of 0-0.5m.Source: D Summary of Domain Randomization · p. 20

E Motivations for IK-based Arm Control

  • IK-based end-effector pose prediction achieves better position and orientation precision than target-joint prediction, supporting effective high-level policy training for complex manipulation.The goal-reaching evaluation sampled goals as in training and ended when the end-effector was within 2 cm or time expired.Source: E Motivations for IK-based Arm Control · p. 20

F Sim-Real Visual Observation Comparison

  • Real-world trajectories match simulation behavior, while TrackingSAM supplies deployment segmentation masks and depth images are hole-filled and clipped.Figure 11 compares sampled trajectories and segmented depth images; the real-world view need not show the same object as the simulation view.Source: F Sim-Real Visual Observation Comparison · p. 20, Figure 11 · p. 21

G Real World System

The real-robot system runs its vision-based planner and low-level controllers on onboard computers in an asynchronous client-server arrangement. Its high-level runtime limits planning to 6–8 Hz while low-level control runs at 50 Hz.

G Real World System

  • Inference and arm control run on the B1 onboard computer, while the vision-based planner runs on a Jetson Orin, leaving the system untethered from PCs.The B1, Z1 arm, and Orin are powered by the robot’s onboard battery.Source: G Real World System · p. 21
  • The high-level runtime limits planning to 6–8 Hz, compared with 50 Hz for quadruped control and more than 800 Hz for the arm controller.The component runtime analysis averages measurements over 100 real-robot runs, but the supplied table caption gives no component values.Source: G Real World System · pp. 21–22, Table 5 · p. 22
  • A goal-receiver client requests updated high-level commands from a vision server, while the quadruped and arm processes continue executing the latest commands asynchronously.The vision pipeline tracks and segments annotated objects with TrackingSAM before sending image and proprioception observations to the high-level model.Source: G Real World System · p. 22

H Details of Used Objects

The study uses 34 simulated objects and 14 real-world objects, spanning multiple object shapes and categories.

H Details of Used Objects

I Failure Analysis and Future Works

The system’s failures involve dependencies across modules, imperfect depth, gripper and vision limitations, and required manual target specification. The authors identify color-dependent tasks as outside the visual input scope and propose hardware and pipeline improvements.

I Failure Analysis and Future Works

  • The hierarchical pipeline depends on coherent, precise modules; depth errors on reflective objects, the beak-like gripper, and unstable tracking also constrain reliable pickup.TrackingSAM losing track through camera deviation or occlusion, or confusion with similarly colored objects, is reported as the most common real-world failure cause.Source: I Failure Analysis and Future Works · p. 22
  • Before autonomous pickup, users must manually specify the target object in both robot-mounted cameras.Source: I Failure Analysis and Future Works · p. 22
  • Mask and segmented-depth inputs exclude color information, restricting tasks that depend on color differentiation.Examples include distinguishing Fanta from Sprite cans, category-based garbage sorting, and pressing a green button.Source: I Failure Analysis and Future Works · p. 23
  • Future work could automate annotation and search, randomize object sizes, improve the gripper and RGB-D cameras, and simplify the pipeline.The authors report training with 33 distinct objects and strong generalization to unseen real-world objects.Source: I Failure Analysis and Future Works · p. 23

J Additional Details of TrackingSAM

TrackingSAM initializes object segmentation with a few user clicks, then tracks the selected object in wrist- and head-camera images.

J Additional Details of TrackingSAM

Loading 2403.16967v5…