PaperlayerView exact revision on arXiv

Source-linked AI summary

Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data

Songbo Hu, Qiayuan Liao, Yufeng Chi, Kevin Zakka, Yakun Sophia Shao, Pieter Abbeel, Koushil Sreenath

arXiv:2610.09117v1cs.ROcs.LG

TL;DR

Humanoid robots still face the challenge of planning contact-rich whole-body manipulation from egocentric vision and proprioception. Workhorse trains a visual planner and whole-body tracker from robot-free human demonstrations, and demonstrates real-robot tasks alongside simulated box-sorting results.

  • Problem

    Humanoid robots must plan contact-rich whole-body motions from egocentric RGB and proprioception, including when scenes change or perturbations require recovery.

    Source: §I · p. 1

  • Method

    Workhorse separately trains a visual planner and whole-body tracker from the same robot-free human five-link poses, without retargeting, and augments each policy with the other’s deployment errors.

    Source: §I · p. 2

  • Results

    The full system succeeds in 77% of simulated box-sorting episodes without pushes and 64% with 40 N for 1 s pushes every 5 s to 10 s.

    Source: B. Ablation · p. 7

  • Takeaways & Limitations

    Workhorse demonstrates real G1 box sorting, box catching, and suitcase toppling and climbing, with recovery from pushes or box removal during sorting.

    Source: D. Summary · p. 8

    One demonstrator recorded all data in one room, and success rates are from simulation; the second humanoid’s result remains simulation-only.

    Source: §VI · p. 8

Abstract

from arXiv · show

Humanoid robots still struggle to plan contact-rich whole-body manipulation from egocentric RGB and proprioception. Workhorse learns such manipulation from robot-free human demonstrations. A visual planner predicts five-link targets: the poses of the torso, both wrists, and both feet. A reinforcement-learning whole-body tracker follows them on the robot. Both policies train separately on the same recorded human poses, without retargeting. We augment the training data of each policy to imitate the errors that the other makes at deployment. On a real Unitree G1, Workhorse sorts boxes with its hands and a kick, catches a thrown box, and topples and climbs a suitcase. During box sorting, we show recoveries after a person pushes the robot or takes the box away. In a simulated copy of the demonstration room, the system completes box sorting in 77% of episodes, and in 64% under 40 N.s pushes. With both policies retrained from the same demonstrations, a simulated second humanoid completes box sorting in 83% of episodes without pushes.

I. INTRODUCTION

Whole-body manipulation requires robots to plan their own motions from egocentric vision and proprioception, including during dynamic interaction and recovery. Workhorse learns this capability from robot-free human demonstrations using a shared five-link interface and separately trained visual-planning and whole-body-tracking policies.

I. INTRODUCTION

  • Humanoid robots must plan whole-body manipulation motions from egocentric RGB and proprioception, including when the scene changes or perturbations require recovery.Source: §I · p. 1

II. RELATED WORK

  • Prior robot-free demonstrations largely used feet for locomotion and studied quasistatic tasks, whereas Workhorse demonstrates active-foot manipulation and recovery from perturbations.Source: §I · p. 2

I. INTRODUCTION

  • Workhorse’s planner and tracker share task-space poses for the torso, wrists, and feet, taken directly from human demonstrations without robot-specific retargeting.Source: §I · p. 2, Figure 2 · p. 2
  • A five-tracker, chest-camera rig records robot-free whole-body demonstrations, while separate policies train on the same five-link poses and augment data with the other policy’s deployment errors.Source: §I · p. 2
  • Workhorse evaluates the complete system in closed loop in a Gaussian-splat reconstruction of the demonstration environment.Source: §I · p. 2

A. Humanoid Whole-Body Controllers

Prior whole-body controllers generally track a reference or accept commands treated as ground truth, while human-data systems face trade-offs in capture and robot-specific conversion. Workhorse instead trains its tracker on the drift and jumps of planner commands and uses a lightweight rig that captures torso, wrist, and foot poses together.

A. Humanoid Whole-Body Controllers

B. Human Data Collection Devices

  • Workhorse’s lightweight rig records torso, wrist, and foot poses together in one gravity-aligned frame, avoiding separate device clocks and poorly conditioned foot-camera odometry.Source: B. Human Data Collection Devices · p. 2

C. Humanoid Policies from Human Demonstrations

  • Robot-free systems record key body-part poses and video, but prior approaches often retarget motion into robot-specific joint references that change the demonstration.Source: C. Humanoid Policies from Human… · p. 2
  • Workhorse trains both policies directly from the human five-link poses without retargeting; a new humanoid requires measuring five mounts, replacing its simulator model, and retraining.Source: C. Humanoid Policies from Human… · pp. 2–3

III. HUMAN DATA COLLECTION

  • The collection protocol splits continuous sessions into repetitions and perturbs objects, not the demonstrator, so recorded poses remain intended motion.Source: §III · p. 3

A. Rig

The collection system synchronizes chest-camera video with wearable-tracker poses for the torso, hands, and feet. Sessions vary object placement and perturb objects, while keeping the demonstrator’s recorded poses intentional.

A. Rig

  • Five wearable trackers capture the chest camera mount, hands, and feet at 60 Hz in a shared gravity-aligned frame, while the chest camera records 129° near-fisheye video at 30 frames/s.Source: A. Rig · p. 3

B. Recording Protocol

  • Each continuous session is divided at T-poses, and recordings vary lighting and object placement while perturbing objects but not the demonstrator.Source: B. Recording Protocol · p. 3

C. Synchronization and Processing

  • Three opening claps align tracker and camera clocks; after synchronization, streams disagree by 5.5 ms, tracker poses are resampled to 50 Hz, and images are selected by timestamp.Source: C. Synchronization and Processing · p. 3

IV. HUMANOID SKILL TRANSFER

Workhorse uses recorded human poses of five links as a shared interface between visual planning and whole-body tracking. Fixed robot-specific mounts convert those poses into link targets without retargeting or scaling the human motion.

A. Human Motion as Interface

1) From Human Motion to Five-Link Targets:

  • Each link target is the recorded alignment-frame pose shifted by a fixed rigid offset, and the planner predicts these targets for the tracker to follow.Source: 1) From Human Motion to Five-Link… · p. 4
  • The human motion is neither scaled nor retargeted; a second humanoid instead uses measured mounts and a replaced simulator model, followed by retraining both policies on the same demonstrations.Source: 1) From Human Motion to Five-Link… · p. 4

2) Torso Frames:

The policies use torso-frame five-link poses, with the planner predicting targets from images and pose history. Training adds drift to histories to better match the robot’s inputs.

2) Torso Frames:

  • Both policies represent the five-link poses in torso frames, making them independent of where the robot stands or faces.The planner’s frame is anchored at observation time t0; the tracker’s is anchored at its control step t.Source: 2) Torso Frames · p. 4

B. Visual Planner

  • At deployment, the planner’s history is drifted by tracker errors and pushes, so training adds drift to recorded histories.The drift can grow in closed loop as the tracker follows a chunk predicted from a drifted history.Source: B. Visual Planner · p. 4
  • The planner’s inputs differ from demonstrations because recorded images show a person while robot cameras see the robot’s body.The recorded images are edited to show the robot, and recorded histories receive added drift.Source: B. Visual Planner · p. 4, Figure 5 · p. 4

1) Observation, Output, and Architecture:

The visual planner maps image and five-link histories to a future chunk of five-link targets. Its training edits human images to show the robot and adds drift to pose histories while keeping target chunks clean.

1) Observation, Output, and Architecture:

  • The planner reads four images and nine samples of five-link history and predicts a 58-action chunk spanning 1.16 s.Images are 0.24 s apart; history samples are 0.12 s apart, and chunk actions are 0.02 s apart.Source: 1) Observation, Output, and Architecture · p. 4
  • A flow-matching policy with a diffusion-transformer backbone encodes each image and history sample as a token and uses cross-attention.A ResNet-18 encodes images; inference uses 10 Euler steps.Source: 1) Observation, Output, and Architecture · p. 4

2) Person-to-Robot Image Edit:

  • Offline image editing segments and removes the person, estimates and calibrates background depth, then renders the robot over pixels behind it.The robot is rendered from the recorded camera pose in the inverse-kinematics pose.Source: 2) Person-to-Robot Image Edit · p. 4

3) Planner Augmentation:

  • Planner-history drift follows a random walk backward from t0, while the chunk target remains the clean demonstration.The five links move together with a rigid drift, alongside smaller independent wrist and foot walks; images are warped consistently with camera motion.Source: 3) Planner Augmentation · p. 5, Figure 6 · p. 5
  • Training also varies brightness, contrast, and lighting gradient, and randomly masks history samples and links so the planner must use images.The photometric changes are shared across the four images in a sample.Source: 3) Planner Augmentation · p. 5

C. Whole-Body Tracker

The whole-body tracker converts five-link commands into joint setpoints using proprioception and reinforcement learning. Its reward combines motion style and task tracking, and its command training reproduces deployment drift and replanning jumps.

C. Whole-Body Tracker

  • The reinforcement-learning tracker turns five-link commands into joint setpoints while tracking wrists and maintaining balance under command drift or pushes.It builds on the BeyondMimic motion tracker [1].Source: C. Whole-Body Tracker · p. 5
  • The tracker observes proprioception only: IMU angular velocity, joint positions and velocities, and previous joint setpoints.Its command contains six samples spanning the next 0.2 s in the current torso frame.Source: C. Whole-Body Tracker · p. 5

1) Reward:

  • The reward combines a 14-link motion-style score with torso and wrist tracking in world coordinates and foot tracking in the torso frame.A clearance kernel rewards 10 cm of ankle height during recorded foot swings; penalties cover setpoint rates and joint-limit violations.Source: 1) Reward · p. 5

2) Command Augmentation:

  • Command augmentation restarts drift at each replan and trains against the drifted command, which can shift up to 10 cm and 10° yaw.Replans occur at random 0.2–1 s intervals; the command corrects at the next replan.Source: 2) Command Augmentation · p. 5

3) Training:

The tracker is trained in simulation on demonstration-derived starts and randomized conditions, while deployment aligns delayed sensor data and accommodates late planner chunks. Evaluation demonstrates real-robot tasks and reports simulated box-sorting success.

3) Training:

  • The tracker is trained per robot and task without real-robot data, using rollouts predicted by a learned dynamics model of the simulator.Training uses a spring in place of a held box and a suitcase model for the suitcase task.Source: 3) Training · p. 6

D. Deployment

  • Timestamping lets the 5-Hz planner and 50-Hz tracker handle sensors with different delays; the tracker skips chunk actions already in the past.The tracker interpolates command samples and transforms them into its current torso frame.Source: D. Deployment · p. 6
  • Overlapping actions from the previous chunk are inpainted into a new chunk, allowing the planner to deliver chunks late and run slower than the tracker.The overlap is moved into the new observation-time torso frame before inpainting.Source: D. Deployment · p. 6

V. SYSTEM EVALUATION

  • Real-robot demonstrations cover three tasks, while simulation evaluates design choices and retraining on a second humanoid from the same demonstrations.The simulated box-sorting table reports percentages over 100 episodes; success means placing all three boxes within 120 seconds without a fall.Source: §V · p. 6, Table II · p. 6

A. Whole-Body Capability

Workhorse demonstrates whole-body manipulation on a real G1 and evaluates simulated box-sorting success and ablations. The tests cover task completion, perturbation recovery, and the effects of removing training components.

A. Whole-Body Capability

  • On the real G1, Workhorse sorts boxes with hands and a kick, catches a thrown box, and pushes, topples, and climbs a suitcase.Box sorting handles three boxes in one episode without a reset; the suitcase weighs 14 kg.Source: A. Whole-Body Capability · p. 6
  • During box sorting, the robot stays upright and continues after pushes or kicks, and reaches again or walks to another box if one is taken.The demonstrations contained perturbed objects but no pushes on the demonstrator.Source: A. Whole-Body Capability · p. 6, Figure 8 · p. 7

B. Ablation

  • 77% of 100 episodes succeeded without pushes and 64% with pushes in simulated box sorting using the full system.Pushes were 40 N·s impulses on the pelvis.Source: B. Ablation · p. 7
  • The joint-command variant succeeded in 54% of episodes without pushes and 42% with pushes, while its online inverse-kinematics solver could hit singularities.With pushes, falls were 4%, versus 15% for the full system.Source: B. Ablation · p. 7
  • The link-local action-frame variant succeeded in 5% of episodes without pushes and 4% with pushes, and the robot fell in most episodes.The full system expresses all five poses in a shared torso frame.Source: B. Ablation · p. 7
  • Without projected gravity and torso height in the history, success fell to 1%, and the robot fell in almost every episode.The planner then had to estimate torso tilt and height from images.Source: B. Ablation · p. 7
  • Without planner augmentation, success fell to 20% without pushes and 0% with pushes, and the robot fell in most episodes.This ablation removes all planner augmentation and does not isolate its individual components.Source: B. Ablation · pp. 7–8
  • Without command augmentation, success was 76% without pushes and 52% with pushes, versus 77% and 64% for the full system.With pushes, falls rose from 15% to 23%.Source: B. Ablation · p. 8

C. Cross-Embodiment

The authors retrain both policies for a simulated Unitree H2 using the same demonstrations and no new recording. The H2 completes box sorting in 83% of episodes, while the comparison with the G1 is within the stated uncertainty.

C. Cross-Embodiment

  • 83% of 100 episodes succeeded for the simulated H2 without pushes, compared with 77% for the G1; the difference is smaller than the roughly ±11-point 95% interval.Both policies were retrained from the same demonstrations, with no new recording.Source: C. Cross-Embodiment · p. 8
  • The simulated H2 planner used images rendered in the reconstructed room while the robot replayed the demonstrations.The room was reconstructed as a simulated copy of the demonstration room.Source: C. Cross-Embodiment · p. 8
  • The H2 is full-size and closer than the G1 to the demonstrator in size and proportion.Source: C. Cross-Embodiment · p. 8

D. Summary

Workhorse performs the three tasks on the real G1 and recovers from perturbations; the full system is the strongest-performing simulated variant. The reported scope remains limited to one demonstrator and one room, with success rates evaluated only in simulation.

D. Summary

  • On the real G1, Workhorse performs the three tasks and recovers from perturbations; no simulated variant succeeds more often than the full system.With pushes, the joint-command variant falls less often; link-local action frames or removing projected gravity or planner augmentation lead to falls in most episodes.Source: D. Summary · p. 8

VI. LIMITATIONS AND FUTURE WORK

  • Results come from one demonstrator and one room or its reconstruction; real-robot trials demonstrate capability, while success rates are reported only in simulation.Under 40 N·s pushes, the full system falls in 15% of simulated episodes.Source: §VI · p. 8
  • The retrained H2 policies have been evaluated only in simulation; running them on a real H2 remains future work.The H2 result is in simulation.Source: §VI · p. 8
  • The five-link interface omits fingers and the head, leaving gripper or dexterous-hand state and camera-carrying neck motion outside the current interface.The paper identifies adding gripper or hand state and a neck as possible future extensions.Source: §VI · p. 8
  • Lighthouse base stations limit data collection to one room, constraining recording and testing in other places.The paper notes that pose estimation without base stations would allow recording elsewhere.Source: §VI · p. 8
Loading 2610.09117v1…