UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers

1 Introduction

• UMI-on-Legs combines real-world human demonstrations with simulation-based reinforcement learning to support cross-embodiment mobile manipulation.The framework aims to provide a scalable path for expressive skills on dynamic robot embodiments.Source: §1 · p. 2
• A hand-held UMI gripper supplies robot-free task demonstrations, letting operators collect diverse challenging tasks without robot trial-and-error.The diffusion-policy data is agnostic to robot embodiment beyond the end-effector.Source: §1 · p. 2
• A simulation-trained whole-body controller tracks end-effector trajectories, avoiding the need to model manipulation objects, task rewards, or object dynamics.This separates robot-centric controller training from manipulation-task simulation setup.Source: §1 · p. 2
• Task-frame end-effector trajectories connect visual manipulation policies to embodiment-specific whole-body control, supporting complex skills and cross-embodiment transfer.The controller can use trajectory preview to anticipate future motions.Source: §1 · p. 2, Figure 3 · p. 3
• An iPhone-based odometry system supports real-time in-the-wild task-space tracking without assuming tightly integrated, precisely calibrated and synchronized hardware.The deployment system also features onboard power and compute.Source: §1 · p. 2, Figure 2 · p. 2
• Over 70% success was achieved across prehensile pick-and-place, non-prehensile weight pushing, and dynamic tossing tasks.The evaluation used onboard sensing and in-the-wild generalization.Source: §1 · p. 2

2 Related Work

• Prior pick-and-place quadrupeds transfer simulation-trained systems to real robots [12] [16], while wheeled platforms demonstrate more challenging skills using engineered primitives [17] [18] [19] [20] [21].Mobile-ALOHA [27] collects demonstrations, but these systems assume quasi-static manipulation and/or heavy, low-center-of-mass bases.Source: §2 · p. 3
• Legged-robot reinforcement learning uses massively parallel simulation [3] [7], but prior quadruped whole-body controllers require human teleoperation or replay outside simplified April-Tag settings [8] [10].Their robot-specific commands and hardware-dependent data collection make effective autonomous learning costly; He et al. use a task-frame interface but robot/controller-specific heuristic demonstrations and a tethered April-Tag setup.Source: §2 · p. 3
• Cross-embodiment behavior cloning has advanced through policy architectures, sensors, action spaces, data quality, size, and cost [1] [2] [27] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49].Zero-shot transfers using ego-centric or wrist-mounted views still assume lower-level controllers can precisely track predicted target-pose sequences.Source: §2 · p. 3

3 Method: Universal Manipulation Interface on Legged Robots

• Task-frame end-effector trajectories connect manipulation and whole-body control, letting the controller compensate for base motion while the manipulation policy focuses on task progress.The interface supports intuitive hand-held demonstrations and asynchronous low-frequency manipulation with a high-frequency controller.Source: §3 · pp. 3–4, Figure 4 · p. 4

3.1 Manipulation Policy with Behavior Cloning

• The diffusion manipulation policy uses wrist-camera views to predict future end-effector poses; cup rearrangement uses a UMI checkpoint, while pushing and tossing use policies trained from collected data.The policy is trained with real-world UMI demonstrations.Source: §3 · pp. 3–4, §3.1 · p. 4

3.2 Whole-body Controller with Reinforcement Learning

• A reinforcement-learned whole-body controller tracks manipulation trajectories in simulation and outputs arm and leg joint targets, without simulating the task or its objects.This avoids setting up manipulation assets, engineering task rewards, or modeling object dynamics for controller training.Source: §3 · p. 3, §3.2 · p. 4
• Unlike body-frame tracking, task-frame trajectory tracking trains the controller to maintain end-effector pose while compensating for base movements and perturbations.The distinction matters when arm momentum causes substantial base movement, including during dynamic arm actions.Source: §3.2 · pp. 4–5, Figure 4 · p. 4
• Controller observations combine robot state with densely sampled near-term and 1,000ms-future target poses; entangled position-orientation rewards use a precision curriculum.An MLP outputs leg and arm joint targets and runs at 50Hz on the onboard CPU.Source: §3.2 · p. 5

3.3 System Integration

• Deployment pairs a Go2 quadruped with an ARX5 arm and iPhone pose estimation; simulation randomizes physical parameters and models 20ms of control latency.The rear-mounted iPhone avoids adding arm weight, arm-phone collisions, and visual occlusion while minimizing motion blur.Source: §3.3 · p. 5

4 Experiments

• Experiments assess dynamic tossing, response to unseen object dynamics, and zero-shot policy transfer; real-world success rates are averaged over 20 episodes.Source: §4 · p. 5, §4.3 · p. 7
• Tossing success was 75% with motion capture and 70% with iPhone odometry; under large disturbances, controller success was 95% and 90%, respectively; manipulation-policy success was 85% and 80%, respectively.Source: §4.1 · p. 6, §4.2 · p. 7, §4.3 · p. 8

4.1 Capability: Whole-Body Dynamic Tossing

• Tossing requires coordinated joint power, momentary one- or two-foot balance, and use of body inertia to regain balance.The controller’s leap-and-tuck strategy helps prevent forward falls and supports a soft landing.Source: §4.1 · p. 6, Figure 5 · p. 6
• Removing task-frame tracking or manipulation trajectories hurts precision, while removing trajectory observations reduces survival by 5.4% and adds 130 W of jitter.Calibrated trajectory replay with motion capture improves tossing by 10%.Source: §4.1 · pp. 6–7

4.2 Robustness: End-effector Reaching Leads to Robust Whole-body Pushing

• Without simulated forces or weights, the controller adapts to kettlebell resistance by leaning forward to push and moving its feet to reach distant targets.The push-and-catch response was observed with both 10-lb and 20-lb kettlebells; the 20-lb one encountered static friction more often.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 Scalability: Plug-and-play Cross-Embodiment Manipulation Policies

• A prior manipulation policy was deployed on the quadruped without fine-tuning, in an unseen environment, despite the shift from a fixed-base arm to a legged manipulator.The task requires precise 6-DoF end-effector movement, including prehensile and non-prehensile actions.Source: §4.3 · pp. 7–8

4.4 Limitations

• The system is limited to gripper-based manipulation; extending the interface to whole-body demonstrations and adding collision avoidance or force feedback are future directions.The gripper limitation is inherited from UMI.Source: §4.4 · p. 8

• UMI-on-Legs combines real-world human demonstrations with simulation-based reinforcement learning for quadruped manipulation.Source: §5 · p. 8
• Task-frame end-effector trajectories make demonstrations from hand-held devices compatible with precise pick-and-place, pushing, and dynamic tossing skills.Source: §5 · p. 8
• The authors see the framework as a step toward transferring manipulation policies across robot platforms, while noting that the community lacks a unanimous foundation manipulation policy.Source: §5 · p. 8

• Huy Ha and Shuran Song are listed with Stanford University affiliations.Source: UMI on Legs Appendix · p. 13
• Huy Ha, Yihuai Gao, Zipeng Fu, and Shuran Song are listed with Columbia University affiliations.Source: UMI on Legs Appendix · p. 13
• Jie Tan is listed with a Google DeepMind affiliation.Source: UMI on Legs Appendix · p. 13

7 Things that did not work

• The section presents these as unsuccessful attempts intended to help the community learn from project mistakes.Source: §7 · p. 13

7.1 Privileged policy distillation and observation history.

• Privileged-policy distillation produced no performance boost, and observation histories longer than one step introduced instability, possibly because imprecise ROS2 timer timestamps made histories out of distribution.The authors report this was less of an issue on the Unitree A1, which has a real-time kernel.Source: §7.1 · p. 13

7.2 Precise grasping for tossing.

• Small controller shakes caused distributional drift during grasping for tossing; the authors hypothesize that more than 500 demonstration episodes could improve autonomous grasping and tossing.Source: §7.2 · p. 13

7.3 System Reliability for Fully-untethered Deployment.

• The untethered system’s reliability was limited by dog-joint overheating after 10 to 30 minutes and battery voltage variation affecting the arm and policy behavior.The authors frequently cooled the motors and fine-tuned the policy toward a more energy-efficient posture; a voltage adaptor was needed when the battery was fully charged.Source: §7.3 · p. 13

7.4 Velocity integration

• iPhone pose updates arrived roughly 140 ms late, causing low-frequency oscillations that slowly diverged; velocity-based pose integration did not address the shaking.Higher action-rate regularization could mitigate the behavior, and the authors expected a better implementation to reduce latency.Source: §7.4 · pp. 13–14

8.1 iPhone placement

• The iPhone was mounted facing backward at 60° to improve ARKit environmental tracking while keeping the robot arm out of view and out of reach of the phone.The authors report more robust tracking than with their original 90° mount.Source: §8.1 · p. 14

8.2 Robot URDF

• Reweighing the ARX5 arm components and recomputing mass, center of mass, and inertia in the URDF were crucial for simulation stability and real-world deployment.Domain randomization and adaptive policies were insufficient for a heavily mis-specified manufacturer model, especially during dynamic arm movements.Source: §8.2 · p. 14

8.3 Latency

• A 20 ms end-to-end joint latency worked best in the real-world system; mismatched simulator and robot latency produced high-frequency shaking.The authors swept simulated latency from 0 to 30 ms in 5 ms intervals; iPhone ARKit pose latency was 140 ms, versus 8 ms for motion capture.Source: §8.3 · p. 14
• Python ROS2 callback contention was reduced by optimizing or separating callbacks, bringing joint-observation updates close to 200 Hz and policy inference close to 50 Hz.The authors state that a C++ implementation with more precise timing and lower latency could improve performance with the same checkpoint and evaluation setup.Source: §8.3 · p. 14

8.4 Safety

• Real-world safety fine-tuning targeted shaking, calf-joint overheating, and leg-twisting poses by increasing action-rate and torque regularization and favoring balanced body poses.The controller could run up to 30 minutes continuously after increased torque regularization; disabling onboard lidar also reduced shaking.Source: §8.4 · p. 15

9.1 Manipulation Policy

• Manipulation-policy settings define observation and proprioception history, output and execution steps, execution frequency, and trajectory-update smoothing.Linear interpolation smooths transitions between executed and newly updated trajectories.Source: §9.1 · p. 15

9.2 Whole-Body Dynamic Tossing

• The controller uses task-frame trajectory tracking, with task-specific policy settings and smoothing adjustments for dynamic tossing, pushing, and cup rearrangement.For tossing, inference and execution steps increase and execution frequency rises; for pushing, fewer demonstrations are used and trajectory smoothing increases. Cup rearrangement uses a released checkpoint without extra data.Source: §9.2 · p. 15, §9.1 · p. 15, §9.3 · p. 16

9.3 End-effector Reaching Leads to Robust Whole-body Pushing

• For pushing, simpler trajectories required fewer human demonstrations, while longer trajectory-update smoothing was used to prevent heavy shakes.Detailed settings are listed in Table 3.Source: §9.3 · p. 16, Table 3 · p. 16

9.4 Plug-and-play Cross-Embodiment Manipulation Policies

• The cup-rearrangement task directly uses the publicly released Chi et al. checkpoint, with no additional data collection.The task uses a lower execution frequency for stability because it requires higher precision but is quasi-static.Source: §9.4 · p. 16, Table 4 · p. 16

9.5 In-the-wild Cup Rearrangement Evaluation

• In-the-wild cup rearrangement tests include unstable terrain, direct sunlight, and unseen tables and cups.The stated scenarios are intended to assess generalizability of the diffusion policy.Source: §9.5 · p. 16

9.6 Reward Terms

• Controller rewards combine arm-body alignment, even foot-force distribution, feet-under-hips stance, and pose reaching with a precision-driven sigma curriculum.The curriculum decreases sigma as position and orientation errors shrink, encouraging more precise reaching.Source: §9.6 · pp. 16–17, Table 5 · p. 17

10.1 Real World Tossing

• Tossing success requires the ball to land within 40 cm of the bin center, and episodes where the robot falls during the toss count as failures.The policy is handed the tennis ball to provide a pre-grasped state.Source: §10.1 · p. 17

10.2 Simulation Ablations

• Simulation ablations sample grasp-and-toss trajectories and perturb initial robot position, orientation, and joint positions; evaluation averages position and orientation errors and measures 17-second survival.Survival requires lasting 17 seconds without terminal collision, defined as contact by any robot part other than the feet or gripper.Source: §10.2 · p. 17
• Simulation evaluation uses 10 ms pose latency and the reported domain-randomization range; results use checkpoint 4000 after 4000 training iterations.Power usage is estimated from hardware voltages, manufacturer torque constants, and simulated motor torques.Source: §10.2 · pp. 17–18, Table 6 · p. 17

11 Misc.

• The complete robot system costs roughly 1/4 as much as the quadruped manipulation systems in [10] [12].Table 7 reports the system cost.Source: §11 · p. 18, Table 7 · p. 18