Source-linked AI summary
HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations
Xiaomeng Xu, Jisang Park, Han Zhang, Eric Cousineau, Aditya Bhat, Jose Barreiros, Dian Wang, Jeannette Bohg, Shuran Song
TL;DR
Robot-free human demonstrations offer scalable data for mobile manipulation, but egocentric sensing increases the human-to-robot embodiment gap. HoMMI addresses this with cross-embodiment visual and head-action representations plus constraint-aware whole-body control, enabling long-horizon bimanual manipulation, navigation, and active perception.
Problem
Egocentric sensing supplies global context for mobile manipulation but introduces visual and kinematic embodiment gaps that make policy transfer difficult.
Method
HoMMI combines robot-free egocentric data collection, embodiment-agnostic 3D vision, a relaxed 3D look-at point, and constraint-aware whole-body control.
Results
HoMMI enables long-horizon whole-body mobile manipulation involving bimanual coordination, navigation, and active perception; naive egocentric RGB and head-motion regression achieved 0% success on two tasks.
Takeaways & Limitations
The framework supports versatile mobile manipulation directly from robot-free human demonstrations while coordinating end-effector tracking and active perception under physical constraints.
Takeaways & Limitations
Short observation history can limit recovery in long-horizon tasks and partial observability, while vision-only sensing and remaining hardware differences constrain contact-rich manipulation and transferability.
Abstract
from arXiv · showhide
We present Whole-Body Mobile Manipulation Interface (HoMMI), a data collection and policy learning framework that learns whole-body mobile manipulation directly from robot-free human demonstrations. We augment UMI interfaces with egocentric sensing to capture the global context required for mobile manipulation, enabling portable, robot-free, and scalable data collection. However, naively incorporating egocentric sensing introduces a larger human-to-robot embodiment gap in both observation and action spaces, making policy transfer difficult. We explicitly bridge this gap with a cross-embodiment hand-eye policy design, including an embodiment agnostic visual representation; a relaxed head action representation; and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under robot-specific physical constraints. Together, these enable long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination, navigation, and active perception. Results are best viewed on: https://hommi-robot.github.io
I. INTRODUCTION
HoMMI addresses the limited global context and embodiment mismatch that make robot-free learning of mobile manipulation difficult. It combines egocentric data collection, cross-embodiment representations, relaxed head actions, and constraint-aware whole-body control for complex tasks.
- Wrist-centric demonstrations provide local end-effector views but under-observe global context needed for navigation, bimanual coordination, and task progress tracking.
- Adding a head-mounted camera captures broader workspace context, hand relationships, and human active-perception behavior, but increases visual and kinematic embodiment gaps.
- HoMMI extends UMI with a head-mounted camera and ARKit-based synchronized multi-view video and 6-DoF pose capture in a globally consistent frame.
- Its 3D egocentric representation uses embodiment-agnostic end-effector frames and removes demonstrator-specific arms and body observations to mitigate appearance and viewpoint mismatch.
- A 3D look-at point transfers active perception across robots with different heights and joint constraints without sacrificing end-effector tracking accuracy.
- The constraint-aware whole-body controller coordinates robot motion for precise tracking and stable, safe behavior, enabling long-horizon tasks involving search, manipulation, and navigation.
B. Robot Learning from Egocentric Demonstrations
Prior egocentric demonstration methods face embodiment and coordination challenges, while teleoperation limits scalability. HoMMI combines robot-free sensing, cross-embodiment policy components, and whole-body execution for mobile manipulation.
- Egocentric demonstrations support bimanual manipulation and active perception, but prior approaches still require robot teleoperation data or assume specialized necks and restricted settings.
- Bimanual mobile manipulation must synchronize two-arm coordination, base positioning, and active perception, while robot-teleoperation dependence limits data scalability.
- HoMMI targets scalable collection, transferability across embodiment gaps, and whole-body coordination for precise manipulation and active perception.
- The system combines an intuitive interface, cross-embodiment hand-eye policy, and constraint-aware whole-body controller to execute policy outputs under robot constraints.
- Three iPhones capture synchronized RGB, depth, 6-DoF poses, and gripper widths across grippers and a cap within a shared coordinate frame.
- The 3D egocentric representation uses an embodiment-agnostic gripper frame and masks embodiment-specific body observations.
V. CROSS-EMBODIMENT HAND-EYE POLICY
HoMMI bridges visual and kinematic embodiment gaps with geometry-aware egocentric observations and a relaxed head-action representation. The policy preserves active-perception intent while producing robot-feasible head behavior.
- Naively adding head RGB and directly predicting human head pose enlarges the embodiment gap and can cause deployment failures.
- HoMMI lifts head observations into 3D geometry-aware tokens instead of directly feeding head RGB, addressing viewpoint and appearance differences.
- B. 3D Look-at Point Action Representation to Mitigate the Kinematic Gap: A 3D look-at point replaces direct 6-DoF head-pose imitation, preserving active-perception intent while respecting robot kinematic constraints.
- B. 3D Look-at Point Action Representation to Mitigate the Kinematic Gap: During training, the look-at point is computed from the center camera ray and scene pointmap; at inference, it is converted into a feasible head orientation.
- B. 3D Look-at Point Action Representation to Mitigate the Kinematic Gap: The desired viewing direction is the normalized vector from the current head position to the predicted look-at point.
C. Gripper-Centric Frame for Spatial Awareness
HoMMI expresses observations and actions in a gripper-centric frame and uses constraint-aware whole-body inverse kinematics to realize end-effector trajectories. The controller balances tracking accuracy with smoothness, stability, and human-like motion.
- C. Gripper-Centric Frame for Spatial Awareness: The gripper-centric frame transforms gripper poses, head pointmaps, and look-at points into the left-gripper frame to keep policy inputs and outputs in-distribution.
- The mobile platform uses two 7-DoF arms, a 6-DoF torso, a holonomic base, and a 2-DoF neck with stereo egocentric cameras.
- The controller converts end-effector poses and head look-at points into whole-body joint and base motions while meeting accuracy, smoothness, stability, and human-likeness requirements.
- Differential whole-body IK prioritizes bimanual SE(3) tracking and combines interpolation, posture and velocity regularization, stability constraints, and human-like motion priors.
- The constrained quadratic program includes end-effector, nominal-posture, current-posture, and center-of-mass costs plus configuration, velocity, collision, and upright constraints.
- The look-at point is controlled separately from whole-body IK so neck orientation does not interfere with bimanual end-effector tracking.
C. Asynchronous Policy Inference
HoMMI separates policy inference from real-time execution so mobile manipulation can continue without pausing for inference. The evaluation compares HoMMI with wrist-only, RGB-only, head-only, and no-active-neck variants across four capabilities.
- Asynchronous Policy Inference: A detached policy server performs inference while a real-time execution bridge synchronizes observations, filters stale actions, and streams actions to the robot.This decoupling avoids base jerks and tracking errors caused by pausing mobile manipulation for inference.
- Asynchronous Policy Inference: The bridge latency-corrects camera streams, interpolates proprioception, and anchors synchronized observation windows to the latest camera timestamp.The resulting window follows the latency-matching approach used in UMI.
- Evaluation: The evaluation tests cross-embodiment transfer, bimanual and whole-body coordination, long-horizon navigation, and active perception.These capabilities cover transfer from robot-free human demonstrations to a robot with different appearance and kinematics.
- Baselines: HoMMI is compared with Wrist-Only, RGB-Only, Head-Only, and w/o Active Neck baselines or ablations.The comparisons isolate the contributions of wrist sensing, egocentric RGB, head observations, and active head motion.
A. Laundry Task
The Laundry task requires coordinated bimanual grasping, whole-body movement, navigation, and active perception to place a cloth into a side-located bin. Baseline failures show complementary weaknesses in global context, embodiment transfer, local contact sensing, and active viewing.
- Task: The robot grasps a cloth with both hands, searches for a bin, navigates, and places the cloth into it.The bin’s lower, lateral placement requires coordinated navigation, rotation, bending, and placement.
- Baseline Failures: Wrist-Only often turns to one side regardless of bin location because the bin is outside the wrist-camera view.Other failures include weak grasping and inaccurate placement, consistent with missing global spatial context.
- Baseline Failures: RGB-Only fails to grasp reliably and can press the table hard enough to trigger the robot’s wrench safety guard.The authors hypothesize that human-to-robot appearance and viewpoint mismatches push egocentric observations out of distribution.
- Baseline Failures: Head-Only achieves 0% success because it can miss the cloth or grasp only one edge, causing the cloth to slip.The comparison indicates that wrist cameras provide local contact information needed for grasping accuracy.
- Baseline Failures: Disabling active neck motion yields 75% success but commonly causes inaccurate cloth placement in the bin.The authors associate these failures with a less informative viewpoint and incomplete visibility of the bin.
C. Tablescape Task
The Tablescape task evaluates whether the system can coordinate both hands and the whole body to unfold a mat across varied starting conditions. HoMMI reaches 80% success and can retry when the mat is misaligned or folds back.
- Task: The robot grasps two mat edges, lifts and unfolds the mat while moving forward, lays it flat, and retracts its hands.Success requires coordinated bimanual rotation and height control together with base, torso, and arm motion.
- Performance: 80% success was achieved across 20 rollouts spanning five initial base positions and two mat configurations.The evaluation used 115 demonstrations with varied standing locations and mat placements.
- Performance: HoMMI recovers from imperfect alignment or a mat folding back by lifting the mat again and retrying.The remaining failures arise from slightly missing the grasp.
- Baseline Failures: Wrist-Only rotates too late and rises above the mat, while Head-Only misses contact, showing complementary limits of global and local sensing.RGB-Only presses too hard against the table, and w/o Neck achieves 55% success while often failing to recover after a folded mat.
- Sensing: Wrist cameras supply local contact cues while egocentric observations support task-relevant attention and active viewpoint adjustment.The findings summary links combined wrist and head sensing to cleaner attention around relevant objects and improved manipulation context.
I. POLICY TRAINING DETAILS
The policy predicts short-horizon action sequences from multimodal observations and uses temporal interpolation plus constrained whole-body inverse kinematics to produce smooth, physically feasible motion.
- Policy architecture: The policy uses two-step observation histories and predicts 32-step action horizons at 20 Hz from wrist and head sensing, pointmaps, and proprioception.Actions are 23-dimensional, combining two gripper poses, a look-at point, and gripper widths.
- Policy architecture: Diffusion Policy with a Diffusion Transformer predicts actions from global observation embeddings using a DDIM scheduler and finetuned DINOv3 visual backbones.The model is trained with 100 diffusion timesteps and 16 inference steps.
- Optimization: AdamW optimization trains the policy and baselines for 500 epochs, using cosine scheduling and separate learning rates for diffusion and vision-backbone parameters.The initial learning rates are 7.5×10^-5 and 7.5×10^-6, respectively.
- Whole-body control: Temporal interpolation converts policy pose commands into continuous targets for the high-frequency controller, using linear position interpolation and spherical-linear orientation interpolation.This design is intended to eliminate jitter in the resulting motion.
III. ROBOT HARDWARE SETUP
The hardware is a bimanual mobile manipulator designed for whole-body coordination, active perception, and high-bandwidth synchronized sensing and control.
- Robot platform: The platform combines a holonomic mecanum base, 6-DoF torso, two 7-DoF arms, and a 2-DoF neck carrying stereo wide-angle RGB cameras.Fin-ray fingers matching the UMI grippers replace the stock parallel grippers.
- Sensing and computing: The sensing and computing stack uses separate fiber-connected camera and robot-control subnetworks to support simultaneous multi-camera streaming and high-frequency closed-loop control.Cameras connect through a PoE GigE switch and a 10 Gbps multimode fiber uplink.
- Sensing and computing: The hardware schematic documents the integrated robot, sensing, and computing configuration used for mobile manipulation experiments.
A. Depth quality sensitivity
The policy remains robust to moderate depth noise and generalizes across unseen objects and lighting conditions, while stereo sensing provides clean pointclouds and task-relevant attention.
- Depth quality sensitivity: The stereo pointcloud is clean, and policy attention focuses on task-relevant regions such as the bin and cloth during inference.
- Depth quality sensitivity: Performance remains unchanged through 1 cm depth-noise standard deviation and degrades only under substantially larger perturbations.The evaluation uses three noise levels with 10 rollouts each.
- Generalization and robustness: 90.63%±12.10% success under unseen objects and 93.75%±10.83% under unseen lighting conditions indicate robustness of the policy.Each variation evaluation covers three cases with four rollouts per case.
- Tracking accuracy: Average tracking error against motion-capture ground truth is within 5.0 mm in position and 0.8° in rotation.