HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations

• Wrist-centric demonstrations provide local end-effector views but under-observe global context needed for navigation, bimanual coordination, and task progress tracking.Source: §I · pp. 1–2
• Adding a head-mounted camera captures broader workspace context, hand relationships, and human active-perception behavior, but increases visual and kinematic embodiment gaps.Source: §I · p. 2
• HoMMI extends UMI with a head-mounted camera and ARKit-based synchronized multi-view video and 6-DoF pose capture in a globally consistent frame.Source: §I · p. 2
• Its 3D egocentric representation uses embodiment-agnostic end-effector frames and removes demonstrator-specific arms and body observations to mitigate appearance and viewpoint mismatch.Source: §I · p. 2
• A 3D look-at point transfers active perception across robots with different heights and joint constraints without sacrificing end-effector tracking accuracy.Source: §I · p. 2
• The constraint-aware whole-body controller coordinates robot motion for precise tracking and stable, safe behavior, enabling long-horizon tasks involving search, manipulation, and navigation.Source: §I · p. 2

• Egocentric demonstrations support bimanual manipulation and active perception, but prior approaches still require robot teleoperation data or assume specialized necks and restricted settings.Source: B. Robot Learning from Egocentric… · p. 2, C. Learning Mobile Manipulation From… · p. 2
• Bimanual mobile manipulation must synchronize two-arm coordination, base positioning, and active perception, while robot-teleoperation dependence limits data scalability.Source: C. Learning Mobile Manipulation From… · pp. 2–3
• HoMMI targets scalable collection, transferability across embodiment gaps, and whole-body coordination for precise manipulation and active perception.Source: §III · p. 3
• The system combines an intuitive interface, cross-embodiment hand-eye policy, and constraint-aware whole-body controller to execute policy outputs under robot constraints.Source: §III · p. 3, Figure 2 · p. 3
• Three iPhones capture synchronized RGB, depth, 6-DoF poses, and gripper widths across grippers and a cap within a shared coordinate frame.Source: §IV · p. 3
• The 3D egocentric representation uses an embodiment-agnostic gripper frame and masks embodiment-specific body observations.Source: Figure 3 · p. 3

• Naively adding head RGB and directly predicting human head pose enlarges the embodiment gap and can cause deployment failures.Source: §V · p. 3
• HoMMI lifts head observations into 3D geometry-aware tokens instead of directly feeding head RGB, addressing viewpoint and appearance differences.Source: A. 3D Visual Representation to… · pp. 3–4

B. 3D Look-at Point Action Representation to Mitigate the Kinematic Gap

• A 3D look-at point replaces direct 6-DoF head-pose imitation, preserving active-perception intent while respecting robot kinematic constraints.Source: B. 3D Look-at Point Action… · p. 4, Figure 4 · p. 4
• During training, the look-at point is computed from the center camera ray and scene pointmap; at inference, it is converted into a feasible head orientation.Source: B. 3D Look-at Point Action… · p. 4
• The desired viewing direction is the normalized vector from the current head position to the predicted look-at point.Source: B. 3D Look-at Point Action… · p. 4

C. Gripper-Centric Frame for Spatial Awareness

• The gripper-centric frame transforms gripper poses, head pointmaps, and look-at points into the left-gripper frame to keep policy inputs and outputs in-distribution.Source: C. Gripper-Centric Frame for Spatial… · p. 4

• The mobile platform uses two 7-DoF arms, a 6-DoF torso, a holonomic base, and a 2-DoF neck with stereo egocentric cameras.Source: A. Bimanual Mobile Manipulator… · p. 4
• The controller converts end-effector poses and head look-at points into whole-body joint and base motions while meeting accuracy, smoothness, stability, and human-likeness requirements.Source: B. Constraint-Aware Whole-body… · pp. 4–5
• Differential whole-body IK prioritizes bimanual SE(3) tracking and combines interpolation, posture and velocity regularization, stability constraints, and human-like motion priors.Source: B. Constraint-Aware Whole-body… · p. 5
• The constrained quadratic program includes end-effector, nominal-posture, current-posture, and center-of-mass costs plus configuration, velocity, collision, and upright constraints.Source: B. Constraint-Aware Whole-body… · p. 5
• The look-at point is controlled separately from whole-body IK so neck orientation does not interfere with bimanual end-effector tracking.Source: B. Constraint-Aware Whole-body… · p. 5

Asynchronous Policy Inference

• A detached policy server performs inference while a real-time execution bridge synchronizes observations, filters stale actions, and streams actions to the robot.This decoupling avoids base jerks and tracking errors caused by pausing mobile manipulation for inference.Source: C. Asynchronous Policy Inference · p. 5
• The bridge latency-corrects camera streams, interpolates proprioception, and anchors synchronized observation windows to the latest camera timestamp.The resulting window follows the latency-matching approach used in UMI.Source: C. Asynchronous Policy Inference · p. 5

Evaluation

• The evaluation tests cross-embodiment transfer, bimanual and whole-body coordination, long-horizon navigation, and active perception.These capabilities cover transfer from robot-free human demonstrations to a robot with different appearance and kinematics.Source: §VII · p. 6

Baselines

• HoMMI is compared with Wrist-Only, RGB-Only, Head-Only, and w/o Active Neck baselines or ablations.The comparisons isolate the contributions of wrist sensing, egocentric RGB, head observations, and active head motion.Source: §VII · p. 6

Task

• The robot grasps a cloth with both hands, searches for a bin, navigates, and places the cloth into it.The bin’s lower, lateral placement requires coordinated navigation, rotation, bending, and placement.Source: A. Laundry Task · p. 6

Baseline Failures

• Wrist-Only often turns to one side regardless of bin location because the bin is outside the wrist-camera view.Other failures include weak grasping and inaccurate placement, consistent with missing global spatial context.Source: A. Laundry Task · p. 6
• RGB-Only fails to grasp reliably and can press the table hard enough to trigger the robot’s wrench safety guard.The authors hypothesize that human-to-robot appearance and viewpoint mismatches push egocentric observations out of distribution.Source: A. Laundry Task · p. 7
• Head-Only achieves 0% success because it can miss the cloth or grasp only one edge, causing the cloth to slip.The comparison indicates that wrist cameras provide local contact information needed for grasping accuracy.Source: A. Laundry Task · p. 7
• Disabling active neck motion yields 75% success but commonly causes inaccurate cloth placement in the bin.The authors associate these failures with a less informative viewpoint and incomplete visibility of the bin.Source: A. Laundry Task · p. 7

Task

• The robot grasps two mat edges, lifts and unfolds the mat while moving forward, lays it flat, and retracts its hands.Success requires coordinated bimanual rotation and height control together with base, torso, and arm motion.Source: C. Tablescape Task · p. 7

Performance

• 80% success was achieved across 20 rollouts spanning five initial base positions and two mat configurations.The evaluation used 115 demonstrations with varied standing locations and mat placements.Source: C. Tablescape Task · p. 8
• HoMMI recovers from imperfect alignment or a mat folding back by lifting the mat again and retrying.The remaining failures arise from slightly missing the grasp.Source: C. Tablescape Task · p. 8

Baseline Failures

• Wrist-Only rotates too late and rises above the mat, while Head-Only misses contact, showing complementary limits of global and local sensing.RGB-Only presses too hard against the table, and w/o Neck achieves 55% success while often failing to recover after a folded mat.Source: C. Tablescape Task · p. 8

Sensing

• Wrist cameras supply local contact cues while egocentric observations support task-relevant attention and active viewpoint adjustment.The findings summary links combined wrist and head sensing to cleaner attention around relevant objects and improved manipulation context.Source: D. Findings Summary · p. 8, Figure 11 · p. 8

Policy architecture

• The policy uses two-step observation histories and predicts 32-step action horizons at 20 Hz from wrist and head sensing, pointmaps, and proprioception.Actions are 23-dimensional, combining two gripper poses, a look-at point, and gripper widths.Source: §I · p. 12
• Diffusion Policy with a Diffusion Transformer predicts actions from global observation embeddings using a DDIM scheduler and finetuned DINOv3 visual backbones.The model is trained with 100 diffusion timesteps and 16 inference steps.Source: §I · p. 12

Optimization

• AdamW optimization trains the policy and baselines for 500 epochs, using cosine scheduling and separate learning rates for diffusion and vision-backbone parameters.The initial learning rates are 7.5×10^-5 and 7.5×10^-6, respectively.Source: §I · p. 12

Whole-body control

• Temporal interpolation converts policy pose commands into continuous targets for the high-frequency controller, using linear position interpolation and spherical-linear orientation interpolation.This design is intended to eliminate jitter in the resulting motion.Source: A. Temporal Command Interpolation · p. 12

Robot platform

• The platform combines a holonomic mecanum base, 6-DoF torso, two 7-DoF arms, and a 2-DoF neck carrying stereo wide-angle RGB cameras.Fin-ray fingers matching the UMI grippers replace the stock parallel grippers.Source: §III · p. 13

Sensing and computing

• The sensing and computing stack uses separate fiber-connected camera and robot-control subnetworks to support simultaneous multi-camera streaming and high-frequency closed-loop control.Cameras connect through a PoE GigE switch and a 10 Gbps multimode fiber uplink.Source: §III · p. 13
• The hardware schematic documents the integrated robot, sensing, and computing configuration used for mobile manipulation experiments.Source: Figure 12 · p. 13

Depth quality sensitivity

• The stereo pointcloud is clean, and policy attention focuses on task-relevant regions such as the bin and cloth during inference.Source: A. Depth quality sensitivity · p. 13, Figure 13 · p. 13
• Performance remains unchanged through 1 cm depth-noise standard deviation and degrades only under substantially larger perturbations.The evaluation uses three noise levels with 10 rollouts each.Source: A. Depth quality sensitivity · p. 13, Table II · p. 13

Generalization and robustness

• 90.63%±12.10% success under unseen objects and 93.75%±10.83% under unseen lighting conditions indicate robustness of the policy.Each variation evaluation covers three cases with four rollouts per case.Source: B. Robustness to unseen lightings and… · p. 13, Table III · p. 13, Table IV · p. 13

Tracking accuracy

• Average tracking error against motion-capture ground truth is within 5.0 mm in position and 0.8° in rotation.Source: C. iPhone-ARKit tracking error · p. 14