Source-linked AI summary
Hand-Eye Autonomous Delivery: Learning Humanoid Navigation, Locomotion and Reaching
Sirui Chen, Yufei Ye, Zi-Ang Cao, Jennifer Lew, Pei Xu, C. Karen Liu
TL;DR
Humanoid navigation and reaching require coordinating articulated whole-body motion, while common visual navigation methods abstract robots as 2D point masses. HEAD combines high-level hand-eye targets with a low-level whole-body controller, drawing on human motion and vision data. The system reports 71% success reaching objects across two obstacle-filled environments.
Problem
Many visual navigation methods abstract robots as 2D point masses, whereas humanoids must coordinate articulated movement, reaching, and obstacle avoidance in 3D.
Source: §1 · p. 2Method
Results
71% success was achieved in reaching different objects across two environments with obstacles.
Source: §5 · p. 8Takeaways & Limitations
Component evaluations found that human and robot data together support reliable navigation, while large-scale human data improves generalization to novel environments.
Source: §1 · p. 2The method uses human head poses without coordinating other body parts, which can make the robot conservative in complex environments and leave ambiguity in lower-body control.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We propose Hand-Eye Autonomous Delivery (HEAD), a framework that learns navigation, locomotion, and reaching skills for humanoids, directly from human motion and vision perception data. We take a modular approach where the high-level planner commands the target position and orientation of the hands and eyes of the humanoid, delivered by the low-level policy that controls the whole-body movements. Specifically, the low-level whole-body controller learns to track the three points (eyes, left hand, and right hand) from existing large-scale human motion capture data while high-level policy learns from human data collected by Aria glasses. Our modular approach decouples the ego-centric vision perception from physical actions, promoting efficient learning and scalability to novel scenes. We evaluate our method both in simulation and in the real-world, demonstrating humanoid's capabilities to navigate and reach in complex environments designed for humans.
1 Introduction
HEAD coordinates humanoid navigation, locomotion, and reaching through a high-level policy that sets eye-and-hand targets and a low-level controller that executes whole-body motion. It combines distinct human data sources to address 3D navigation and reaching, and is evaluated in simulation and on a real humanoid.
1 Introduction
- HEAD commands eye and hand targets through a high-level policy while a low-level controller executes corresponding full-body movements.This modular design separates egocentric perception from physical actions and allows different human data sources and algorithms to train the skills.Source: §1 · p. 2
- Humanoid navigation must coordinate an articulated body through 3D spaces while reaching for and avoiding objects at varying heights, unlike point-mass methods limited to 2D movement.HEAD combines large-scale internet human exploration data, target-environment demonstrations, and a small amount of robot experience to address scene and embodiment domain shifts.Source: §1 · p. 2
- The low-level controller tracks the eyes and both hands from large-scale motion-capture data, using imitation-based reinforcement learning to handle varied target configurations.The authors identify sparse targets, the need for many demonstrations, and limited real-world root-state measurements as controller-training challenges.Source: §1 · p. 2
- 71% success was achieved in real-world reaching across different objects in two obstacle-filled environments.The paper also reports robust navigation and reaching on a Unitree G1 in a human indoor space.Source: §1 · p. 2
2 Related Works
Related work covers human-data sources for robot learning, humanoid whole-body control, and visual navigation. HEAD combines these threads around three-point tracking for joint navigation, reaching, and locomotion.
Learning from Human Data
- Human training data ranges from internet-scale videos useful for visual pretraining to task-relevant demonstrations for manipulation and indoor navigation.VR, AR glasses, and SLAM cameras can capture modalities such as head and hand poses, but different tasks may require different forms of human data.Source: §2 · pp. 2–3
Humanoid whole body control
- Whole-body controllers have used detailed kinematic trajectories, full-body joint positions, or sparse head-and-hand tracking; HEAD also tracks head and wrist orientations for manipulation such as wrist twisting.The paper places its interface alongside prior sparse-tracking approaches [20] [19].Source: §2 · p. 3
Navigation
3 Method
HEAD uses high-level navigation and reaching modules to command a shared low-level whole-body policy tracking head-and-hand poses. Its components use different human and robot data sources and training approaches.
3 Method
- HEAD sends high-level hand-eye targets to a shared low-level whole-body policy for both navigation and reaching.The high-level policy updates targets at a lower frequency than the low-level controller tracks them.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Whole-body Controller
- The whole-body controller uses end-to-end GAN-like motion imitation under reinforcement learning alongside goal-directed tracking of target positions and orientations.It controls the humanoid through PD servos rather than relying on a two-stage distillation procedure.Source: §3.1 · p. 3
- The controller's five-hour motion dataset retargets AMASS and OMOMO capture data to the G1, while its deployable observations omit world-frame root position and velocity.The observation space uses local link poses over two consecutive steps and local joint velocity.Source: §3.1 · pp. 3–4
- Separate upper- and lower-body discriminators support combinations of poses, while relative target transformations and a multi-objective reward combine imitation with sparse target tracking.The GAN-like approach can imitate arbitrary motion-dataset segments without a precomputed full imitation trajectory.Source: §3.1 · pp. 4–5
- Dynamics parameters and sensor noise are randomized during training, alongside a regularization term, to aid sim-to-real transfer and robustness.These additions accompany the target-tracking task reward.Source: §3.1 · p. 5
3.2 Navigation Module
- The navigation module uses DINO image features and a tracked 2D goal to predict a future eye trajectory, while Aria glasses provide goal-conditioned human images, trajectories, and goals.The model is trained with both human and robot data; human recordings are augmented to resemble robot views, and large-scale ADT data addresses unseen scenes.Source: §3.2 · p. 5
3.3 Reaching Module
- Navigation hands control to reaching when the target is visible in the downward-facing RGB-D camera and within reach; correspondence matching transfers the goal into that camera frame.The low-level policy continues running during the handoff for a smooth transition.Source: §3.3 · p. 5
- For final reaching, inverse kinematics infers missing head poses and hand orientations from the 3D hand-position goal, initialized from the current robot state.The optimization also encourages small changes in center-of-mass position and pelvis orientation.Source: §3.3 · p. 6
4 Experiment
The experiments evaluate HEAD in a training lab and deployment kitchen, testing integrated reaching, whole-body tracking, and navigation-data choices.
4.1 Hardware Setup
- Tests use furnished lab and kitchen rooms, with robot-specific training data only in the lab and unseen test layouts and objects.The kitchen emulates deployment without robot ground-truth training data.Source: §4.1 · p. 6
4.2 Whole Body Reaching with Different Scenes
- 71% overall success across the evaluated environments, with lab-room success 25% greater than kitchen success.The evaluation covers three layouts in each room; Table 1 reports successes and failure types.Source: §4.2 · p. 6, Table 1 · p. 7
4.3 Performance of Whole Body controller
- Controller evaluations compare Isaac Gym training performance with MuJoCo simulation-to-simulation robustness, measuring positional and orientation errors and head-height failures.A timestep counts as a failure when head height deviates from target by ≥0.4 m.Source: §4.3 · p. 7, Table 2 · p. 7
- Separate upper- and lower-body discriminators outperform a single whole-body discriminator on unseen tasks and support more diverse motions under three-point tracking.The authors suggest separate discriminators prevent irrelevant motion entanglement, such as arm-swing patterns interfering with carrying a box while walking.Source: §4.3 · p. 7
4.4 Performance of Navigation Module
- Navigation data include 200 human clips per room and robot trajectories collected in the lab and kitchen, with kitchen robot clips reserved for testing.The lab robot data are split into training and test clips.Source: §4.4 · p. 7
- Combining all available human and robot data gives the best lab-room navigation performance; in-domain human data help more than out-of-distribution data.Human-only in-domain training performs poorly relative to robot-only in-domain training.Source: §4.4 · p. 8
- In the deployment kitchen, added in-domain human data help, while ADT data substantially boost success where no robot training data are available.ADT does not significantly improve lab-room performance.Source: §4.4 · p. 8
- Sharing one decoder branch for human and robot data improves navigation scene generalization, unlike common practice in manipulation tasks [6, 35].The authors hypothesize that human–robot embodiment differences are smaller in short-term navigation than in manipulation.Source: §4.4 · p. 8
5 Conclusion
HEAD is an autonomous hand-eye delivery system for humanoid navigation and reaching. It achieves a 71% success rate across objects and two environments, while broader grasping and body-aware navigation remain future directions.
5 Conclusion
- HEAD is an autonomous hand-eye delivery system for humanoid navigation and reaching.The system achieves a 71% success rate reaching different objects placed in two environments.Source: §5 · p. 8
- The system reaches different objects across two environments with a 71% success rate.Source: §5 · p. 8
- Future work includes general grasping across objects and locations, and finer-grained whole-body navigation that avoids collisions with different body parts.Source: §5 · p. 8
6 Limitations
The method uses human head poses as the robot control interface without modeling other body parts. This constrains how it handles complex obstacles and can leave lower-body intent ambiguous.
6 Limitations
- The control interface collects human head poses but does not account for other body parts.Source: §6 · p. 9
- For heavily occluded targets or narrow gaps, lacking whole-body human coordination can make the robot take a more conservative approach and underuse its agility.Source: §6 · p. 9
- Head and wrist poses can leave lower-body intent ambiguous, potentially making the robot hesitate between bending and walking forward.The authors suggest foot-pose information could help reduce hesitation.Source: §6 · p. 9
A Whole Body Controller
The controller combines partial-motion imitation with goal-directed tracking. Its policy receives humanoid state and tracking goals, while separate imitation objectives and a goal objective are balanced during training.
A Whole Body Controller
- The controller splits full-body motions into arm and torso groups and uses two discriminators to assess partial-motion imitation.This lets arm motions combine with torso and lower-body motions from different clips.Source: A Whole Body Controller · p. 13
- The training objective balances two imitation objectives against goal-directed tracking, using standardized advantages and objective weights.Each imitation weight is 0.2 and the goal-directed weight is 0.6.Source: A Whole Body Controller · pp. 13–14
- Policy training uses PPO and Adam with 1,363 selected locomotion and locomotion-manipulation clips from AMASS and OMOMO.The motion dataset totals about five hours.Source: A Whole Body Controller · p. 14
A.1 Observation Space
The observation combines two historical humanoid-state frames, local root and joint velocities, and target poses for the head and hands. A GRU processes the state vector, while a value network has three outputs for two imitation objectives and one goal-directed objective.
A.2 Observation Space
- The state vector covers 33 links over two frames, with each link's position and quaternion orientation expressed relative to the root.A GRU processes the state vector.Source: §A.1 · p. 14
- The value network has three outputs for two imitation objectives and one goal-directed objective.Source: Figure S2 · p. 14
- The goal vector specifies target positions and quaternion orientations for the left hand, right hand, and head in the next three frames.Source: §A.1 · p. 14
- The observation concatenates the humanoid state, local control state, and goals for the head and two hands.The state contains two historical frames; the control state contains root angular velocity and joint velocity.Source: §A.1 · p. 14
A.2 Reward Terms
The controller balances two imitation objectives with a goal-directed reward. Its reward terms combine target tracking with stepping, foot-contact, and joint-energy regularization.
A.2 Reward Terms
- The policy uses an ensemble of 32 discriminators for each motion group to compute imitation-related rewards.The discriminators use hinge loss with gradient penalty.Source: §A.2 · p. 14
- The goal-directed reward combines tracking, in-air, sliding, and energy terms as r = r_tracking + 0.8r_in-air + 0.5r_sliding + 0.005r_energy.The passage states that tracking is the main component and the other terms stabilize motions for sim-to-real consideration.Source: §A.2 · p. 14
- Tracking reward measures position and orientation errors for the left hand, right hand, and head.Position error is Euclidean distance; orientation error is the angle between current and target orientations.Source: §A.2 · p. 15
- The in-air term encourages the swing foot to remain airborne for at least 0.5 seconds during stepping.It is computed when the swing foot contacts the ground, using its preceding hanging time.Source: §A.2 · p. 15
- The sliding term penalizes a foot's linear velocity when that foot contacts the ground.Source: §A.2 · p. 15
- The energy term penalizes each joint's energy cost using its applied torque and rotation velocity.Source: §A.2 · p. 15
A.3 Domain Randomization
For sim-to-real transfer, the authors found simulated action delay during training essential to prevent robot jitter. They use a constant one-step delay in a fixed portion of training environments.
A.3 Domain Randomization
- Simulated delay during training was essential to prevent the robot from jittering in sim-to-real tests.Source: §A.3 · p. 15
- A constant one-step delay on a fixed portion of environments improved training speed compared with random delay across all environments.Source: §A.3 · p. 15
- The selected delay fraction was ρ_delay = 0.5, applying action delay in half of the training environments.Source: §A.3 · p. 15
B Image Augmentation of Navigation Data
Navigation images from Aria Glasses are adjusted to approximate the robot camera's view by correcting lens distortion and camera pitch. In real-world experiments, head-mounted motion-capture markers support world-frame camera tracking to improve reaching accuracy.
B Image Augmentation of Navigation Data
- Aria fisheye images are undistorted to approximate a pinhole view, then homography aligns the human camera pitch with the robot camera pitch.Aria's camera has 110° horizontal and vertical fields of view, versus the robot camera's 90° horizontal and 67.5° vertical fields of view.Source: B Image Augmentation of Navigation Data · p. 15
C Usage of motion capture system
- Head-mounted motion-capture markers let the authors transform the camera trajectory into the world frame, improving reaching accuracy through closed-loop tracking.The authors report difficulty keeping the robot at an exact location without drifting.Source: C Usage of motion capture system · p. 16