Source-linked AI summary
HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping
Runpei Dong, Ziyan Li, Arjun Gupta, Xialin He, Saurabh Gupta
TL;DR
Visual whole-body grasping needs both generalizable scene understanding and accurate end-effector control, but existing monolithic learning approaches are difficult to scale. The paper combines large vision models for modular planning with simulation-trained HERO control, achieving 2.44cm tracking error and strong real-world grasping performance. Its scope is bounded by failure modes such as slipping or knocking over unstable objects.
Problem
Visual loco-manipulation of arbitrary in-the-wild objects requires generalizable RGB-D scene understanding and precise end-effector control, which existing end-to-end approaches jointly learn with limited scalability.
Method
The system separates open-vocabulary visual planning from simulation-trained whole-body control, using HERO with residual targets, learned forward models, replanning, and goal adjustment.
Results
2.44cm real-world translation error is achieved, while the modular system reaches 90% success on 10 daily objects, 73.3% across 10 daily scenes, and 80% in cluttered scenes.
Takeaways & Limitations
Modular perception and task-space control support open-vocabulary humanoid grasping in novel everyday environments without large-scale real-world imitation.
Takeaways & Limitations
Reported failures include slipping on large or irregular objects and knocking over unstable objects when grasp orientation or hand size is unsuitable.
Abstract
from arXiv · showhide
Visual loco-manipulation of arbitrary in-the-wild objects requires accurate end-effector (EE) control and a generalizable understanding of the scene from visual inputs (eg, RGB-D images). Existing imitation and sim2real methods jointly learn both these aspects via monolithic end-to-end learning and are thus hard to scale. In this work, we bring to bear the best tools for each of these problems -- large vision models for generalizable scene understanding and simulated training for accurate EE control -- leading to an overall modular loco-manipulation system that exhibits strong generalization. Our core technical innovation is HERO, an accurate residual-aware EE tracking policy made possible by combining classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, and c) goal adjustment and replanning. Together, these innovations reduce the end-effector tracking error to 2.44cm, outperforming the strongest prior method by 5.5x. Our overall system operates in diverse real-world environments, from offices to coffee shops, where the robot reliably grasps various everyday objects (eg, mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests demonstrate the effectiveness of our proposed design. We believe our advances open up new ways of training humanoids to interact with daily objects.
1 Introduction
The paper proposes a modular humanoid system that separates visual action planning from accurate whole-body end-effector control. HERO addresses humanoid tracking limitations through residual-aware planning, learned pose estimation, replanning, and goal adjustment, enabling open-vocabulary grasping.
- Motivation: Novel-object grasping requires RGB-D scene understanding, precise end-effector control, and whole-body balance in unseen environments.Unlike backflips, manipulation requires the hand to reach a precise goal location while operating with onboard sensing.
- Motivation: End-to-end imitation and visual reinforcement learning are limited by difficult real-world data collection and unrealistic diverse simulation scenes.The paper therefore separates high-level visual planning from simulation-trained low-level control.
- HERO: 13.38cm real-world tracking error from SONIC is reduced 5.5× to 2.44cm by HERO.The paper identifies accurate humanoid hand control as the key technical barrier to modular object manipulation.
- HERO: HERO combines motion-planned reference trajectories, residual end-effector targets, learned forward models, closed-loop replanning, and goal adjustment.Neural upper-body and base-odometry models improve pose estimates, while iterative goal adjustment addresses the systematic sim2real gap.
- System: The modular system uses large vision models and AnyGrasp to identify, segment, and grasp queried objects before HERO executes the retargeted Dex-3 hand pose.This decomposition assigns visual generalization to pretrained models and control accuracy to the tracker.
- System: 90% success is achieved on 10 daily objects, with 73.3% success across 10 daily scenes and 80% in cluttered scenes.These results come from real-world tests on novel object queries and environments.
2 Related Works
Prior visual loco-manipulation systems rely heavily on teleoperation, demonstrations, generated assets, or restricted object categories. HERO instead combines open-vocabulary perception with modular task-space control and learned residual corrections for novel queries and objects.
- Loco-manipulation via Motion Tracking: Humanoid loco-manipulation commonly uses teleoperation or generated references with reinforcement-learning policies trained to track whole-body motions.Related work targets tracking accuracy, agility, robustness, reachability, and generalization.
- Visual Loco-Manipulation: Visual loco-manipulation methods use teleoperated imitation data, depth-based reference-state policies, end-to-end visual reinforcement learning, or foundation-model policies.These approaches map visual observations to commands or generated motions at different levels of abstraction.
- Visual Loco-Manipulation: Existing visual systems remain limited by collected demonstrations, generated assets, or object categories.These constraints restrict the breadth of supported training or target objects.
- HERO: HERO pairs open-vocabulary perception with modular task-space control, enabling novel language queries and objects without task-specific teleoperation demonstrations.Its learned residual forward-kinematics and base-odometry models correct task-space signals needed by the tracking policy.
3 HERO: Humanoid End-Effector ContROl
HERO combines inverse kinematics, motion planning, learned residual-aware feedback, and closed-loop correction to control a humanoid end-effector accurately during whole-body reaching. Learned forward models estimate end-effector and base motion, while replanning and goal adjustment compensate drift and sim-to-real bias.
- Whole-body End-Effector Tracking Policy: HERO converts a desired end-effector pose into upper-body goals and planned reference trajectories, then outputs 29-DoF joint commands through a learned whole-body policy.The policy receives proprioception, reference base height and upper-body joints, locomotion commands, end-effector residual error, and recent history.
- Residual-Aware End-Effector Tracking: The policy uses residual end-effector pose error computed from a learned forward-kinematics model rather than relying only on joint-space references.The residual is defined between the estimated current end-effector pose and the target pose in the robot frame.
- Learned Residual Neural Forward Models: Residual neural forward kinematics corrects analytical FK to estimate accurate end-effector poses, while residual leg odometry estimates base displacement relative to the initial time step.Both residual models use separate translation and rotation predictions; the odometry model addresses inaccuracies in analytical base-pose estimation during balancing.
- Learned Residual Neural Forward Models: Whole-body balancing can make the original robot-frame target inaccurate and move the object outside the robot’s narrow egocentric view, limiting purely visual correction.The system therefore uses odometry-based correction to account for base shifts during reaching.
- Replanning and Goal Adjustment: The system replans the remaining trajectory every k = 300 steps, or 6 sec, using the current state and odometry-corrected goal, with planning taking approximately 20 ms.Goal adjustment also updates the commanded grasp translation with β = 0.6 when the translational residual is below 1.5cm; rotation is not scaled because it does not empirically improve tracking.
4 A Modular System for Open-Vocabulary Humanoid Object Grasping
The modular pipeline combines language-based object grounding, grasp proposal and filtering, hand retargeting, and HERO execution to grasp novel objects in novel environments using onboard sensing.
- The pipeline uses Grounding DINO to segment the queried object, AnyGrasp to propose parallel-jaw grasps, geometric filters to select table-parallel candidates, and HERO to execute the reach.
- Selected grasps are retargeted from AnyGrasp to the Dex3 hand before execution.
- A 45° z-axis rotation positions the Dex3 thumb opposite the other fingers, improving contact area and force closure.
- Rotation is clipped within 70° to avoid twisted inverse-kinematics postures that degrade tracking.
5 Experiments
Experiments evaluate HERO’s open-vocabulary grasping, tracking accuracy, learned forward models, and closed-loop adaptations across simulated and real-world settings. HERO achieves strong grasping success and substantially lower end-effector tracking errors than the compared controllers.
- End-to-end system testing: 22/30 (73.3%) success occurs across broader novel scenes, while cluttered layouts achieve 12/15 (80%) success.The broader scenes include environments such as robot labs and classrooms.
- End-to-end system testing: 90% success across objects at two table heights exceeds 10% for PD control and 16.7% for automated SONIC.Trials require grasping the queried object using onboard sensing and lifting it for more than 2 seconds.
- End-effector tracking accuracy evaluation and ablation study: 2.44cm real-world translation error and 8.22° orientation error make HERO more accurate than PD control and automated SONIC.The comparison uses real-world MOCAP measurements; PD records 12.09cm and 40.82°, while SONIC records 13.38cm and 16.75°.
- End-effector tracking accuracy evaluation and ablation study: 2.48cm simulation translation error is lower than SONIC’s 4.10cm, AMO’s 8.29cm, and FALCON’s 13.57cm.Simulation results are averaged over three table heights in MuJoCo.
- End-effector tracking accuracy evaluation and ablation study: 4.67cm decreases to 2.56cm when learned EE and base estimates replace less accurate analytical estimates, approaching the 2.44cm MOCAP oracle.The ablation attributes the improvement to reduced analytical-FK bias and more accurate pose estimates.
- End-effector tracking accuracy evaluation and ablation study: Removing replanning increases translation error from 2.44cm to 5.17cm, while removing goal adjustment causes a smaller degradation.These closed-loop feedback ablations are conducted on real hardware.
6 Discussions and Limitations
The discussion reports successful language-grounded object selection and identifies scope boundaries involving visual field of view, motion efficiency, modular failures, and dexterity.
- Limitations: Restricted egocentric field of view limits reaching for far or high targets, motivating active neck control and visual servoing.The discussion also identifies inefficient motions, modular failures, and dexterity limits as remaining limitations.
- Discussions: The system selects the language-specified object among relevant distractors, distinguishing red and green apples according to the query.The reported trials demonstrate language-sensitive object selection in both query directions.
A.2 Tracking Error Distribution Analysis
HERO produces lower and tighter end-effector tracking errors than prior baselines, while replanning substantially improves translation accuracy. The system also adapts grasp trajectories to moving targets through visual closed-loop replanning.
- At 80%, HERO reaches 3.9 cm translation and 6.7° rotation error, versus 20.9 cm / 25.3° for FALCON and 9.8 cm / 19.5° for AMO.
- At 90%, HERO remains below 4.6 cm translation and 8.2° rotation error, indicating robust performance in the error tail.
- At 80%, replanning reduces HERO’s translation error from 6.3 cm to 3.1 cm, a 2.1× improvement.
- Replanning also improves 90% translation error from 7.1 cm to 3.4 cm, with smaller but consistent rotational gains from 16.1° to 13.6° median error.
- Visual closed-loop replanning lets HERO update grasp trajectories as a moving object changes position.
A.4 Extending HERO to Other Tasks Like Door Opening
HERO’s modular separation of high-level grasp planning and low-level end-effector control extends beyond object pickup to fridge-door opening. The system identifies a handle grasp pose and executes the reaching and return motion needed to open the door.
- HERO identifies a fridge-door-handle grasp pose and uses the object-grasping pipeline to reach, grasp, and open the door.The door was left unlatched because magnetic attraction required substantial opening force.
- The result demonstrates the modular system’s potential for broader loco-manipulation tasks.The passage specifically points to extending the system with off-the-shelf trajectory-generation frameworks.
A.5 Field of View Analysis
HERO’s onboard egocentric camera provides a short-range view, so object search requires walking toward targets before perception and grasping can proceed. Its visual pipeline converts language queries and RGB-D inputs into retargeted dexterous grasp poses.
- Field of View Analysis: The robot cannot detect the stapler from 1.28 m, but detects and grasps it after walking forward to within 0.6 m.The robot stops at about 0.5 m after continuously detecting the object.
- Field of View Analysis: The limited field of view makes prior 3D spatial understanding important for searching objects.
- Closed-Loop Replanning: Visual closed-loop replanning enables HERO to grasp a moving object despite its limited view.
- Task Extension: The door-opening extension applies the same grasping-pose pipeline to a fridge handle.
- Visual Perception Pipeline: GroundingDINO detects the language-specified object, SAM-3 segments it, and AnyGrasp filters grasp proposals using the segmentation mask.
- Grasp Selection and Retargeting: HERO selects grasps using hand-side, height, ground-parallelism, and confidence criteria before retargeting the result to a 6-DoF Dex-3 hand pose.
A.7 Analytical FK Error Visualization
Analytical forward-kinematics error varies with end-effector location, increasing at larger Y and Z coordinates. HERO’s replanning and learned forward models address the resulting tracking drift over time.
- Analytical forward-kinematics error generally increases as end-effector location grows along the Y and Z axes.The visualization contains 60 MOCAP-room samples collected at one minute.
- The spatial error pattern may be learnable by a neural forward model.
- Without replanning, tracking error plateaus higher because the policy follows a stale reference; replanning reduces accumulated drift with fresh references.
- Learned η and ξ estimates closely track the MOCAP-oracle curve, supporting their sufficiency for accurate real-world tracking.
A.9 Whole-Body Reaching Workspace Analysis
The analysis compares arm-only and arms-plus-waist inverse-kinematics configurations and estimates reachable workspace by voxelized sampling. Adding waist motion substantially expands the workspace, enabling farther-forward and lower-height targets while supporting whole-body object retrieval.
- Workspace estimation: Reachability is estimated by testing 3D grid points with cuRobo IK under joint-limit and end-effector residual constraints.The candidate region is discretized at 0.02 m resolution, and converged points below tolerance are counted as reachable.
- Workspace estimation: The workspace volume is approximated as N_reach · (0.02)^3, where N_reach is the number of reachable grid points.This voxel-counting estimate converts the reachable-point count into cubic meters.
- Effect of waist DoFs: 0.248 m3 to 0.523 m3: the combined two-arm reachable workspace grows approximately 2.1× when waist DoFs are enabled.The comparison is between arms-only and arms+waist configurations, with 14 and 17 optimized DoFs respectively.
- Effect of waist DoFs: 0.166 m3 to 0.426 m3: the single-arm reachable workspace increases when the waist is included in IK.Bending and twisting reposition the shoulder frame, extending reach toward farther-forward and lower-height targets.
- Workspace showcase: HERO retrieves objects distributed across a tabletop workspace with every object positioned beyond 0.4 m from the robot base.The showcase requires whole-body coordination for precise reaching while maintaining stability, including objects at varied heights, poses, and locations.
B Additional Implementation Details
The implementation combines motion-capture-based calibration, onboard RGB-D sensing, cuRobo planning, thresholded grasp execution, simulation training, and randomized sim-to-real transfer. Evaluation covers diverse everyday objects and novel scenes with documented physical and language-query attributes.
- MOCAP and calibration: ≤0.2 mm: the Optitrack MOCAP system provides measurement accuracy for calibration and pose evaluation.Markers are placed on robot links and the base, with link-frame alignment obtained using measured offsets and the Kabsch-Umeyama algorithm.
- MOCAP and calibration: 1.8 cm: analytical forward kinematics exhibits systematic hardware-induced error, so MOCAP ground-truth poses are used for camera calibration.The MOCAP-assisted procedure uses 60–70 board poses and achieves reprojection error within 2.5 mm.
- Onboard RGB-D camera: 640 × 480 at 60Hz: the onboard D435i head-mounted camera supplies RGB-D observations for egocentric 3D perception.The neck pitch degree of freedom makes precise camera calibration necessary as the head rotates within a limited range.
- Grasp execution: δ = 1.5cm: the robot closes the hand when hand-to-target grasp distance ΔE_t falls below the grasping threshold.The same local trajectory waypoint is passed to the policy at threshold crossing to improve stability during closure.
- Simulation and sim-to-real: π_t is trained in IsaacGym with 4,096 parallel environments and transferred to MuJoCo for Sim2Sim evaluation before real-world deployment.Training uses 20K iterations, 500Hz simulation, and AdamW optimization; randomized end-effector mass is essential to avoid high-frequency hand oscillations.
- Evaluation setup: 20 daily objects and novel scenes are evaluated with recorded sizes, weights, materials, language queries, and table heights.The scene set is drawn mainly from CSL Studio and the Siebel Center, while the object set spans varied physical properties and irregular shapes.