Source-linked AI summary
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Kyochul Jang, Seohyeon Park, Ohchul Kwon, Sangjun Park, Junhyeok Choi, Seungyeop Yi, Chaeyun Kim, Sangkyu Lee, Idan Szpektor, Avi Caciularu, Jongmin Park, Youngjae Yu
TL;DR
Humanoid tool use requires choosing a suitable tool and coordinating manipulation and, when needed, locomotion, yet evaluating the full process remains a challenge. HumanoidToolBench pairs an 18-task benchmark with ToolBook demonstrations and evaluates policies in simulation and on a real robot. Results expose gaps between tool interaction and task completion, while focused probes reveal sensitivity to unseen tools and unrelated instructions.
Problem
Humanoids need tools to extend their physical capabilities, making evaluation of suitable tool selection and coordinated use through task completion important.
Source: §I · p. 1Method
HumanoidToolBench organizes 18 tasks across three scenarios, execution levels, and tool-set modes, and ToolBook supplies 3.1k demonstrations from simulation and a real Unitree G1.
Source: §I · pp. 1–2Results
Recorded events reveal gaps between candidate contact, suitable-tool lifting, and task success, while decoys change initial choices more than final success.
Source: §IV-C · p. 7Takeaways & Limitations
The findings motivate policies that adapt selection and execution to functional requirements and task goals beyond familiar training patterns.
Source: §VI · p. 8The main benchmark comparison uses one training run per policy; real-robot evaluation covers three policies on two L1 scenarios, and instruction probes use one fixed IceBreak scene.
Source: §V · pp. 7–8
Abstract
from arXiv · showhide
As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/.
I. INTRODUCTION
Humanoid tool use requires selecting a suitable tool and coordinating grasping, manipulation, and sometimes locomotion through task completion, capabilities existing benchmarks do not jointly evaluate on humanoids. HumanoidToolBench addresses this gap with tasks spanning tool selection and pickup through stationary and mobile use, alongside demonstrations and evaluations in simulation and on a real robot.
I. INTRODUCTION
- Humanoid tool use requires a suitable tool plus coordinated grasping, manipulation, and sometimes locomotion; full-process evaluation must extend from selection to task completion.The grasp affects available reach and working-end orientation, while body motion must preserve contact with the target.Source: §I · p. 1
- Existing benchmarks cover only parts of the problem: humanoid tasks omit choices among tools with contrasting properties, while dedicated tool-use benchmarks focus on VQA or fixed-base arms.The section identifies the need to jointly evaluate functional selection and task completion on humanoids in stationary and mobile settings.Source: §I · p. 1
- HumanoidToolBench combines three scenarios, three execution levels, and two tool-set modes into 18 tasks, while ToolBook provides 3.1k human demonstrations from simulation and a real Unitree G1.The levels separate selection and pickup, stationary use, and mobile use; decoy mode adds a competing tool lacking a required property.Source: §I · pp. 1–2
- The evaluation compares seven policies in simulation and three on the real G1, using interaction events to distinguish candidate contact, suitable-tool lifting, and target application.Focused GR00T N1.7 probes examine unseen suitable tools and instruction changes; the reported results show gaps between acquisition and completion, with further declines when locomotion and manipulation are combined.Source: §I · p. 2
- The benchmark combines execution levels, decoy tool sets, and recorded interaction events to distinguish limitations in tool selection from limitations in task execution.The paper presents it as the first humanoid tool-use benchmark spanning selection and pickup, stationary use, and mobile use, with ToolBook demonstrations.Source: §I · p. 2
II. RELATED WORK
Prior manipulation, humanoid, and robotic tool-use benchmarks evaluate subsets of tool selection and execution, but do not jointly cover functional selection, stationary and mobile use, and controlled tool-set variations on a humanoid. HumanoidToolBench structures its scenarios across these dimensions.
II. RELATED WORK
- HumanoidBench includes spoon and window-wiping tasks but not choices among contrasting tools; SIMPLE and WOLF-VLA also lack the joint evaluation of functional selection, stationary and mobile use, and controlled tool-set variation.SIMPLE stages tasks but does not vary the decision required, and WOLF-VLA releases no human demonstrations for training.Source: §II · p. 2
- Robotic tool-use benchmarks cover separate capabilities: PhysToolBench uses VQA, DexCraft provides the tool to a single arm, and RoboWits tests dual-arm tool selection in unexpected situations.None evaluates tool use on a humanoid while combining execution levels with controlled tool-set variation.Source: §II · p. 2
- HumanoidToolBench pairs three execution levels with standard and decoy modes across 18 tasks, linking functional suitability to the physical demands of grasping, target contact, and body motion.The task structure comprises three scenarios, three execution levels, and two tool-set modes.Source: §II · p. 3, Figure 2 · p. 3
III. HUMANOIDTOOLBENCH
HumanoidToolBench evaluates humanoid tool selection and use across scenarios, execution levels, and tool-set modes, while ToolBook supplies human demonstrations for these tasks. Its design separates selection and pickup from stationary and mobile tool use, and evaluates policies in simulation and on a real robot.
III-B. Design Philosophy and Tasks
- Three scenarios, three execution levels, and two tool-set modes form 18 tasks, separating selection and pickup from stationary and mobile tool use.BallMove tests reach, BallRetrieve tests tool shape for engaging and pulling, and IceBreak tests material and structure for striking; decoys lack a required property.Source: §III · p. 3, §III-B · p. 4
III-A. Problem Formulation
- Tasks are modeled as goal-conditioned POMDPs in which the language goal omits the suitable tool’s name and success is binary.Observations combine a head-camera image with robot state, and policies act from their observation history; selection tasks require lifting the suitable tool, while use tasks require reaching the scenario goal.Source: §III-A · p. 3
III-C. Environment
- Each simulation scene samples tools from 55 Objaverse assets and randomizes candidate order, placements, robot spawn, surface materials, and lighting.The Unitree G1 uses 29 body joints, two seven-joint Dex3 hands, and a head camera, with a decoupled control stack and pretrained balance/walk controllers for locomotion.Source: §III-C · p. 4
- ToolBook contains 3,094 trajectories: 1,803 simulation L0 trajectories, 1,200 simulation L1/L2 demonstrations, and 91 curated real-robot L1 demonstrations.The L0 trajectories are truncated from attempts when the suitable tool first rises 0.08 m; the simulation L1/L2 demonstrations comprise 100 successful demonstrations for each of 12 tasks.Source: §III-C · p. 4
III-E. Evaluation Protocol
- Evaluation reports success rates across scenarios, levels, and modes in simulation, with real-robot evaluation covering L1 BallMove and BallRetrieve in both modes.Simulation policies train on simulation demonstrations; real-evaluation policies are additionally fine-tuned on real-robot demonstrations.Source: §III-E · pp. 4–5, Table III · p. 5, Table IV · p. 5
IV. EXPERIMENTS
The experiments compare seven policies in simulation and three adapted policies on the real G1, then use interaction events and focused GR00T N1.7 probes to examine where performance breaks down. Results vary with policy, tool function, locomotion, competing tools, and tool or instruction changes.
IV-A. Experimental Setup
- Policies were trained on ToolBook demonstrations and evaluated across 18 simulation tasks, with three policies also tested on the real G1 after fine-tuning.Simulation training used 3,003 trajectories; real-robot adaptation used 91 demonstrations, followed by 10 trials per scenario-mode combination.Source: §IV-A · p. 5
IV-B. Results
- FastWAM and GR00T N1.7 outperform the other simulated policies on stationary tool use, but every policy loses success from L1 to L2 when L1 success is nonzero; stronger simulated policies also outperform ACT on the real G1.On standard BallMove, FastWAM falls from 76% at L1 to 9% at L2; on the real robot, GR00T N1.7 and FastWAM succeed in 6/10 and 5/10 trials versus ACT's 1/10 after adaptation.Source: §IV-B · p. 6
IV-D. Responses to Tool and Instruction Changes
- For GR00T N1.7, unseen suitable tools reduce correct first contact from 85% to 74% in BallMove and from 59% to 36% in BallRetrieve.Unseen instances form groups apart from most seen instances in BallRetrieve; the reviewed rollouts also show decoy pickup and tool-target alignment errors.Source: Figure 4 · p. 6, Figure 6 · p. 7, §IV-D · p. 7
IV-C. Where Do Policies Fail in Tool Use?
- Execution-event rates distinguish candidate contact, suitable-tool pickup, tool-target contact, and task success, revealing that frequent contact or pickup does not ensure completion.In standard mode, GR00T N1.7 and FastWAM lift suitable tools in 90.0% and 89.2% of episodes, while success is 36.0% and 52.8%; decoys reduce their correct-first-contact rates more than their success rates.Source: §IV-C · pp. 6–7, Figure 5 · p. 6
IV-D. Responses to Tool and Instruction Changes
- In a fixed IceBreak scene, GR00T N1.7 succeeds in 77% of original-instruction, 70% of paraphrase, and 84% of unrelated-instruction rollouts.Successful rollouts retain similar broad arm profiles and a shared middle-phase wrist pattern across the three instructions.Source: Figure 7 · p. 7, §IV-D · p. 7
V. DISCUSSION AND LIMITATIONS
The evaluation has limited breadth and leaves several sources of variation unresolved. The authors identify broader tools, scenes, goals, and training seeds as ways to extend the findings.
V. DISCUSSION AND LIMITATIONS
- The main benchmark comparison uses one training run per policy; real-robot evaluation covers three policies on two L1 scenarios, and focused probes cover GR00T N1.7.Episode variation reflects scene and rollout variability rather than training-seed uncertainty.Source: §V · p. 7
- The instruction probe varies wording and task relevance in one fixed IceBreak scene, so testing different goals would further assess goal switching.Broader tools, scenes, goals, and additional training seeds would extend the findings.Source: §V · p. 8
- Egocentric visual input is only a hypothesized contributor to spatial-alignment difficulties; controlled viewpoint and spatial-observation comparisons are proposed to clarify its role.Separate simulation-only and real-data-adaptation comparisons would also measure fine-tuning's contribution to real-robot performance.Source: §V · p. 8
VI. CONCLUSION
HumanoidToolBench and ToolBook support evaluation from tool selection and pickup through mobile task completion. The findings distinguish selection and execution failures and reveal sensitivity to unseen tool appearances and unrelated language.
VI. CONCLUSION
- HumanoidToolBench and ToolBook support evaluation of humanoid tool use from selection and pickup to mobile task completion.Source: §VI · p. 8
- Across seven policies, recorded events distinguish failures to lift a suitable tool from failures to complete the task after contact, while decoys change initial choices more than final success.Source: §VI · p. 8
- Focused probes reveal sensitivity to unseen tool appearances and continued task execution under unrelated language, motivating policies that adapt selection and execution to functional requirements and task goals.Source: §VI · p. 8