HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning

• Humanoid bimanual dexterous manipulation requires coordinated arm and hand operations, making autonomous demonstration collection especially challenging.Source: §1 · p. 1
• Existing teleoperation pipelines require many real-world objects, diverse scenes, and skilled operators, limiting task and scenario coverage.Source: §1 · pp. 1–2
• HumanoidGen uses atomic hand operations, spatial asset annotations, and LLM-generated relational constraints to plan executable arm and hand movements.Source: §1 · p. 2
• MCTS enhances LLM reasoning for long-horizon tasks and cases with insufficient spatial annotations.Source: §1 · p. 2
• HGen-Bench contains 20 bimanual dexterous tasks on a Unitree H1-2 with Inspire hands, using constrained optimization to generate successful simulated demonstrations.Source: §1 · p. 2
• Policy performance improves as more generated demonstrations are incorporated for both 2D and 3D diffusion policies.Source: §1 · p. 2

2.1 Spatial Annotation and Scene Generation

• HumanoidGen annotates assets and dexterous hands with key points and axes, enabling spatial and relational reasoning for atomic operations.Annotations cover hand operations, asset functionality, and asset-operation interactions, including grasping, pinching, and articulated-object manipulation.Source: §2 · p. 2, §2.1 · p. 3
• The LLM generates tabletop scenes, asset configurations, and success criteria from task descriptions, scene details, and asset-library information.Generated configurations specify asset categories, quantities, poses, and other attributes, while success criteria determine task completion.Source: §2.1 · p. 4

2.2 LLM-based Task Planning

• The planner decomposes long-horizon tasks into atomic hand operations and arm movements represented by LLM-inferred goal and path constraints solved through trajectory optimization.Dynamic coordinate frames encode point and axis relations such as coincidence, parallelism, and orthogonality across hands, objects, and the world.Source: §2.2 · pp. 4–5
• Collision handling combines proactive avoidance with dynamic contact management, allowing free-space motion and in-contact articulated-object manipulation within the same plan.The planner can retract one hand to make space, maintain object-ignoring lists during contact, and restore collision checking afterward.Source: Figure 3 · p. 4, §2.2 · p. 5

2.3 Enhancing Reasoning with MCTS

• STCR and MCTS improve planning for long-horizon tasks or insufficiently annotated objects by segmenting, truncating failures, combining operations, and resuming search from stored execution states.MCTS uses selection, expansion, and backpropagation, with execution simulation incorporated into expansion.Source: §2.3 · pp. 5–6
• Room-level scene scaling transforms table-level demonstrations across more than 120 RoboCasa scenes to diversify task environments.Coordinate transformations align the original and new scene systems while preserving demonstration scripts.Source: §2.3 · p. 6

3 The HumanoidGen Benchmark

• HGen-Bench contains 20 tasks executed by Inspire hands on a Unitree H1-2 humanoid with a 26-dimensional action space.The platform has 7 DoFs per arm and 6 DoFs per hand.Source: §3 · p. 6
• The benchmark spans atomic hand operations, bimanual coordination, long-horizon geometric reasoning, and easy-to-hard task difficulties.Scenes include blocks, laptops, drawers, bottles, cups, plates, and randomized articulated-object positions, poses, and joint angles.Source: §3 · p. 6
• HGen-Bench provides RGB and depth observations from six wrist, first-person, and third-person camera views in home-like tabletop scenes.The observation setup includes cameras on both wrists, one first-person view, and three third-person views.Source: §3 · p. 6, Figure 4 · p. 6

LLM-based Data Generation

• LLM-based generation methods predominantly center on single-arm robots, while HumanoidGen targets bimanual dexterous manipulation for humanoids.Its code-form planning frames manipulation as a sequence of constraints, reducing reliance on manually defined functions.Source: §4 · p. 7
• RoboGen, Gensim, and related methods automate scene or demonstration generation, but existing approaches generally do not address bimanual dexterous tasks.Spatial-reasoning systems such as Rekep, OmniManip, and RoboTwin also have stated limitations for this setting.Source: §4 · p. 7

Bimanual Manipulation

• Bimanual dexterous manipulation faces data scarcity, enlarged action spaces, diverse collaboration modalities, and intricate hand control.Existing dual-arm and humanoid benchmarks also have limited task and scene diversity and typically exclude dexterous hands.Source: §4 · p. 7

Datasets and Benchmarks

• Real-world datasets are often unsuitable for directly training humanoid bimanual dexterous policies because of embodiment differences, while large-scale collection remains costly.Simulation benchmarks provide assets and configurations, but this work focuses on automatically creating task variations and collecting humanoid demonstrations.Source: §4 · p. 7

5.1 Evaluation of Data Generation and Execution

• HumanoidGen outperforms Robotwin on long-horizon and complex-collision tasks, while achieving over 50% average success and above 75% on most task types.Robotwin remains comparable on single-arm and bimanual short-horizon tasks; the comparison spans 20 tabletop tasks categorized by arms, horizon length, and collision complexity.Source: §5.1 · pp. 7–8, Figure 5 · p. 8

5.2 Effectiveness Evaluation of MCTS

• MCTS improves LLM reasoning with minimal additional token consumption and supports correction of erroneous execution strategies during difficult tasks.In block stack single, a 26% token increase yielded a 55% reasoning-success improvement, while tokens per successful method fell from 24.17K to 19.63K.Source: §5.2 · pp. 8–9, Table 1 · p. 9
• MCTS generates more diverse successful plans: among 20 successful solutions, it produces at least 12 distinct plans versus at most 5 for non-MCTS.The method explores tree nodes and uses prior execution errors to revise subsequent strategies.Source: §5.2 · p. 9, Figure 6 · p. 9

5.3 Validity Evaluation of Collected Data

• Policy success generally improves with more generated demonstrations, although challenging long-horizon tasks remain data-hungry and some high-success policies are unstable.In open box hard, DP rises from 11.1% with 20 demonstrations to 100% with 100; DP3 reaches 67% on cup pour easy with 20.Source: §5.3 · pp. 9–10, Table 2 · p. 10

6 Conclusion

• HumanoidGen provides spatial annotations, LLM-generated task decomposition and relational constraints, collision-aware trajectory optimization, and MCTS reasoning for complex or long-horizon tasks.The framework also constructs a diverse bimanual dexterous benchmark for evaluation.Source: §6 · p. 10

A.1 Automatic Asset Annotation

• Automatic annotation transfers key points and axes from annotated assets to similar unannotated assets using diffusion-feature matching and 3D back-projection.The method reduces repetitive manual annotation effort.Source: §A.1 · pp. 15–16

A.2 Constraint Solver

• The constraint solver uses LLM-inferred goal and path constraints to optimize end-effector poses and arm trajectories while enforcing kinematics, bounds, regularization, and collision avoidance.SNOPT solves the target-pose problem, while mplib handles intermediate constrained motion.Source: §A.2 · p. 16
• Relational constraints encode distances between points and lines plus angle differences between axes, using points, vectors, and lines as primitive geometric components.The nominal-angle regularizer biases solutions toward safe robot configurations.Source: §A.2 · p. 16

A.3 Scene Scaling

• Coordinate transformations map points, vectors, poses, and spatial relationships from tabletop scenes to larger scenes without additional annotations or complex geometric reasoning.This scene scaling increases dataset diversity while preserving geometric consistency.Source: §A.3 · p. 17

A.4 MCTS-Based LLM Reasoning

• MCTS uses selection, expansion, and backpropagation over executable-code branches, reusing successful branches and rewarding valuable intermediate events during exploration.QDUCB balances exploration and exploitation with discounted feedback, while expansion incorporates runtime scene information.Source: §A.4 · pp. 17–18

Benchmark Task Design

• The benchmark spans 20 tasks covering pinching and grasping, easy and hard variants, bimanual handover, long-horizon manipulation, and geometric stacking.Initial poses, object positions, and articulated-object joint angles are randomized to test policy generalization.Source: B Details of HGen-Bench · p. 18, Table 3 · p. 19

Policy Evaluation

• Across 20 tasks, increasing generated trajectories generally improves success on moderately difficult tasks, while extremely challenging long-horizon and fine-grained tasks remain difficult for DP and DP3.The full evaluation uses 100, 50, and 20 generated trajectories, with success rates reported across tasks.Source: B Details of HGen-Bench · p. 18, Table 3 · p. 19
• DP3 achieves strong few-shot performance on relatively simple tasks with as few as 20 trajectories, whereas harder tasks require more data for consistent improvement.The reported results include mean and standard deviation across three evaluation seeds.Source: B Details of HGen-Bench · p. 18
• Some DP results are marked as anomalously high because random movements can occasionally achieve success, making the apparent performance unstable and unsafe.In Close Box Hard, deliberate closing motions expose collisions caused by insufficient spatial understanding and perception.Source: B Details of HGen-Bench · p. 19

Policy Implementation

• DP uses image and proprioceptive inputs, while DP3 uses 3D point clouds and proprioceptive inputs under separate diffusion-policy architectures and training configurations.Both policies use conditional denoising diffusion but differ in observation modality and architecture.Source: B Details of HGen-Bench · pp. 19–20

Experiment Setup

• The real-world setup uses a Unitree H1-2 with Inspire hands and a head-mounted D435i camera, evaluating dual-bottle picking and hard laptop closing under randomized object poses.The two tasks use 6 cm × 6 cm position randomization with task-specific orientation ranges.Source: §C.1 · p. 20, Figure 9 · p. 20

Experimental Procedure

• The transfer pipeline estimates object poses with FoundationPose, generates trajectories in simulation, executes them on the robot, and trains policies with real and simulated data.Each task includes 25 collected real-world trajectories, with an additional 100-simulation-plus-5-real setting.Source: §C.1 · p. 20

Experimental Results

• About 14 seconds are needed per real-world trajectory, and collection success exceeds 85% for both evaluated tasks.These metrics are reported alongside diffusion-policy performance under multiple data settings.Source: §C.1 · p. 21, Table 5 · p. 21
• Policies trained with 25 real trajectories outperform those trained with 5, while 100 simulated trajectories plus 5 real trajectories can match or exceed the 25-real-trajectory setting on dual-bottle picking.The result supports simulation-data augmentation as a way to improve real-world policy performance under the tested conditions.Source: §C.1 · p. 21, Table 5 · p. 21

• The evaluation selected five parts from four common object types, using 20 assets per category and ManiSkill3 for rendering and reprojection.Source: §C.2 · p. 21
• 0.7 seconds per asset on average, versus about 4–6 seconds per part for manual annotation, substantially accelerates asset labeling.Manual annotation took about 4 seconds for cups and bowls and 6 seconds for laptops and drawers.Source: §C.2 · p. 22, Table 6 · p. 22
• Automatic annotation reached 100% success for cup rims, bowls, and laptops, and over 90% for other evaluated parts.Failures mainly involved geometric variation, such as circular versus rectangular handle shapes.Source: §C.2 · p. 22

• The supplementary evaluation covers safe-knob rotation, toaster pressing, safe-door opening, dual-pot lifting, and block stacking with barriers.These tasks require fingertip control, articulated-object manipulation, coordinated bimanual grasping, or severe spatial navigation constraints.Source: §C.3 · p. 22, Figure 11 · p. 22
• 73% success in blocks stack hard with barrier despite over 60% of the workspace being blocked, outperforming comparable baselines.Three of the five additional tasks exceeded 90% success.Source: §C.3 · p. 22, Table 7 · p. 22

C.4 Resource and Efficiency Analysis of HumanoidGen

• HumanoidGen achieves high efficiency in automated data generation and resource utilization, supporting scalable and reproducible evaluations in HGen-Bench.Table 8 reports average time and resource consumption across all 20 HGen-Bench tasks, with manual annotation as the limited human-effort stage.Source: §C.4 · p. 23, Table 8 · p. 23

C.5 Comparison with Existing Generation Frameworks

• HumanoidGen provides object-level reuse and full automation, unlike augmentation methods that require expert trajectories, manual segmentation, and new expert data when execution modes change.Source: §C.5 · p. 23
• HumanoidGen uniquely unifies bimanual dexterous hands, scene generation, room-level synthesis, and dynamic collision management in a zero-shot generation pipeline.The comparison identifies these capabilities as absent from the other frameworks considered.Source: §C.5 · p. 23, Table 9 · p. 23

C.6 Comparison Across Different Large Models

• All four tested planner models achieve high success rates for valid constraints and executable code, while DeepSeek models decline slightly on bimanual and long-horizon tasks.GPT-4o performs consistently well with and without visual input across the four representative tasks.Source: §C.6 · p. 24, Figure 12 · p. 24

C.7 Task Descriptions in Data Generation Experiment

• HGen-Bench contains 20 randomized tabletop tasks, including 12 single-arm and 8 bimanual tasks, with dexterous hands used throughout.Some tasks include handoffs and cooperation between both arms.Source: §C.7 · p. 24

• Minimal task descriptions omit operation order and executor information, testing whether the reasoning LLM can infer both during demonstration generation.Source: §C.8 · p. 24

D More Discussion on Limitations and Future Work

• Novel asset categories still require manual annotation because annotation transfer cannot label unseen manipulation primitives, although zero-shot predictors may enable future automation.The authors cite learning-based contact-point, axis, and grasp-pose predictors [59] [61] as possible integrations.Source: D More Discussion on Limitations and… · p. 24
• The framework cannot simulate deformable or fluid-object tasks in ManiSkill3 [52] or reliably handle ambiguous, dynamic objectives requiring continuous state adjustment.Current arm control uses a low-level motion planner, limiting tasks such as push-T and cloth flattening.Source: D More Discussion on Limitations and… · pp. 24–25
• Future work will add physics engines such as IsaacSim [15] and MuJoCo [16], combine goal-conditioned planning with goal-agnostic atomic-operation models, and expand HGen-Bench.Planned benchmark expansion covers additional assets, scenes, tasks, and evaluation policies.Source: D More Discussion on Limitations and… · p. 25

E Prompts and Generation Samples

• The supplementary material provides prompt templates, placeholder definitions, and sample LLM-generated code for scene and demonstration-script generation, with and without MCTS.Table 12 explains the meanings of the prompt placeholders.Source: E Prompts and Generation Samples · p. 26, Table 12 · p. 26

E.1 Prompt Template for Scene Generation

• The scene-generation prompt specifies robot and table geometry, available assets, coordinate conventions, hand workspaces, bounding boxes, and a code example for initializing and validating object placement.The example places two cubes and checks their target positions within specified tolerances.Source: §E.1 · pp. 26–27

E.2 Sample Scene Code for Blocks Stack Hard Generated by HumanoidGen

• The Blocks Stack Hard sample initializes three cubes with optional randomized poses and verifies stacking through aligned x-y positions and fixed height differences.The success check uses eps_xy = 0.03, eps_z = 0.005, and height_diff = 0.04.Source: §E.2 · p. 28

E.3 Prompt Template for Demonstration Script Generation

• The demonstration-generation prompt exposes hand actions, grasp and pinch controls, constraint generation, end-effector pose computation, screw-motion execution, object attachment, and stepwise code-format requirements.Examples cover grasping, target placement, relative motion, articulated-object manipulation, hand selection, and world-coordinate conventions.Source: §E.3 · pp. 28–31

E.4 Sample Demonstration Code for Open Drawer Generated by HumanoidGen

• The open-drawer sample demonstrates a three-step sequence: pre-grasp and approach, grasp and move to openness = 1 with attachment, then release.The generated code uses left-hand grasping, an openness constraint, attached-object motion, and hand opening.Source: §E.4 · p. 31

E.5 Prompt Template for MCTS-integrated Demonstration Script Generation

• The MCTS-integrated prompt template defines spatial constraints, hand operations, motion functions, collision-aware planning rules, and operational-range requirements for generating executable manipulation scripts.The template distinguishes point-to-point and parallel constraints, requires pre-grasp or pre-pinch actions, and advises staged movements and slightly elevated releases to avoid collisions and inverse-kinematics issues.Source: §E.5 · pp. 34, 35, 38

E.6 Sample Demonstration Code for Block Stack Hard Generated by Applying MCTS

• MCTS-generated demonstration code executes a three-step bimanual stack: the right hand places cube1, the left hand stacks cube0, and the right hand completes the stack with cube2.Each step uses pre-pinch preparation, point-to-point and parallel constraints, generated end-effector poses, attachment during transport, release, and return to an initial pose.Source: §E.6 · pp. 39–40

F Broader Impacts

• The framework automates task creation and demonstration collection without human intervention, reducing data-acquisition cost and complexity while supporting consistent, diverse, long-horizon bimanual manipulation tasks.Potential applications include assistive robotics, disaster response, and intelligent automation involving operations such as opening containers, pouring, handling tools, and manipulating irregular objects in constrained environments.Source: F Broader Impacts · p. 41