HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

• 18,000 curated simulation demonstrations provide approximately 56 hours of interaction, with re-rendering and visual and prompt randomization broadening observation and language coverage.Source: §1 · p. 4
• Core-50 supplies demonstrations for 50-task learning and fresh-rollout evaluation, while Transfer-16 reserves 16 tasks for generalization, including three deformable interactions without target-task demonstrations.Source: §1 · p. 5
• 29.1% is the best trained multi-task Core-50 score; without HandsOn finetuning, GPT-6 with move_eef reaches 5.2% Core-50 and 1.9% Transfer-16 over 660 trials.Source: §1 · p. 5
• Trained-policy failures concentrate in premature contact-triggered actions, simultaneous target holding, and maintaining reached goals, with composed tasks adding step-order and completion failures.Source: §1 · p. 5

• DexJoCo, DexVerse, Bench2Dex, and LabDex cover functional and bimanual manipulation with diverse embodiments, while DexMimicGen expands human seeds and SPD studies simulation pretraining and real-data adaptation.Source: §2 · p. 5
• HandsOn evaluates diffusion, five VLA, three WAM, and prompted GPT-6 move_eef methods across a demonstrated suite and a separate dexterous task-transfer suite.Source: §2 · p. 5
• Earlier dexterous suites learn mainly through reinforcement or small demonstration sets, while large manipulation corpora are dominated by grippers; teleoperation and motion-capture data were collected outside benchmarks.Source: §2 · p. 5

3. The HandsOn Benchmark

• Tasks use native task-specific success predicates, and every task was verified in person by data-collection operators before release.Source: §3 · p. 6

3.1. Core-50 and Transfer-16

• Core-50 supports learning and fresh-rollout execution evaluation, whereas Transfer-16 tests held-out asset relations, skill compositions, and deformable interactions without target-task demonstrations.Source: §3.1 · p. 6
• Figure 3 classifies tasks by interaction, structure, and asset type; its cells report category intersections, with deformable manipulation reserved for evaluation-only Transfer-16.Source: Figure 3 · p. 7

3.2. Task Design and Feasibility Verification

• The collection protocol separates episode recording, independent review, and disagreement arbitration; task cards specify goals, evidence, action rules, time limits, and versioned checkers.Source: §3.2 · p. 8
• Every task was operator-verified, each Core-50 task is completed by curated demonstrations, and 47 of 50 Core-50 tasks had at least one evaluated multi-task checkpoint succeed.Source: §3.2 · p. 8

3.3. Evaluation Protocol

• Evaluation executes actions at 30 Hz under physical-time budgets, and success requires the complete native predicate; native failure or budget exhaustion without success scores zero.Source: §3.3 · pp. 8–9
• GPT-6 has an additional model-call budget and is reported separately from trained-checkpoint aggregates without HandsOn finetuning.Source: §3.3 · p. 9

3.4. Diagnostics and Statistical Reporting

• Ordered progress records the longest valid completed stage prefix but never overrides the final success predicate; capability diagnostics distinguish tool acquisition from operation and other task-specific measures.Source: §3.4 · p. 9

4.1. Collection System and Operators

• The collection system records synchronized observations, measured state, commanded actions, language, and metadata, using glove and tracker data retargeted to arm and hand commands.At least ten operators contributed over 1,050 sessions; all demonstrations are simulated, while the physical platform is used only for qualitative rollouts.Source: §4.1 · p. 10

4.2. Dataset Composition and Identity

• The 18,000-trajectory release is a curated distribution, not an unbiased sample; its 54,000 video streams are three views of the same interactions.The package contains 15,178 training and 2,822 validation trajectories; task-group quotas favor shorter eligible demonstrations.Source: §4.2 · p. 11, Table 3 · p. 11

4.3. High-Fidelity Re-rendering

• Re-rendering varies recorded appearance while preserving interaction labels, but geometry and dynamics changes require valid re-execution rather than frozen-state replay.Appearance is sampled per trajectory and fixed during replay; the reported material, palette, and roughness factors are inside the training distribution, making their test an invariance check rather than a visual-generalization result.Source: §4.3 · pp. 11–12

4.4. Language Bank and Prompt Sampling

• For 91 active task–goal entries, the language bank provides 32 training and eight held-out paraphrases per entry, with lexical disjointness between held-out and canonical or training strings.Paraphrases preserve the same goal; changing an object, setting, or operation order requires a corresponding executable episode.Source: §4.4 · p. 12

4.5. Physical Platform and Simulated Correspondence

• The simulated platform follows ROKAE AR5-08 arms and first-generation Wuji Hands, including their kinematics, asset scale, cameras, state and action frames, and timing.Source: §4.5 · p. 12

4.6. Portability Across Arms, Hands, and Physics Backends

• The task and asset layer is hardware-independent, and the same suite definitions can be instantiated across supported arms, hands, and physics back-ends.However, every reported experiment uses ROKAE AR5-08 arms, first-generation Wuji Hands, and NVIDIA PhysX; supported alternatives do not contribute to the reported results.Source: §4.6 · pp. 12–13

5.1. Models and Experimental Setup

• The setup evaluates eight Core-50-trained multi-task checkpoints, a single-task diffusion-policy reference, and GPT-6; differing pretraining and compute prevent isolating architecture.GPT-6 receives RGB views, measured state, the instruction, and execution feedback, and controls palm poses and individual finger targets through move_eef.Source: §5.1 · p. 13

5.2.1. Performance on Core-50

• 29.1%: OpenWAM leads trained multi-task checkpoints on Core-50, while choosing the best checkpoint per task reaches 35.4%.Tool use and precision fits remain near-unsolved: only 2.9% of tool-use episodes and 2.2% of precision-fit episodes succeed.Source: Table 5 · p. 14, §5.2.1 · pp. 14–15

5.2.2. Generalization to Transfer-16

• 12.9%: OpenWAM leads trained checkpoints in Transfer-16 ordered progress, but success is near zero across task families and all deformable tasks remain unsolved.Its ordered-progress score exceeds HY-Embodied, LingBot-VLA, and Spatial Forcing by 6.0, 3.7, and 2.6 points, respectively.Source: Table 6 · p. 17, §5.2.2 · p. 17

5.2.3. GPT-6 with move_eef

• 5.2%: GPT-6 with move_eef succeeds on Core-50, 1.9% on Transfer-16, and 4.4% overall without benchmark demonstrations or finetuning.Success occurs on only ten Core-50 and two Transfer-16 tasks; its Core-50 score is below every learned-policy reference, under a different interface and call budget.Source: §5.2.3 · pp. 17–19, Table 7 · p. 18

5.3.1. Does Demonstration Scale Explain the Remaining Gap?

• 7%→83%: 𝜋0.5 success rises on Place Ball in Bowl across tested demonstration sizes, while Squeeze Scissors rises from 1% to 44%.Using the full pool rather than 200 trajectories adds 18 and 26 points, respectively; after adjusting for demonstrations and expert duration, skill-group gaps remain 20–25 points below grasp–transport–release.Source: §5.3.1 · p. 19, Figure 8 · p. 20, Table 8 · p. 20

5.3.2. Do Policies Read the Instruction?

• With swapped object names, HY-Embodied and LingBot-VLA still head for the originally named ball in 57% and 71% of target-level probe cases.Their palm endpoints shift only 7.3 and 8.5 cm under instruction swaps, and the findings suggest limited instruction conditioning rather than establishing its cause.Source: §5.3.2 · p. 21

5.3.3. Where Do Episodes Fail?

• Core-50 failures concentrate at actuation, coordination, and goal-maintenance transitions: grasping can trigger functions early, two hands rarely hold targets together, and achieved states are often lost.For example, 443 of 800 Move and Click Mouse episodes click before readiness; only 38 of 800 Set Dual Knobs episodes bring both knobs to target simultaneously, and Open Laptop displays are often pushed past their target angle.Source: §5.3.3 · pp. 21, 23
• Tool bypass and transfer expose further failures: direct hand contact dominates several tool tasks, new objects can block acquisition, and composed tasks add sequencing errors.Matched Transfer-16 transitions fail more often than their Core-50 sources; composition tasks can repeat or start steps too early.Source: §5.3.3 · p. 23
• 311 of 399 Place Fruit and Photograph Logo episodes, 107 of 400 Scrub Bottle Then Fill Cup episodes, and 340 of 400 Cover Pitcher and Box episodes show premature next-step actions.The shutter is pressed before the fruit is in the bucket, water runs before scrubbing finishes, and the box lid moves before the pitcher lid; completion must be inferred from the scene.Source: §5.3.3 · p. 23

5.3.4. Task Grouping and Reliability

• Core-50 tasks are assigned to exactly one group by the hand skill that determines success, with tasks requiring reaching and grasping assigned to the later skill.This grouping is an analysis lens independent of the catalog’s interaction and structure labels; Figure 10 reports per-task success for eight trained checkpoints and GPT-6 within it.Source: §5.3.4 · p. 23
• 32.9% versus 29.1%: selecting a checkpoint per task on half the episodes and scoring it on the other half outperformed the best single checkpoint.Twelve tasks selected by signal-to-noise reproduced 96% of 28 pairwise checkpoint orders (88% for random subsets); 24 tasks reproduced all orders. Episode-only 95% half-widths for the 50-task macro score were about 1.9, 1.3, and 0.9 points at 25, 50, and 100 episodes per task.Source: §5.3.4 · p. 24
• Across 48 shared tasks, the earlier campaign yielded 24.8 versus 23.4% (π0.5) and 30.5 versus 28.5% (OpenWAM), with per-task Spearman ρ of 0.98 and 0.97.Within Core-50, appearance effects were at most 0.8 points (permutation p≥0.17); paraphrases stayed within 2.2 points for seven checkpoints, while OpenWAM scored canonical +4.0 points (z=2.9). These appearance factors and paraphrases are in-distribution invariance checks, not generalization results.Source: §5.3.4 · p. 24

6. Discussion and Conclusion

• HandsOn supports systematic study of policy behavior by treating embodiment, environment, and task structure as configurable variables.The benchmark is presented as a shared experimental basis for investigating manipulation competence and generalization limits.Source: §6 · p. 26
• 35–51% success for pushing, articulation, and pick-and-place contrasts with 2–3% for tool use and precision fitting across eight multi-task checkpoints.The gaps persist after accounting for demonstration count and expert demonstration duration; all tasks were demonstrated successfully by human operators.Source: §6 · p. 26
• 29 of 660 trials succeed for GPT-6 with move_eef: 5.2% on Core-50 and 1.9% on Transfer-16.The minimal harness maps subgoals to palm poses and up to 40 finger-joint targets with limited geometric and contact feedback.Source: §6 · p. 26
• Policies must regulate interaction state throughout execution, including contact roles, inter-hand dependencies, achieved goals, and task progress.Composed-task failures also point to tracking completed steps and satisfying prerequisites before advancing.Source: §6 · p. 26
• Future evaluations should measure task completion and maintenance of success conditions while testing transfer across embodiments and physical robots.Proposed directions include richer VLM harnesses, broader embodiment and composition coverage, paired simulation and real-world evaluation, and modeling contact and task progress.Source: §6 · p. 26

Tactile sensing

• HandsOn policies use vision and proprioception without tactile feedback, while grasped objects can occlude the wrist-camera region being controlled.Adding tactile sensing requires matched taxel placement in simulation and on the physical hand, a verified contact model, and tasks that retain a visual solution.Source: §7 · p. 26

Quantified physical evaluation

• The real-robot rollouts are recorded evidence, not a measured physical success rate.A physical success rate requires paired simulation and real instances, executable predicates, a fixed trial count, and reported failure modes.Source: §7 · p. 27

Long-horizon household and production tasks

• HandsOn scores individual skills, while Transfer-16 compositions chain at most a few operations, leaving longer household and production work outside its current scope.Such evaluation would require long-horizon demonstrations, recovery-capable stage models, and reporting progress and interventions.Source: §7 · p. 27

Evaluating VLMs across richer agent harnesses

• Richer VLM harnesses could add estimated object poses, contact feedback, reusable skills, and hierarchical control to direct move_eef commands.Comparisons should hold the model fixed across harnesses and the harness fixed across models, while matching task, reset, physical-time, and query budgets.Source: §7 · p. 27

A. Task Inventory and Taxonomy

• Core-50 contains 50 learning-and-evaluation tasks, while Transfer-16 contains 16 generalization-only tasks, including six compositional, seven generalization, and three EVAL-Flex tasks.The taxonomy uses interaction type, applicable task-relevant asset types, and task structure; evaluation-only tasks are excluded from policy-training data.Source: A. Task Inventory and Taxonomy · pp. 31, 33, Table 10 · pp. 31–32

A.1. Task Provenance and Adaptation

• Documented source markers distinguish task-design adaptations from asset reuse; shared infrastructure or similar task names alone do not establish lineage.Compositions and variants may reuse credited components without transferring an entire upstream task.Source: A. Task Inventory and Taxonomy · p. 31, §A.1 · p. 33, Table 11 · p. 34

B.1. Physical System and Alignment Record

• The physical correspondence record covers geometry, asset scale, camera calibration, coordinate frames, ordering, timing, initial conditions, and success.The ROKAE AR5-08 has seven degrees of freedom and the Wuji Hand has 20 actively actuated joints; calibration data are disclosed independently of target-task demonstrations.Source: §B.1 · p. 34

B.2. Transfer-16 Duration Map

• Table 12 reports Transfer-16 physical-time budgets under a 30 Hz protocol, with Flex budgets identified as native exceptions rather than P75 estimates.The source-frame column sums each mapped atomic task’s P75 before multiplying by three.Source: §B.2 · p. 34, Table 12 · p. 35

B.3. Task Partitions and Generalization Design

• Transfer-16 tasks are evaluation-only, grouped as six EVAL-Comb, seven EVAL-Gen, and three EVAL-Flex tasks, and may combine rather than isolate generalization factors.Transfer covers new assets or relations, skill transfer or composition, and three deformable interactions without deformable-task demonstrations; those deformable tasks do not isolate compliance as a causal factor.Source: §B.3 · pp. 35–36

B.4. Visual and Language Robustness Protocols

• Fresh training-distribution samples measure in-distribution performance, while held-out combinations or expanded visual ranges test specified shifts separately from evaluation-only physical tasks.The language-only test changes instructions while holding environment, target values, reset seeds, and success predicate fixed; paired outcomes prevent universal failure from appearing language-robust.Source: §B.4 · p. 36

B.5. Matched Physical-Transfer Protocol

• Physical-transfer analysis pairs simulation and real instances only when physical counterparts exist, and direct transfer uses the same checkpoint without target-task real demonstrations or finetuning.Recorded real-robot rollouts are qualitative, not a scored physical evaluation; same-task real transfer and Transfer-16 generalization are independent claims.Source: §B.5 · pp. 36–37

B.6. Additional Task-Capability Examples

• Capability examples include articulated motion, numerical and spatial goals, intermediate-state requirements, and overlapping task categories rather than mutually exclusive labels.The catalog includes 17 primitive, 18 functional, and 15 contact-rich tasks, alongside 31 single-stage, seven repetitive, and 12 long-horizon tasks.Source: §B.6 · p. 37

C.1. Model, Task Population, and Observations; C.3. Context and Interaction Budgets

• GPT-6 receives images, measured robot state, task guidance, and a 100-counted-call final budget, but the 660-trial batch retains earlier runs with differing settings.The ten slots per each of 66 tasks reflect API-cost and evaluation-time budgets, yielding coarser per-task estimates than trained-policy evaluations.Source: §C.1 · pp. 37–38, §C.3 · p. 38

C.2. The move_eef Palm-and-Finger Interface

• move_eef commands palm poses and absolute finger-joint targets, with omitted targets held at measured or last-commanded values and no task-specific grasp primitive.Its read-only planner schedules arm and finger paths but does not guarantee measured velocity, collision clearance, or target arrival; each schedule is limited to 300 actions.Source: §C.2 · p. 38

C.4. Scoring, Verification, and Comparability

• The reported GPT-6 comparison is descriptive because its inputs, task information, trial counts, query timing, and call budgets differ from trained policies.The comparison does not identify an architectural effect or equate external pretraining, task information, or inference compute across model families.Source: §C.4 · p. 39

C.5. Aggregation and Uncertainty

• Suite scores are unweighted means of task success rates, and bootstrap intervals quantify specified rollout variation rather than task-selection, model-version, or protocol-setting uncertainty.Each task has ten GPT-6 trials, and an observed 0/10 does not imply zero underlying success probability.Source: §C.5 · p. 39

C.6. Per-Task GPT-6 Results

• Each GPT-6 task result uses ten trials, with budget stops counted in the denominator as unsuccessful outcomes.The per-task tables identify Cap as the number stopped by the applicable call budget, and shade positive success entries.Source: §C.6 · p. 40, Table 14 · pp. 40–41
• A measured 0/10 is not evidence that the true success probability is zero.The Lift Carton and Open Latched Door qualifications apply to the reported task results.Source: §C.6 · p. 40

D. Core-50 Results and Coverage

• The trained-policy comparison uses 45,000 episodes across all 50 Core-50 tasks, while GPT-6’s ten trials per task are reported separately rather than pooled into trained-policy aggregates.The eight multi-task checkpoints contribute 100 episodes per task, and single-task DP contributes 5,000 episodes from task-specific checkpoints.Source: D. Core-50 Results and Coverage · p. 41
• Single-task DP is a descriptive reference because 38 earlier evaluations use the earlier native goal sampler, unlike newer runs using frozen training-frame-weighted quotas.Ten tasks use completed retraining with corrected near-degenerate normalization, and Lift Carton is excluded from the 11-task matched comparison.Source: D. Core-50 Results and Coverage · p. 42
• The two bootstrap intervals exclude training-run and checkpoint-selection uncertainty; the task-and-episode interval additionally resamples tasks from the same catalog.The episode-only interval assumes independent episodes with the 50 tasks fixed.Source: D. Core-50 Results and Coverage · p. 42
• Table 15 reports Core-50 task-macro success across all 50 tasks with 95% bootstrap intervals over episodes and over tasks and episodes; GPT-6 uses ten trials per task.The intervals describe sampling uncertainty, not equivalence of model inputs or query budgets.Source: Table 15 · p. 42

E. Single-Task Demonstration Scaling

• The trajectory-size comparison uses one frozen nested subset draw and fixed 20,000-update training runs, with held-out trajectories excluded at every size.Small-data runs use normalization statistics computed from the full training pool and therefore have access to larger-pool summary statistics; fixed updates also imply different data-reuse rates.Source: E. Single-Task Demonstration Scaling · p. 44
• Each checkpoint is evaluated on 100 accepted, runtime-valid episodes with shared planned task-specific seeds, goals, and prompts, but shared planned draws do not certify identical complete physical states.Evaluation uses the native EEF controller, 30 Hz action execution, fixed prompt bank, and task-specific physical budgets.Source: E. Single-Task Demonstration Scaling · p. 44
• 6.99%, 22.89%, 30.88%, 64.67%, and 82.80% are Task 24 success rates reweighted by the frozen training-frame prior, in increasing data-size order.Task 43 has one goal and is unchanged by this reweighting.Source: E. Single-Task Demonstration Scaling · p. 44
• The intervals quantify approximate rollout-level uncertainty conditional on each checkpoint, not training-seed or subset-selection variance.There is only one training seed and one subset draw at each size; the curves describe the implemented closed-loop reset distribution.Source: E. Single-Task Demonstration Scaling · pp. 44–45

F. Qualitative Real-Robot Rollouts

• The videos show adapted policies establishing contact, carrying objects, and preserving object pose through hand transitions on a physical dexterous hand.The rollouts cover single-hand reach-and-place interactions and bimanual tasks requiring both hands to hold targets simultaneously.Source: F. Qualitative Real-Robot Rollouts · p. 45
• The recordings are unscored, cover few episodes on one platform, and report no success rate or matched-budget training comparison.A budget-controlled comparison is left as future work.Source: F. Qualitative Real-Robot Rollouts · p. 45

G. Task Profiles

• Table 18 profiles tasks with an identifier, name, screenshot, and instruction; interaction, task-structure, and asset labels are cross-referenced in Table 10.For training tasks, the displayed prompt is one canonical example; for Transfer-16, it is the configured task prompt.Source: G. Task Profiles · p. 46, Table 18 · p. 46
• Bold instruction text marks randomized task parameters, while visual randomization and meaning-preserving prompt variants provide separate forms of variation.Marked parameters can include objects, spatial relations, orientations, angles, durations, and action choices; the displayed values are one example configuration.Source: G. Task Profiles · p. 46
• Generalization profiles include precise end conditions, such as opening a randomized trash-bin lid to at least 75 degrees or keeping a mustard bottle upright for 0.5 seconds.Other examples include lid alignment, stable object transfer, and setting two knobs to specified angles.Source: G. Task Profiles · pp. 54–55
• Composition profiles specify ordered multi-step interactions, including transferring a glue stick between hands and inserting a charger plug before its cable connector.Other examples combine fruit placement with photographing a logo, bottle scrubbing with mug filling, or seasoning and stirring.Source: G. Task Profiles · pp. 46, 54, 55
• A deformable-task profile requires compressing sponge foam, lifting it edge-on through a mug rim, then releasing both objects while leaving the mug upright and stable.Source: G. Task Profiles · p. 55