Source-linked AI summary
DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo
Hanwen Wang, Weizhi Zhao, Xiangyu Wang, Siyuan Huang, He Lin, Boyuan Zheng, Rongtao Xu, Gang Wang, Yao Mu, He Wang, Lue Fan, Hongsheng Li, Zhaoxiang Zhang, Tieniu Tan
TL;DR
Dexterous manipulation lacks standardized benchmarks that test capabilities beyond parallel-gripper tasks. DexJoCo introduces a functionally grounded benchmark and toolkit, and its evaluation reveals limited robustness and recurring failures in fine-grained, insertion, and memory-demanding actions.
Problem
Dexterous manipulation lacks standardized benchmarks that evaluate fine-grained, contact-rich interactions and capabilities beyond manipulator-gripper tasks.
Method
DexJoCo provides a task-oriented benchmark and toolkit spanning fine-grained manipulation, tool use, bimanual coordination, long-horizon execution, and reasoning.
Results
Visual randomization sharply reduces success across nearly all policies, while policies commonly fail at fine-grained actions, insertion, and temporally extended manipulation.
Takeaways & Limitations
DexJoCo exposes robustness, precision, temporal-memory, and bimanual coordination as persistent challenges for current dexterous manipulation policies.
Takeaways & Limitations
Existing vision-action policies lack dexterous-hand-centric representations and contact-force cues needed for expressive, precise contact-rich manipulation.
Abstract
from arXiv · showhide
Achieving human-level manipulation requires dexterous robotic hands capable of complex object interactions. Advancing such capabilities further demands standardized benchmarks for systematic evaluation. However, existing dexterous benchmarks lack tasks that reflect the unique manipulation capabilities of dexterous hands over parallel grippers, as well as comprehensive evaluation pipelines. In this paper, we present DexJoCo, a benchmark and toolkit for task-oriented dexterous manipulation, comprising 11 functionally grounded tasks that evaluate tool-use, bimanual coordination, long-horizon execution, and reasoning. We develop a low-cost data collection system and collect 1.1K trajectories across these tasks, with support for domain randomization to assess robustness. We benchmark modern models under diverse settings, including visual and dynamics randomization, multi-task training, and action-head adaptation. Through extensive empirical analysis, we identify several important insights and common limitations of current policies in dexterous manipulation, highlighting key challenges for future research in dexterous hand robot learning. Project page available at: https://dexjoco.github.io
1 Introduction
DexJoCo addresses the lack of a standardized benchmark for dexterous hand manipulation by introducing functionally grounded tasks and an accessible data-collection toolkit. It also provides 1.1K human demonstration trajectories and evaluates modern policies.
- Motivation: Existing robot-learning systems and datasets primarily focus on manipulator-gripper platforms, despite human-level manipulation requiring fine-grained, contact-rich dexterous-hand interactions.
- Motivation: DexJoCo targets the lack of standardized dexterous-manipulation evaluation caused by differing environmental setups and robot configurations across laboratories.
- Contributions: DexJoCo introduces functionally grounded tasks evaluating fine-grained manipulation, tool use, bimanual coordination, long-horizon execution, and reasoning capabilities.
- Contributions: DexJoCo develops a low-cost teleoperation toolkit with retargeting for efficient demonstration collection and gathers 1.1K human demonstration trajectories in simulation while evaluating modern policies.
2 Related Works
Related work contrasts mature, low-cost trajectory collection for manipulator–gripper systems with the greater benchmarking and data-collection challenges posed by dexterous hands. These differences motivate dedicated dexterous-hand benchmarks and specialized collection pipelines.
- Dexterous Manipulation Benchmark: Low degrees of freedom enable large-scale trajectory collection for manipulator–gripper robots at low cost or through automated procedures.The passage attributes this feasibility to the robots’ relatively low degrees of freedom.
- Dexterous Manipulation Benchmark: Dedicated benchmarks are needed for manipulator–hand systems to support human-level manipulation.The passage distinguishes these systems from manipulator–gripper robots and notes that existing dexterous-hand benchmarks are primarily designed for an unspecified purpose in the supplied excerpt.
- Dexterous Hand Trajectory Collection: Manipulator–gripper trajectory collection is technically mature because recording actions requires only the end-effector’s target 6D pose and a typically single-degree-of-freedom gripper.This setup eliminates the need for specialized hardware.
- Dexterous Hand Trajectory Collection: Dexterous-hand trajectory collection is considerably more challenging because these systems have high degrees of freedom.The supplied passage ends after introducing this challenge and does not provide further specifics.
3 DexJoCo Benchmark and Toolkit
DexJoCo is a MuJoCo-based benchmark and toolkit that integrates task construction, low-cost human demonstration collection, policy training, and evaluation. Its tasks encode functional success constraints and support dexterity-focused evaluation under varied visual and physical conditions.
- Benchmark and Toolkit: DexJoCo integrates task environments, demonstration collection tools, policy-training interfaces, and evaluation utilities into a complete pipeline.The pipeline imports 3D assets into MuJoCo, defines structured success conditions, records teleoperated position-control actions, and supports replay-based visual augmentation.
- Robot Setup and Observation State: The simulated robot combines a Rethink Robotics mount, Franka Panda manipulator, and Allegro Hand, with third-person and wrist-mounted RGB observations.DexJoCo is built on MuJoCo to provide accurate physics modeling and rich perceptual observations.
- Teleoperation System: Approximately $2,300 USD is required for the low-cost teleoperation hardware, which uses Rokoko Smartgloves and HTC Vive trackers for hand and wrist motion capture.GeoRT retargets human fingertip keypoints to Allegro joint positions without paired human-robot annotations, while tracker alignment controls the Franka end effector.
- Task Design: Tasks are formulated as interactive objects O and functional constraints G covering temporal sequence, object pose, articulated joint state, and contact conditions.These constraints define structured success criteria rather than simple object relocation.
- Domain Randomization: Domain randomization varies object placement, table height, camera poses, illumination direction and color, and tabletop textures across task scenarios.The same trajectories can be replayed with visual randomization to broaden the evaluation distribution efficiently.
- Policy Evaluation: Four policy families are benchmarked: ACT, Diffusion Policy, π0.5, and GR00T N1.5, with pretrained language-conditioned models adapted for bimanual action heads.ACT and Diffusion Policy train from scratch on vision and proprioception, whereas π0.5 and GR00T N1.5 use LoRA fine-tuning with flow matching and language conditioning.
4 Experiments
Experiments show that DexJoCo remains challenging under visual and task variation, exposing trade-offs among pre-training, model scale, and architecture. Analyses also reveal failures in fine-grained interaction, insertion, multi-task learning, action-head initialization, and language generalization.
- Failure Modes: 550 evaluation trials identify button interaction, insertion, and squeezing as recurring failure modes despite successful object pickup or grasping.The failures suggest policies perceive objects but overlook interactive elements, while insertion steps frequently fail.
- Baseline and Robustness: Visual randomization causes success rates to drop sharply across nearly all policies, while π0.5 leads overall and DP-T remains competitive on bimanual tasks.Some policies never succeed on difficult bimanual tasks; DP-T is approximately 100M parameters and trained from scratch.
- Multi-task Training: Joint multi-task training degrades DP-T on every task, whereas π0.5 improves on Click Mouse and Pinch Tongs but has a lower average success rate.The comparison uses the same number of training steps as single-task training.
- Dynamics Randomization: Under randomized friction, stiffness, and object mass, π0.5 achieves a higher average success rate than DP-T.The result supports using the simulated benchmark to capture performance trends under varying dynamics, despite sim-to-real gaps.
- Action-head Adaptation: Retaining pretrained action-head weights produces higher success rates on most tasks and a better average than fully reinitializing the action head.This compares partial pretrain-AH with the rand-AH condition.
- Language Generalization: π0.5 fails to generalize language conditioning beyond trained single-digit passwords, defaulting instead to a fixed action bias.Evaluation includes seen digits, arithmetic expressions, and English-word inputs.
5 Discussion
The discussion identifies a lack of dexterous-hand-centric foundation models as a key limitation: gripper-based pretraining creates an action-space mismatch, while current action heads poorly capture dexterous joint coupling.
- Limitations: Current VLA models are primarily pretrained on gripper-based data, creating an action-space mismatch for dexterous hands and limiting expressivity and transfer.Their action heads fail to capture high-dimensional joint coupling, motivating embodiment-aware representations and hand-centric pretraining.
Appendix · A Statistical Analysis for Language Generalization Results
The appendix analyzes language-generalization results and finds severe mode collapse: apparent success on some unseen prompts reflects a nearly constant tendency to output “2,” rather than language conditioning. This failure extends to the seen digit “4,” for which precision also breaks down.
- A Statistical Analysis for Language Generalization Results: The policy exhibits severe mode collapse in the language-generalization experiment.This behavior is analyzed in the appendix’s statistical analysis.
- A Statistical Analysis for Language Generalization Results: 30.0%±5.3 precision on the unseen prompt “two” appears moderate but is described as a statistical illusion.The illusion is attributed to the model’s prior bias.
- A Statistical Analysis for Language Generalization Results: 24.7%±10.3 precision on the unseen prompt “1+1” likewise appears moderate but reflects prior bias rather than language conditioning.The passage identifies this result as part of the same statistical illusion.
- A Statistical Analysis for Language Generalization Results: The probability of outputting “2” remains nearly constant at 30% even when the correct answer is “1” or “4”.This output distribution is presented as evidence of a lack of language conditioning.
- A Statistical Analysis for Language Generalization Results: The analysis further evidences weak language conditioning through failure on the seen digit “4”.The supplied passage states that precision fails on this seen digit, but its continuation is truncated.
- A Statistical Analysis for Language Generalization Results: Figure 7 shows the output distribution of π0.5 across instructions on the Unlock iPad.The policy was trained on single digits 1-5.
- A Statistical Analysis for Language Generalization Results: Table 4 provides detailed language instructions for the language-generalization experiment.The supplied material includes the table caption but no further table contents.
B Visualization and Language Instruction of DexJoCo Tasks
This section presents visualization and language instruction for DexJoCo tasks. The material is organized across Table 5 and its continuation.
- Table 5 covers visualization and language instruction of DexJoCo tasks.
- The visualization and language-instruction table continues in a subsequent section.
- A further continuation extends Table 5’s visualization and language-instruction content.
C Randomization Settings of DexJoCo Tasks
DexJoCo applies shared visual and table-height randomization across all 11 task environments. At reset, environments vary camera pose, tabletop texture, lighting, table height, and consistently shift task-relevant object heights.
- Visual randomization: Preset third-person camera poses form the replay-camera pool used for visual randomization.Figure 9 visualizes the preset camera-pose pool, and the protocol samples one pose per environment reset.
- Visual randomization: All 11 task environments share a visual randomization protocol that samples camera pose and tabletop texture while perturbing scene lighting at reset.Each environment samples one preset third-person camera pose from the replay-camera pool and randomly selects a tabletop texture from the texture library.
- Visual randomization: Light positions are perturbed in the x and y axes by U(−0.3, 0.3), while light directions are perturbed in the x and y axes by U(−0.4, 0.4).The protocol also samples light diffuse RGB values from U(0.3, 0.8), headlight ambient RGB values from U(0.3, 0.7), and headlight diffuse RGB values from U(0.2, 0.6).
- Table-height randomization: All task environments share table-height randomization, with the table height offset sampled as ∆h ∼U(0, 0.05) m at reset.Task-relevant object heights are shifted consistently with the sampled table-height offset.