Source-linked AI summary
DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, He Wang
TL;DR
Dexterous grasping is under-explored partly because large-scale datasets are scarce, despite its importance for human-like manipulation. DexGraspNet uses an accelerated differentiable force-closure synthesis pipeline to build a large, validated ShadowHand dataset, and cross-dataset experiments report improved quality and diversity. The dataset does not cover every grasp type, especially precision and functional grasps.
Problem
Dexterous grasping lacks the large-scale datasets needed by learning-based methods, although it is an important step toward dexterous manipulation.
Method
DexGraspNet uses an efficient optimization-based synthesis pipeline with reverse penetration energy to robustly process low-quality thin meshes.
Results
Training grasp synthesis methods on DexGraspNet consistently improves grasp quality and diversity across baselines, test sets, and metrics relative to DDGdata.
Takeaways & Limitations
DexGraspNet provides scale, quality, and diversity intended to support future dexterous grasping research.
Takeaways & Limitations
The dataset is dominated by contact-rich power grasps and lacks semantic guidance for precision and functional grasps.
Abstract
from arXiv · showhide
Robotic dexterous grasping is the first step to enable human-like dexterous object manipulation and thus a crucial robotic technology. However, dexterous grasping is much more under-explored than object grasping with parallel grippers, partially due to the lack of a large-scale dataset. In this work, we present a large-scale robotic dexterous grasp dataset, DexGraspNet, generated by our proposed highly efficient synthesis method that can be generally applied to any dexterous hand. Our method leverages a deeply accelerated differentiable force closure estimator and thus can efficiently and robustly synthesize stable and diverse grasps on a large scale. We choose ShadowHand and generate 1.32 million grasps for 5355 objects, covering more than 133 object categories and containing more than 200 diverse grasps for each object instance, with all grasps having been validated by the Isaac Gym simulator. Compared to the previous dataset from Liu et al. generated by GraspIt!, our dataset has not only more objects and grasps, but also higher diversity and quality. Via performing cross-dataset experiments, we show that training several algorithms of dexterous grasp synthesis on our dataset significantly outperforms training on the previous one. To access our data and code, including code for human and Allegro grasp synthesis, please visit our project page: https://pku-epic.github.io/DexGraspNet/.
I. INTRODUCTION
Dexterous hands enable more varied manipulation than parallel grippers, but progress is constrained by the scarcity of large-scale, diverse grasp datasets. DexGraspNet addresses this gap with an efficient synthesis pipeline and a large, stable dataset that improves downstream grasp synthesis.
- 26-DoF ShadowHand enables more diverse grasp types and complex manipulation than low-DoF parallel-jaw grippers.Examples include using scissors, solving a Rubik’s cube, and task-specific grasping.
- 6.9K grasps on 565 objects made DDG the only available dexterous grasp dataset, motivating substantially larger and more diverse data.The passages state that high-DoF hands require larger and more diverse datasets for generalization.
- 1.32 million grasps on 5355 objects give DexGraspNet more than 200 diverse grasps per object across more than 133 hand-scale categories.The dataset uses ShadowHand and combines objects from synthetic and scanned datasets.
- 400 to 7 GPU hours reduces the time required to synthesize 10000 valid grasps through improved initialization, contact selection, and penetration-energy computation.The alternative penetration-energy method also supports lower-quality object meshes.
- Cross-dataset experiments show that models trained on DexGraspNet achieve better grasping quality and higher diversity than models trained on DDGdata.The experiments train two dexterous grasping algorithms on both datasets.
II. RELATED WORK
Dexterous grasping research has evolved from analytical force-closure optimization toward methods that simplify search and optimization for complex, high-dimensional hands.
- Dexterous hands require complex grasp descriptors, with ShadowHand representations reaching up to 24 DoF.Suction cups and parallel-jaw grippers can use descriptors of at most 7 dimensions.
- Early analytical methods optimized grasp poses for force closure, which resists external forces and torques.
- Differentiable force-closure estimation enables diverse grasp synthesis for arbitrary hands by simplifying the optimization process.
B. Data-Driven Grasping
Data-driven dexterous grasping methods generate poses directly or through intermediate representations, but their diversity remains bounded by training data and existing datasets.
- Direct data-driven methods generate grasping-hand pose vectors for an object and commonly refine them to remove penetration.
- Indirect methods generate contact points, contact maps, or occupancy fields before obtaining poses through optimization, planning, reinforcement learning, or generative models.
- Data-driven methods improve inference speed and grasp-pose diversity over most analytical methods, but training data still limits diversity.
- Programmatically synthesized training poses commonly use GraspIt!, whose low-dimensional eigengrasp search and finger squeezing produce firm grasps.
- Human-pose datasets use image, RGBD, or contact sensing, but remain restricted to humanoid hand structures and daily hand poses.
III. DATASET GENERATION METHOD
DexGraspNet constructs dexterous grasps from diverse object models using parameterized hand poses, sampled contact points, differentiable force-closure energies, and stochastic optimization.
- Object Collection: 3980 categorized CAD models and 1375 uncategorized scanned models provide the object collection for dataset generation.The models come from ShapeNet-Core, ShapeNetSem, YCB, BigBIRD, Grasp, KIT, and Google’s scanned Object Dataset.
- Grasp Parameterization: Grasps are parameterized by global translation, rotation, and 22 ShadowHand joint angles, with four sampled contact points added during optimization.The augmented tuple is g′ = (T, R, θ, x); contact points are intermediate variables discarded after generation.
- Energy-Based Synthesis: The method optimizes a total energy combining differentiable force closure, attraction, penetration, and a natural-hand prior.The force-closure term encourages contact points to form force closure, while attraction and repulsion energies ensure contact and prevent penetration.
- Optimization: A modified MALA algorithm updates hand pose and joint variables while periodically resampling contact points, accepting moves stochastically under Metropolis-Hastings.Optimization starts from a randomly initialized hand pose and ends after 10000 steps.
3) Our Method:
The improved synthesis pipeline addresses low yield, mesh-quality failures, and unnatural poses through guided initialization, reverse penetration energy, and additional regularization.
- Motivation: The original method faced low success rates, slow convergence, zero penetration energy for thin meshes, and twisted poses from random initialization.These limitations motivated changes targeting efficiency, effectiveness, and robustness.
- Initialization: Opening the fingers, orienting the palm toward the object, and sampling around an inflated convex hull improves initialization diversity and grasp naturalness.The canonical hand pose is jittered within joint limits, while sampled object-surface directions guide hand placement and orientation.
- Penetration Handling: Reverse penetration energy computes object-point distances to the hand mesh, enabling synthesis on thin meshes without requiring object thickness.This expands processing to many more low-quality CAD models.
- Regularization: The final energy adds penalties for self-penetration and joint-limit violations to the force-closure, distance, and penetration terms.The resulting objective is Efc+wdisEdis+wpenEpen+wspenEspen+wjointsEjoints.
- Implementation: A modified Kaolin distance implementation removes category-level DeepSDF pretraining and significantly reduces optimization memory costs.This change simplifies signed-distance computation during grasp optimization.
C. Grasp Validation
The pipeline physically validates generated grasps in Isaac Gym and retains only grasps that remain stable under gravity across all six axis-aligned directions.
- Simulation Validation: Generated grasps are filtered after optimization through physical simulation in Isaac Gym using PhysX.The simulator initializes the gripper from the optimized grasp parameters before testing stability.
- Validation Distribution: Figure 3 displays the distribution of object counts by average post-validation success rate, retaining only successful grasps.The figure summarizes validation outcomes at the object level.
- Success Criterion: A grasp succeeds if contact is maintained for 100 simulation steps under gravity applied along all six axis-aligned directions.Gravity has magnitude 9.8m/s2, and active forces are applied by slightly moving contacting links along their contact normals.
IV. DATASET ANALYSIS AND COMPARISON
DexGraspNet provides large-scale, diverse grasps and is compared with DDGdata in terms of generation efficiency, diversity, and stability. The dataset contains 1.32 million grasps, while evaluations report greater diversity and higher average Q1 than DDGdata.
- Dataset scale: 1.32 million grasps across more than 200 grasps per object form the largest ShadowHand grasping dataset.The pipeline also generates grasps for other dexterous hands, including MANO and Allegro.
- Generation efficiency: 74 minutes optimize 10000 grasps with an 18% valid rate, compared with 37 minutes for 512 grasps and a 3% valid rate for the original algorithm.Generating 1.32 million valid grasps took 950 GPU hours, versus an estimated 50000 GPU hours with the original algorithm.
- Diversity: DexGraspNet exhibits broader finger-pose coverage than DDGdata, whose finger poses collapse into a fan-shaped space and whose joint angles cluster near limits.The comparison uses visualizations and probability distributions of joint angles over grasp poses.
- Stability: DexGraspNet has higher average Q1 than DDGdata, indicating stronger measured grasp stability under the reported metric.Q1 is described as the norm of the smallest wrench that can destabilize the grasp, and the pipeline does not explicitly optimize these metrics.
V. BENCHMARKS
The paper benchmarks DDG and GraspTTA on DexGraspNet and compares them with the same methods trained on DDGdata.
- DDG and GraspTTA are evaluated on DexGraspNet against versions of the same methods trained on DDGdata.
A. Benchmark Methods
The benchmark methods use different strategies to synthesize dexterous grasps from object observations. DDG directly regresses hand pose and joint angles, while GraspTTA adapts sampled grasps using predicted contact consistency.
- DDG: DDG predicts the ShadowHand’s 6D pose and joint angles from five depth images using a differentiable generalized Q1 metric.Training is divided into two stages to ease learning.
- GraspTTA: GraspTTA combines a CVAE for grasp sampling with a contact network that predicts target object contact regions.The method optimizes hand parameters during testing to reduce the difference between current and target contact maps.
B. Experiments and Results
Experiments evaluate grasp validity, quality, penetration, and diversity, showing consistent advantages for models trained on DexGraspNet while exposing a quality–diversity trade-off.
- DexGraspNet-trained models score higher than DDG-trained models across every tested baseline, test set, and metric.The reported metrics include Isaac Gym simulation success rate, mean Q1, and maximal penetration depth.
- Joint angle entropy is consistently higher with DexGraspNet training, while its standard deviations are lower than those from DDG training.This indicates improved grasp diversity and more consistent entropy across results.
- DDG cannot fully recover DexGraspNet’s grasp diversity, whereas GraspTTA can exceed DexGraspNet’s entropy but has very low success rate.The comparison highlights that entropy alone does not establish useful diverse grasping when stability is poor.
- None of the existing grasping methods simultaneously learns highly diverse grasp poses and maintains a reasonable success rate.The paper attributes this status quo to a trade-off between stability and diversity.
VI. LIMITATIONS
DexGraspNet does not cover every grasping type: its optimization favors contact-rich power grasps, while precision and functional grasps remain insufficiently represented.
- DexGraspNet cannot cover every grasping type in the referenced taxonomy.The limitation concerns the breadth of grasp types represented in the dataset.
- Optimization pulls candidate points toward objects, causing final grasps to be contact-rich power grasps and making precision grasps rare.The authors identify precision grasps as important for representing dexterous multi-finger hand capabilities.
- The method lacks semantic guidance, making functional grasps such as lifting a mug by its handle difficult to generate.The authors leave precision and functional grasps as important issues for future exploration.