Source-linked AI summary
GenDexGrasp: Generalizable Dexterous Grasping
Puhao Li, Tengyu Liu, Yuyang Li, Yiran Geng, Yixin Zhu, Yaodong Yang, Siyuan Huang
TL;DR
Prior grasping methods often focus on specific robot hands and struggle to rapidly generate diverse, successful grasps for unseen hands. GenDexGrasp uses hand-agnostic contact maps and MultiDex to generate grasps, achieving a reported balance of quality, diversity, and speed.
Problem
Prior grasping methods often focus on specific robot hands and struggle to rapidly generate diverse grasps with high success rates for unseen hands.
Source: §I · p. 1Method
GenDexGrasp uses hand-agnostic contact maps, an aligned distance for contact representation, and MultiDex, a force-closure-optimized dataset spanning five hands and 58 household objects.
Source: §I · pp. 1–2, B. Contact Map Synthesis · p. 4Results
GenDexGrasp achieves a reported three-way trade-off among grasp quality, diversity, and speed, with a slightly lower success rate than DFC and UniGrasp top-1 but diverse grasps generated quickly.
Source: §V · p. 5, §VI · p. 6Takeaways & Limitations
GenDexGrasp is reported to generalize grasp generation to unseen hands while balancing quality, diversity, and inference time.
Source: §I · p. 1, §VI · p. 6
Abstract
from arXiv · showhide
Generating dexterous grasping has been a long-standing and challenging robotic task. Despite recent progress, existing methods primarily suffer from two issues. First, most prior arts focus on a specific type of robot hand, lacking the generalizable capability of handling unseen ones. Second, prior arts oftentimes fail to rapidly generate diverse grasps with a high success rate. To jointly tackle these challenges with a unified solution, we propose GenDexGrasp, a novel hand-agnostic grasping algorithm for generalizable grasping. GenDexGrasp is trained on our proposed large-scale multi-hand grasping dataset MultiDex synthesized with force closure optimization. By leveraging the contact map as a hand-agnostic intermediate representation, GenDexGrasp efficiently generates diverse and plausible grasping poses with a high success rate and can transfer among diverse multi-fingered robotic hands. Compared with previous methods, GenDexGrasp achieves a three-way trade-off among success rate, inference speed, and diversity. Code is available at https://github.com/tengyu-liu/GenDexGrasp.
I. INTRODUCTION
GenDexGrasp targets grasp generation for unseen hands while addressing slow inference and limited diversity in prior methods. It uses hand-agnostic contact maps, an aligned distance, and MultiDex to balance generalizability, speed, and diversity.
I. INTRODUCTION
- Prior methods often target particular hands, generate grasps slowly, or produce limited diversity, while the goal is rapid, diverse grasping for unseen hands.The paper defines generalizable dexterous grasping as generating grasping poses for unseen hands.Source: §I · p. 1
- Joint angles are unsuitable as a transferable grasp representation because similar hands can use different joint definitions.Prior work also uses contact points, contact maps, and approach vectors, but the paper identifies a need for a simple, effective representation.Source: §I · p. 1
- GenDexGrasp generates a hand-agnostic contact map with a conditional variational autoencoder, optimizes a hand pose to match it, and refines the pose in physics simulation.Random initialization and an efficient optimization scheme support diverse generation and faster inference.Source: §I · p. 1
- Aligned distance uses surface-normal direction to avoid labeling both sides of a thin shell as contacts when only one side is touched.The paper says this improves contact maps for grasp generation and helps address contact ambiguity during optimization.Source: §I · p. 1
- MultiDex contains 436,000 grasp poses for five hands and 58 household objects, synthesized with force-closure optimization.The paper presents it as a large-scale multi-hand dataset for learning hand-agnostic contact maps.Source: §I · p. 1
- The authors report a three-way trade-off among speed, diversity, and generalizability, with faster inference than hand-agnostic methods and greater diversity than hand-aware methods.They also report generalizability comparable to existing hand-agnostic methods.Source: §I · p. 2
II. RELATED WORK
Related work divides generalizable grasping into hand-aware and hand-agnostic approaches, with limitations in diversity or generation speed, respectively.
A. Generalizable Dexterous Grasping
- The paper contrasts hand-aware methods’ limited pose diversity with hand-agnostic methods’ slow grasp generation.Fig. 2 presents exemplar grasps for five hands across four objects.Source: A. Generalizable Dexterous Grasping · p. 2, Figure 2 · p. 2
- Hand-aware methods learn representations of hand structure and predict intermediate grasp goals, but train and evaluate encoders only on two- and three-finger grippers.Their final grasp solution is deterministic, yielding similar poses and limiting transfer to unseen grippers or scenarios.Source: A. Generalizable Dexterous Grasping · p. 2
- Hand-agnostic methods match sampled contacts or forces to hand poses, but stochastic sampling can take minutes to overcome local minima.The cited approaches include ContactGrasp, Liu et al., and Turpin et al..Source: A. Generalizable Dexterous Grasping · p. 2
B. Contact Map
The related-work review positions contact maps as useful grasp representations and identifies dataset diversity as a requirement for learning a versatile, hand-agnostic generator.
B. Contact Map
- GenDexGrasp treats contact maps as a transferable intermediate representation and uses a less restrictive map and more efficient optimization than prior approaches.Prior contact-map work uses them for retargeting, reconstruction, retrieval, affordances, grasp synthesis, or interaction policies.Source: B. Contact Map · p. 2
C. Grasp Datasets
- Human demonstration datasets are costly to collect at scale, and prior MoCap and contact-map collections largely capture pick-up-and-use settings and similar poses.The paper notes similar limitations in programmatically generated datasets.Source: C. Grasp Datasets · p. 3
III. DATASET COLLECTION
- A versatile hand-agnostic contact-map generator requires diverse grasp poses and corresponding maps across objects and robotic hands with varied morphologies.Source: §III · p. 3
A. Grasp Pose Synthesis
The grasp-synthesis process constructs grasps with force-closure, hand-prior, and penetration terms, then uses sampled contact points and poses to produce valid grasp data.
A. Grasp Pose Synthesis
- The dataset synthesis objective combines a differentiable force-closure estimator, hand-prior energy, and penetration energy.The hand pose is represented by global root-link rotation and translation plus revolute-joint angles.Source: A. Grasp Pose Synthesis · p. 3
- For thin-shell contacts, the figure contrasts Euclidean distances that confuse opposite-side contacts with aligned distances that distinguish noncontact regions.The aligned-distance illustration uses red and blue for shorter and longer distances, respectively.Source: Figure 3 · p. 3
- The differentiable force-closure estimator describes total wrench under equal forces at each contact point, neglecting friction forces.The paper characterizes it as a strong estimator of classical force closure.Source: A. Grasp Pose Synthesis · p. 3
- Contact points are sampled from predefined rectangular regions on each hand so a gradient-based optimizer can update their positions efficiently.DeepSDF approximates object signed distance and surface normal.Source: A. Grasp Pose Synthesis · p. 3
- MALA sampling produced 436,000 valid poses across five hands and 58 objects, using a 48/10 training/test object split and about 1,400 GPU hours.Source: A. Grasp Pose Synthesis · p. 3
- The pipeline collects multi-hand data with DFC, trains a CVAE to generate hand-agnostic maps for unseen objects, then optimizes poses for unseen hands.Source: Figure 4 · p. 4
B. Contact Map Synthesis
The section defines object-centric contact maps using distances from object surface points to the hand. Its aligned distance uses surface-normal direction to resolve thin-shell contact ambiguity.
B. Contact Map Synthesis
- A contact map is a set of normalized distances from each object surface point to the hand surface.The section computes this map from a given grasping pose.Source: B. Contact Map Synthesis · p. 4
- Aligned distance incorporates the object-surface normal, distinguishing contact on one side of a thin shell where Euclidean distance marks both sides.The passage says this reduces contact ambiguity for grasp generation.Source: B. Contact Map Synthesis · p. 4
- The contact value at an object surface point is 1 when it is in contact with the hand and 0 when it is far away.The passage states that the value lies in (0, 1].Source: B. Contact Map Synthesis · p. 4
IV. GENDEXGRASP
For an object and an arbitrary robotic hand with N joints, the goal is to generate a dexterous, diverse, and physically stable grasp pose.
IV. GENDEXGRASP
- The target grasp pose should be dexterous.The stated objective is for an arbitrary robotic hand with N joints and a given object.Source: §IV · p. 4
- The target grasp pose should be diverse.Diversity is one of the stated properties of the generated pose.Source: §IV · p. 4
- The target grasp pose should be physically stable.Physical stability is one of the stated properties of the generated pose.Source: §IV · p. 4
A. Generate Hand-Agnostic Contact Maps
GenDexGrasp generates an object-centric, hand-agnostic contact map and then fits a hand to it, rather than directly predicting a pose for an unseen hand. A CVAE learns the map distribution from object point clouds and contact maps.
A. Generate Hand-Agnostic Contact Maps
- For unseen hands, direct pose generation is challenging because observed hands are sparse and pose depends nonlinearly on hand geometry.The method instead generates a hand-agnostic contact map and fits the hand to that map.Source: A. Generate Hand-Agnostic Contact Maps · p. 4
- The CVAE encodes an object point cloud with pointwise contact values into a latent distribution, then decodes object-point features and a sampled code into a contact map.The encoder and decoder use PointNet and a shared-weight MLP; the approach adopts CVAE.Source: A. Generate Hand-Agnostic Contact Maps · p. 4
- The generative model is trained by maximizing the contact-map log likelihood through its equivalent ELBO, with a standard normal latent prior.The reconstruction loss approximates the ELBO expectation using expected and generated contact maps across examples.Source: A. Generate Hand-Agnostic Contact Maps · p. 4
- Because generated contact maps are empirically more ambiguous than ground-truth maps, the method sharpens the generated map.The passage introduces this sharpening step but does not specify its form in the supplied text.Source: A. Generate Hand-Agnostic Contact Maps · p. 4
B. Grasp Optimization
The grasp optimization fits a hand pose to a generated contact map while accounting for contact mismatch, penetration, and pose validity. It uses differentiable optimization with parallel starts to retain a strong result.
B. Grasp Optimization
- The hand pose is initialized by randomly rotating its root link and translating it toward the object from behind the palm.The translation distance is the radius of the object's minimum enclosing sphere.Source: B. Grasp Optimization · p. 5
- The objective combines contact-map mismatch with terms for hand-object penetration and pose validity.The contact mismatch is the MSE between the goal and current contact maps.Source: B. Grasp Optimization · p. 5
- Adam minimizes the differentiable objective by updating the hand pose, using 32 parallel optimizations to keep the best result and avoid bad local minima.Source: B. Grasp Optimization · p. 5
C. Implementation Details
The implementation trains the hand-agnostic contact-map CVAE for 36 epochs and separately configures the grasp optimizer.
C. Implementation Details
- The contact-map CVAE uses Adam with a learning rate of 1e-4 and default values for its other hyperparameters.Source: C. Implementation Details · p. 5
- CVAE training runs for 36 epochs and takes roughly 20 minutes on an NVIDIA 3090Ti GPU.Source: C. Implementation Details · p. 5
- The grasp optimizer uses an Adam learning rate of 5e-3.Source: C. Implementation Details · p. 5
V. EXPERIMENT
Experiments compare grasp success, diversity, and inference speed, including baseline comparisons, aligned-distance and generalization studies, qualitative examples, and tabletop scenarios.
Evaluation metrics
- Success is tested with external acceleration and a grasp fails if it fails any of six directional tests; diversity is joint-angle standard deviation among successful grasps.Inference speed measures the time for the entire pipeline.Source: §V · p. 5
Comparative experiments
- DFC has the best success rate and diversity but is overwhelmingly slow; GraspCVAE has low success and cannot generalize, while UniGrasp struggles to balance success and diversity.Source: §V · p. 5
- GenDexGrasp has a slightly lower success rate than DFC and UniGrasp top-1, while generating diverse grasps quickly for a three-way trade-off.The comparative experiments evaluate success rate, diversity, and inference speed.Source: §V · p. 5, Table I · p. 5
Ablation Study - Contact
- Euclidean-distance contact maps significantly lower success rate while slightly improving diversity across EZGripper, Barrett, and ShadowHand.The authors attribute this pattern to contact ambiguities, which add uncertainty counted as diversity by their metrics.Source: §V · p. 5, Table II · p. 6
Ablation Study - Generalization
- Out-of-domain tests train on the other four hands and show robustness across various hand structures.The study compares in-domain and out-of-domain versions for each selected hand.Source: §V · p. 6, Table III · p. 6
Qualitative results
- Generated grasps include wraps, pinches, tripods, quadpods, and hooks, while common failures include penetrations and floatations from imperfect optimization.The examples include unseen hands and objects; the caption lists Barrett, Allegro, and ShadowHand from top to bottom.Source: §V · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
Tabletop scenarios
- Tabletop-object use is demonstrated after proper training.Source: §V · p. 6, Figure 7 · p. 6
VI. CONCLUSION
GenDexGrasp targets grasp generation for unseen hands using contact maps and aligned distance. The paper reports a balance among quality, diversity, and speed, and contributes the MultiDex dataset.
VI. CONCLUSION
- GenDexGrasp uses contact maps as an intermediate representation and aligned distance to generate diverse, high-quality grasps in reasonable inference time for unseen hands.Source: §VI · p. 6
- Quantitative experiments suggest GenDexGrasp properly balances grasp quality, diversity, and speed.Source: §VI · p. 6
- MultiDex is a synthetic dataset with diverse grasp poses, household objects, and five robotic hands with varied kinematic structures.Source: §VI · p. 6