Source-linked AI summary
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, Tengyu Liu, Li Yi, He Wang
TL;DR
The paper addresses universal dexterous grasping from point-cloud observations, where high-dimensional hands make diverse pose generation and reliable execution difficult across seen and unseen object categories. It proposes a two-stage system with factorized grasp-pose generation and realistic-input goal-conditioned execution, supported by state canonicalization, curriculum learning, and teacher-student distillation. The integrated pipeline achieves more than 60% average success across thousands of object instances and outperforms all baselines, while the study is limited to rigid objects.
Problem
Universal dexterous grasping must generate diverse, high-quality grasps and execute them across seen and unseen categories from realistic observations, despite the difficulty caused by high-dimensional hand actuation.
Method
A two-stage pipeline generates diverse point-cloud-conditioned grasp poses with rotation factorization, then executes a selected goal pose using a goal-conditioned policy trained with state canonicalization, curriculum learning, and teacher-student distillation.
Results
More than 60% average success rate is achieved on thousands of object instances, with the full pipeline outperforming all baselines.
Takeaways & Limitations
The pipeline demonstrates universal dexterous grasping under a realistic robotic setting and provides a large benchmark spanning 5,519 instances and 133 object categories.
Takeaways & Limitations
The study tackles only rigid-object grasping; functional dexterous grasping and articulated objects such as scissors remain outside its scope.
Abstract
from arXiv · showhide
In this work, we tackle the problem of learning universal robotic dexterous grasping from a point cloud observation under a table-top setting. The goal is to grasp and lift up objects in high-quality and diverse ways and generalize across hundreds of categories and even the unseen. Inspired by successful pipelines used in parallel gripper grasping, we split the task into two stages: 1) grasp proposal (pose) generation and 2) goal-conditioned grasp execution. For the first stage, we propose a novel probabilistic model of grasp pose conditioned on the point cloud observation that factorizes rotation from translation and articulation. Trained on our synthesized large-scale dexterous grasp dataset, this model enables us to sample diverse and high-quality dexterous grasp poses for the object point cloud.For the second stage, we propose to replace the motion planning used in parallel gripper grasping with a goal-conditioned grasp policy, due to the complexity involved in dexterous grasping execution. Note that it is very challenging to learn this highly generalizable grasp policy that only takes realistic inputs without oracle states. We thus propose several important innovations, including state canonicalization, object curriculum, and teacher-student distillation. Integrating the two stages, our final pipeline becomes the first to achieve universal generalization for dexterous grasping, demonstrating an average success rate of more than 60\% on thousands of object instances, which significantly outperforms all baselines, meanwhile showing only a minimal generalization gap.
1. Introduction
The paper addresses universal dexterous grasping from realistic point-cloud and proprioception inputs, where high hand dimensionality complicates pose generation and execution. It proposes a two-stage pipeline combining diverse grasp proposals with goal-conditioned execution, achieving over 60% success across thousands of objects.
- Motivation: Dexterous grasping supports diverse functional manipulation but is harder than parallel-gripper grasping because a ShadowHand has 26 DoF versus 7 DoF for a typical parallel gripper.The higher-dimensional actuation space complicates both valid grasp-pose generation and execution-trajectory planning.
- Problem Setting: The target is universal dexterous grasping that generalizes across hundreds of seen and unseen object categories using depth observations and robot proprioception.The benchmark contains more than one million grasps for 5,519 object instances spanning 133 categories.
- Approach: The pipeline decomposes grasping into diverse point-cloud-conditioned pose generation and goal-conditioned execution that produces physically correct trajectories.The predicted grasp goal pose from the first stage conditions the execution policy.
- Approach: The proposal model factorizes hand rotation from translation and articulation to generate diverse grasp poses from point-cloud observations.It is trained on a synthesized large-scale table-top dexterous grasp dataset to address the limited diversity of prior approaches.
- Approach: The execution stage learns a goal-conditioned policy from realistic point-cloud and proprioception inputs instead of relying on oracle states.The paper addresses policy-learning difficulty with innovations including teacher-student learning, while the introduction specifically describes the oracle-teacher framework.
- Results: More than 60% average success rate is achieved across thousands of object instances, outperforming all baselines while maintaining high proposal diversity and grasp quality.The proposal stage is reported as the only method with high diversity and the highest grasping quality.
2. Related Work
Prior work includes synthetic and learning-based dexterous grasp synthesis, but learning methods struggle to achieve grasp quality and diversity simultaneously. Dexterous execution also remains difficult because analytical planning often relies on simplifying hand and object geometries.
- Dexterous Grasp Synthesis: Dexterous grasp synthesis generates valid grasping poses from object meshes or point clouds and includes both non-learning and learning-based approaches.Non-learning methods commonly create synthetic datasets, while learning-based methods predict grasps from those datasets.
- Dexterous Grasp Synthesis: GraspIt! synthesizes stable grasps through collision detection, while optimization-based methods improve grasp diversity using differentiable force-closure estimation.These methods support the construction of larger or higher-quality synthetic dexterous grasp datasets.
- Dexterous Grasp Synthesis: Learning-based grasp-synthesis methods often struggle to achieve high grasp quality and diversity simultaneously.The paper addresses this tension with a conditional generative model that decouples rotation from translation and articulation.
- Dexterous Grasp Execution: Analytical dexterous-grasping approaches optimize control trajectories from hand-object kinematics and dynamics but typically simplify geometries to keep planning tractable.Reinforcement and imitation learning have more recently shown promise for dexterous grasping.
3. Method
UniDexGrasp uses a two-stage pipeline: it samples dexterous grasp proposals from point clouds, then executes a selected goal pose with a vision-based policy using point clouds and proprioception. Proposal generation factorizes rotation from translation and articulation, while execution training adds canonicalization, curriculum learning, and teacher-student distillation.
- Method Overview: The pipeline first samples a grasp proposal g=(R, t, q), then uses it as the goal pose for goal-conditioned grasp execution.R, t, and q represent hand root rotation, translation, and joint angles.
- Goal-Conditioned Dexterous Grasping Policy: The execution policy receives the goal pose, scene point cloud, and robot proprioception at each time step, and outputs an action.It is designed to operate across seen and unseen object categories.
- Grasp Proposal Generation: The grasp distribution is factorized as p(R|X0)·p(t, q|X0, R), with GraspIPDF generating rotations and GraspGlow generating translation and joint angles.The sampled pose is subsequently optimized with ContactNet to improve physical plausibility.
- Grasp Proposal Generation: GraspIPDF models the conditional distribution of hand-root rotation from the object point cloud and supports sampling rotations on SO(3).It extracts geometric features with PointNet++ and predicts an unnormalized joint log probability density.
- Grasp Proposal Generation: Canonicalization transforms the point cloud as ˜X0=R−1X0, simplifying translation-and-articulation modeling to p(˜t,q|˜X0) with ˜t=R−1t.Glow models this conditional distribution over Euclidean spaces, and test-time samples are reconstructed into grasp poses.
- End-to-End Training with ContactNet: ContactNet predicts object-point contact heatmaps from canonicalized object and hand point clouds, while additional penetration and contact losses train proposal generation end-to-end.The joint objective combines the negative log-likelihood with contact-map, object-penetration, table-penetration, and self-penetration terms.
4. Experimentals
Experiments evaluate synthetic grasp data, grasp-goal generation, policy execution, and language-guided selection. UniDexGrasp produces diverse proposals, improves cross-category success over baselines, and supports instruction-conditioned grasping.
- 4.1. Data Generation and Statistics: 1.12 million valid grasps cover 5519 object instances across 133 categories, split into training, seen-category unseen-instance, and unseen-category sets.The dataset is synthesized through randomized object drops, grasp optimization, table-penetration penalties, and simulation filtering.
- 4.2. Results on Grasp Proposal Generation: CVAE collapses to a single grasp mode for a fixed point cloud, whereas GraspGlow produces diverse proposals.The diversity comparison fixes rotation and examines variation in translation and articulation.
- 4.3. Grasp Execution: 74% train-set and 69% test-set success rates for the teacher policy exceed PPO’s 14% and DAPG’s 13% train-set averages.The reported teacher gains are approximately 49% and 47% over ILAD on the train and test sets, respectively.
- 4.4. Language-guided Dexterous Grasping: Around 90% accuracy is achieved for bottle and hammer categories after 10 minutes of fine-tuning with CLIP-based proposal filtering.The filtering selects rendered grasp proposals with the highest image-text similarity to instructions such as grasping a hammer by its handle.
5. Conclusions and Discussions
The paper presents a two-stage pipeline for universal dexterous grasping in a realistic setting, while limiting its scope to rigid objects and leaving functional grasping open.
- 5. Conclusions and Discussions: The proposed pipeline combines grasp proposal generation with goal-conditioned grasp execution for universal dexterous grasping over thousands of objects.The authors describe this as the first demonstration under a realistic robotic setting and note potential for real-world transfer.
- 5. Conclusions and Discussions: The study addresses rigid objects only, excluding articulated objects such as scissors.Functional dexterous grasping remains identified as a challenging direction for future work.
A. Dataset Generation
The dataset synthesizes physically plausible, diverse tabletop dexterous grasps for thousands of objects, then filters them for stability and limited penetration.
- Object Dataset: The object dataset includes 5519 instances across 133 categories for universal dexterous grasping.The dataset was designed to provide numerous object instances and diverse grasping labels.
- Grasp Generation: Objects are randomly rotated, dropped onto a tabletop plane, and paired with an optimized dexterous gripper pose.The optimization saves object pose and gripper translation, rotation, and joint angles for validation.
- Energy Function: The energy function combines force-closure, contact attraction, penetration repulsion, table clearance, joint-limit, and self-penetration terms.These terms adapt prior grasp-synthesis energy design to the tabletop setting.
- Grasp Validation: 1.12 million valid grasps were harvested after testing stability in six axis-aligned directions and rejecting penetration deeper than 1mm.The generation process ran 1000 trials per object instance.
B.1.1 Details about Our Method
The method models dexterous grasp proposals probabilistically from point clouds, separating rotation-related modeling from other grasp parameters and refining contact consistency.
- Grasp Proposal Generation: GraspIPDF maps a point cloud and candidate rotation to an unnormalized joint log probability using PointNet++ features.The model follows an IPDF-style probabilistic formulation over SO(3).
- Grasp Proposal Generation: GraspGlow uses Glow-style normalizing flows to transform a single grasp-parameter vector conditioned on PointNet features.Each coupling layer uses actnorm, 1 × 1 convolution, and affine coupling components.
- Grasp Proposal Generation: The implementation composes 21 Glow blocks with residual MLP coupling networks.The hidden dimension is 64, with ReLU activations, batch normalization, and dropout probability 0.5.
- Contact Refinement: ContactNet predicts a 10-bin discretized per-point contact map from canonicalized object and hand point clouds.Hand features are broadcast and concatenated with object per-point features before MLP prediction.
- Baseline: GraspTTA combines a CVAE-based grasp sampler with ContactNet-guided test-time adaptation, recalculating the target contact map only initially in this pipeline.The baseline samples hand rotation, translation, and parameters before optimizing toward predicted contacts.
B.2.1 Details about Our Method
The execution method trains a goal-conditioned dexterous grasp policy with structured rewards and representation-learning techniques, alongside motion-planning and learning baselines.
- Our Method: The method jointly updates PointNet object representations with behavior cloning during RL policy training.It adds state canonicalization, object curriculum learning, and joint object-classification training.
- Our Method: The reward combines goal-pose, reaching, lifting, and object-moving terms.The goal term measures joint, hand-root translation, and rotation differences, while other terms encourage contact and lifting.
- Our Method: The reaching reward penalizes Euclidean distances between each robot finger and the object.Its purpose is to encourage the fingers to approach the object.
- Our Method: The lifting reward activates when the hand satisfies the lifting condition and rewards upward force at the hand root.Otherwise, the lifting component contributes zero.
- Our Method: Object Curriculum Learning trains progressively from individual objects toward categories and broader category sets.The supplied configurations use one-, two-, and three-stage curricula.
- Baselines: The baselines include CEM-based motion planning, PPO, DAPG, ILAD, and IBS-Grasp.DAPG uses demonstrations, ILAD adds geometric representation learning, and IBS-Grasp uses SAC with an interaction-based state representation.
C. Experiment Details
The experiments define teacher and student states using robot, object, and grasp information, and configure a 24-actuator hand with tabletop RGBD sensing.
- State Definition: The teacher state includes robot state, object oracle state, and a pre-sampled object point cloud, while the student uses robot state, scene point cloud, and grasp goal.The object oracle state contains pose, linear velocity, and angular velocity.
- Action Space: The action space contains 24 normalized motor commands: six for global hand position and orientation and 18 for the fingers.Actions are normalized to (-1,1) according to actuator specifications.
- State Definition: The robot-state definition is documented separately in Table 5.The supplied passage identifies the table but does not enumerate its fields.
- Camera Setup: Five RGBD cameras observe the tabletop, with four positioned around it and one above it.Their focus point is the table-centered location [0, 0, 0.05].
C.2. Training Details
Training uses Adam-based optimization for the proposal models and PPO followed by DAgger for the grasping policy, with separate hyperparameter tables for end-to-end training, adaptation, and policy learning.
- GraspIPDF and GraspGlow are trained with Adam using a learning rate of 10^-3.
- GraspGlow training uses batch sizes of 64 in its first stage and 32 in its second stage, with eight samples per object.
- GraspIPDF trains for 24 hours, while GraspGlow requires 160k first-stage and 8k second-stage iterations and about one day overall.
- The execution policy πE is learned with PPO and distilled into πS using DAgger.
- Large object meshes are filtered, leaving 3200 training instances and 241 test instances for grasp-execution experiments.
C.3. Metric Details
The evaluation separates grasp-proposal quality and diversity from execution success and goal following, using metrics defined for both generated poses and policy behavior.
- Grasp proposal metrics: Proposal evaluation uses seven grasp-quality metrics and two diversity metrics.
- Grasp proposal metrics: Mean Q1 measures the minimal wrench needed to destabilize a grasp, with relaxed contact and invalid-penetration handling for vision-based predictions.
- Grasp proposal metrics: Rotation standard deviation measures diversity around the chordal L2 mean of rotation samples from GraspIPDF.
- Grasp proposal metrics: Conditional translation and joint-angle standard deviations measure diversity given a sampled rotation from GraspIPDF.
- Grasp proposal metrics: Log-likelihood evaluates how well the model fits ground-truth grasp distributions using p(R,t,θ|X) = p(R|X) · p(t,θ|R^-1X).
- Policy metrics: Policy evaluation measures grasping success and adherence to the specified goal, including simulation success rate and mean joint position error.
- Policy metrics: Simulation success requires the object to finish within 0.05m of a target positioned 0.3m above its initial location.
D.1. Goal-conditioned Dexterous Grasping Policy Results
The policy achieves strong goal-conditioned and non-goal-conditioned grasping behavior, while qualitative results show diverse proposals and corresponding executions across varied objects.
- Quantitative results: In non-goal-conditioned settings, the teacher policy has the highest success rate across training and all testing datasets.
- Quantitative results: Among reinforcement-learning methods, UniDexGrasp achieves the lowest MPE except for motion planning, while motion planning has the lowest success rate.
- Quantitative results: Generated proposals can be noisy on unseen categories, making direct motion planning to the goal position unreliable for firm grasping.
- Qualitative results: GraspIPDF produces diverse rotation predictions, while GraspGlow produces diverse translation and articulation predictions for the same objects.
- Quantitative results: UniDexGrasp can modify noisy grasp goals to make grasping possible, unlike motion planning’s low-MPE behavior that cannot close the hand-object gap.
- Qualitative results: Qualitative demonstrations cover bottle, camera, toy dog, and headphone categories, pairing generated goal poses with policy-generated grasping sequences.