Source-linked AI summary
UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data
Sizhe Yang, Yiman Xie, Zhixuan Liang, Yang Tian, Jia Zeng, Dahua Lin, Jiangmiao Pang
TL;DR
Bimanual dexterous grasping lacks diverse, physically plausible data and remains difficult across multiple grasp strategies. UltraDexGrasp combines optimization-based synthesis with planning-based demonstration generation, builds a 20-million-frame dataset, and trains a point-cloud policy. The policy achieves 84.0% average simulation success and 81.2% average real-world success with zero-shot sim-to-real transfer.
Problem
Bimanual universal dexterous grasping is underexplored, with data generation challenged by limited diversity, open-loop synthesis, dynamic scenarios, and arm-kinematic constraints.
Method
UltraDexGrasp integrates optimization-based grasp synthesis with planning-based demonstration generation, then trains a point-cloud policy using unidirectional attention to predict control commands.
Results
81.2% average success rate was achieved in real-world universal dexterous grasping after training exclusively on synthetic data.
Takeaways & Limitations
The policy demonstrates robust zero-shot sim-to-real transfer and adapts grasp strategies to novel objects with varied shapes, sizes, and weights.
Abstract
from arXiv · showhide
Grasping is a fundamental capability for robots to interact with the physical world. Humans, equipped with two hands, autonomously select appropriate grasp strategies based on the shape, size, and weight of objects, enabling robust grasping and subsequent manipulation. In contrast, current robotic grasping remains limited, particularly in multi-strategy settings. Although substantial efforts have targeted parallel-gripper and single-hand grasping, dexterous grasping for bimanual robots remains underexplored, with data being a primary bottleneck. Achieving physically plausible and geometrically conforming grasps that can withstand external wrenches poses significant challenges. To address these issues, we introduce UltraDexGrasp, a framework for universal dexterous grasping with bimanual robots. The proposed data-generation pipeline integrates optimization-based grasp synthesis with planning-based demonstration generation, yielding high-quality and diverse trajectories across multiple grasp strategies. With this framework, we curate UltraDexGrasp-20M, a large-scale, multi-strategy grasp dataset comprising 20 million frames across 1,000 objects. Based on UltraDexGrasp-20M, we further develop a simple yet effective grasp policy that takes point clouds as input, aggregates scene features via unidirectional attention, and predicts control commands. Trained exclusively on synthetic data, the policy achieves robust zero-shot sim-to-real transfer and consistently succeeds on novel objects with varied shapes, sizes, and weights, attaining an average success rate of 81.2% in real-world universal dexterous grasping. To facilitate future research on grasping with bimanual robots, we open-source the data generation pipeline at https://github.com/InternRobotics/UltraDexGrasp.
I. INTRODUCTION
UltraDexGrasp addresses the data and algorithm bottlenecks limiting universal dexterous grasping for bimanual robots by combining grasp synthesis with demonstration planning. Its synthetic dataset and policy support multiple grasp strategies and strong simulation and real-world performance across diverse objects.
- Motivation: Universal bimanual dexterous grasping remains limited because existing methods struggle with data diversity, dynamic scenarios, and arm kinematics.RL experts tend to produce repetitive postures, while optimization- and learning-based synthesis are largely open-loop and often ignore arm kinematics.
- Framework: UltraDexGrasp integrates optimization-based grasp synthesis with planning-based demonstration generation for coordinated dual-arm manipulation.The integration is designed to produce kinematically feasible, natural closed-loop motions while preserving data diversity.
- Dataset: UltraDexGrasp-20M contains 20 million frames across 1,000 objects and supports bimanual, whole-hand, pinch, and tripod grasp strategies.The dataset is presented as the first large-scale multi-strategy dexterous grasp dataset for bimanual robots.
- Policy: The grasp policy uses point clouds, unidirectional attention, and predicted control commands to enable multiple strategies and generalization across diverse objects.It is trained on the synthetic dataset and evaluated on objects varying in shape, weight, and size.
- Results: 84.0% average success was achieved in simulation, while real-world deployment reached an 81.2% success rate across diverse objects.The simulation evaluation covered 600 objects, and the policy transferred from synthetic training to real-world grasping without real-world training data.
III. PRELIMINARIES
The preliminaries define bimanual grasp poses and model grasping through contact forces, torques, and the resulting grasp wrench space. The formulation uses hard-finger point-on-plane contacts and requires the wrench space to resist external wrenches.
- A. Definition of Grasp Pose for Bimanual Robots: A bimanual grasp pose specifies each hand’s translation, rotation, and joint positions.The hand index is h ∈ {0, 1}.
- B. Basics of Grasp Modeling: The contact model uses point-on-plane contacts and hard fingers, with forces constrained by the friction cone.The normal and tangential force components are defined separately.
- B. Basics of Grasp Modeling: Each contact applies a wrench consisting of force and torque, represented through a contact-specific wrench basis matrix.The torque depends on the contact’s relative position to the object’s center of mass.
- B. Basics of Grasp Modeling: The grasp wrench space is the set of wrenches achievable from all contact points and should be large enough to resist external wrenches.For k contacts, the space aggregates feasible contact forces through the grasp maps.
IV. UNIVERSAL DEXTEROUS GRASP DATASET
The dataset pipeline initializes randomized simulation scenes with 1,000 objects, then synthesizes, filters, and ranks feasible grasps. This process selects high-quality grasp candidates for subsequent use.
- Scene Initialization: The pipeline imports object assets and the robot URDF into a simulation environment during scene initialization.The object assets are selected from DexGraspNet.
- Scene Initialization: 1,000 distinct objects are used, while camera pose and joint impedance are randomized to reduce the sim-to-real gap.The simulation receives table and object meshes and poses before grasp synthesis.
- Grasp Synthesis: A grasp synthesizer generates batches of feasible grasps that are filtered and ranked to select the best candidates.Synthesis is conditioned on the simulated table and object scene.
A. Grasp Synthesis
Grasp synthesis initializes strategy-specific hand configurations on object convex hulls, then optimizes physically plausible, geometrically conforming grasps and selects reachable, collision-free candidates requiring minimal motion.
- Strategy-specific initialization samples one or two convex-hull surface points and positions the hands along surface normals facing the object.
- The formulation optimizes bimanual hand poses and contact forces using distance, hand–object collision, and inter-hand penetration energies.
- A lower-level quadratic program optimizes contact forces while the upper level updates hand poses to reduce target-wrench errors.
- Different grasp strategies use distinct dexterous-hand contact points within a unified optimization program.
- 500 candidate grasps per object are physically validated, checked for bimanual reachability and collisions, then ranked using SE(3) distance.
- Selecting grasps requiring minimal motion improves grasping efficiency and produces more natural, smooth robot movements.
B. Demonstration Generation
Demonstration generation converts each preferred grasp into a staged bimanual trajectory and uses an imaged point cloud during rendering to reduce the sim-to-real gap.
- Each demonstration comprises pregrasp, grasp, squeeze, and lift stages, with approach offset by 0.1 m and lifting by 0.2 m.
- Bimanual motion planning generates collision-free coordinated trajectories for executing the grasping stages.
- An additional imaged point cloud supplements the robot point cloud in simulation and can be generated from known real-world joint positions.
A. Overall Architecture
The universal grasp policy encodes scene point clouds, aggregates their features with decoder-only attention, and decodes action commands for multiple grasp strategies. The experiments evaluate its generalization, comparisons, data scaling, design components, and real-world performance.
- Overall Architecture: The policy encodes input scene point clouds into point features and processes them with a decoder-only transformer.
- Overall Architecture: The architecture is designed to support multiple grasp strategies and improve generalization across diverse objects without redundant structures or auxiliary tasks.
- Overall Architecture: Learnable action query tokens integrate scene information through unidirectional attention before an MLP transforms action latents into executable action vectors.
- Experimental Questions: Experiments assess universal grasping and object generalization, comparisons with prior and closed-loop methods, data scaling, component effects, and real-world performance.
A. Simulation Experiments
Simulation evaluates the policy on 600 seen and unseen objects spanning substantial variation in shape, weight, and size, using the dual-arm, dual-hand robot setup.
- Experimental Setup: The simulation system uses two 6-DoF UR5e robots and two 12-DoF XHands in a dual-arm, dual-hand configuration.
- Experimental Setup: 600 test objects include categories seen and unseen during training and vary substantially in shape, weight, and size.
- Experimental Setup: Object weights range from 5 g to 1,000 g, while bounding-box dimensions span from a longest edge below 0.03 m to a shortest edge above 0.5 m.
- Policy Architecture: The policy architecture takes point clouds as input, aggregates scene features through unidirectional attention, and predicts control commands for multiple strategies.
2) Baselines:
The policy is evaluated against DP3 and DexGraspNet on simulation benchmarks, while additional analyses examine data scaling and overall benchmark performance.
- DP3 is a diffusion policy using point clouds and robot state, whereas DexGraspNet uses complete object meshes and motion planning to generate trajectories.
- 84.0% average success rate was achieved by the proposed policy on the simulation benchmark, including small, medium, and large objects.
- 83.4% success rate was achieved on unseen objects, indicating generalization across objects not present during training.
- 37.3 percentage points was the proposed policy’s improvement over DP3 when both were trained on UltraDexGrasp-20M.
- 68.5% was the average success rate of grasping data generation, while learned-policy performance significantly surpassed it beyond 1M training frames.
- Both bounded Gaussian distribution prediction and unidirectional attention significantly improve performance in the reported ablation study.
6) Effectiveness of Design Choices:
The policy’s design choices are tested through ablations, while the real-world setup uses two bimanual robot systems with camera-based point-cloud input.
- 6) Effectiveness of Design Choices:: More than 10% improvement in success rate was obtained over both ablation baselines when the full policy retained bounded Gaussian prediction and unidirectional attention.
- 6) Effectiveness of Design Choices:: Removing bounded Gaussian distribution prediction or unidirectional attention defines the two ablated policy variants used for comparison.
- B. Real-World Experiments: Two UR5e robots, two 12-DoF XHands, and two Azure Kinect DK cameras form the real-world experimental setup.
- B. Real-World Experiments: 10 Hz is the control frequency of the robotic system, with tabletop cameras supplying scene point clouds as policy input.
- B. Real-World Experiments: Camera calibration, coordinate alignment, outlier removal, and joint impedance randomization are used for sim-to-real transfer.
2) Sim-to-Real Implementation Details:
Real-world evaluation tests the synthetic-data-trained policy and baselines on diverse objects using repeated trials and reports success rates.
- 2) Sim-to-Real Implementation Details:: 25 objects across small, medium, and large categories are tested, with 15 trials per object and different poses across trials.
- 2) Sim-to-Real Implementation Details:: 81.2% average success rate was achieved by the proposed policy in real-world evaluation, significantly outperforming the baselines.
- 2) Sim-to-Real Implementation Details:: The policy successfully transfers from synthetic training to real-world grasping of novel objects with varied shapes, sizes, and weights.
- 2) Sim-to-Real Implementation Details:: Object volumes range from 18 cm3 to 26,400 cm3, while weights range from 3.6 g to 1,095 g in the tested real-world set.
- 2) Sim-to-Real Implementation Details:: Three-finger tripod, whole-hand, and bimanual grasps are among the strategies used for these diverse real-world objects.