PaperlayerView exact revision on arXiv

Source-linked AI summary

GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training

Adithyavairavan Murali, Balakumar Sundaralingam, Yu-Wei Chao, Wentao Yuan, Jun Yamada, Mark Carlson, Fabio Ramos, Stan Birchfield, Dieter Fox, Clemens Eppner

arXiv:2507.13097v1cs.ROcs.AI

TL;DR

In the FetchBench (Han et al., 2024) benchmark, state-of-the-art grasping systems achieve sub-20% accuracies. GraspGen combines an object-conditioned diffusion grasp generator with an efficient discriminator trained using grasps sampled by the generator. GraspGen outperforms baseline methods and achieves state-of-the-art results on FetchBench grasping in clutter, validated across multiple objects and three gripper embodiments.

  • Problem

    In the FetchBench (Han et al., 2024) benchmark, state-of-the-art grasping systems achieve sub-20% accuracies.

    Source: §1 · p. 1

  • Method

    GraspGen combines an object-conditioned diffusion grasp generator with an efficient discriminator trained using grasps sampled by the generator.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    GraspGen outperforms baseline methods and achieves state-of-the-art results on FetchBench grasping in clutter, validated across multiple objects and three gripper embodiments.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors present GraspGen as a foundation for future downstream applications.

    Source: §5 · p. 9

    Performance depends on depth sensing and instance segmentation, and the authors report difficulty predicting grasps for cuboids in practice.

    Source: §5 · p. 9

Abstract

from arXiv · show

Grasping is a fundamental robot skill, yet despite significant research advancements, learning-based 6-DOF grasping approaches are still not turnkey and struggle to generalize across different embodiments and in-the-wild settings. We build upon the recent success on modeling the object-centric grasp generation process as an iterative diffusion process. Our proposed framework, GraspGen, consists of a DiffusionTransformer architecture that enhances grasp generation, paired with an efficient discriminator to score and filter sampled grasps. We introduce a novel and performant on-generator training recipe for the discriminator. To scale GraspGen to both objects and grippers, we release a new simulated dataset consisting of over 53 million grasps. We demonstrate that GraspGen outperforms prior methods in simulations with singulated objects across different grippers, achieves state-of-the-art performance on the FetchBench grasping benchmark, and performs well on a real robot with noisy visual observations.

1. Introduction

Learning-based grasping systems still face generalization and practicality gaps across unknown objects, clutter, and different gripper morphologies. GraspGen pairs a diffusion generator with a discriminator and presents an object-centric approach intended to scale across grippers and settings.

1. Introduction

  • FetchBench systems achieve below 20% accuracy, while reported grasp-model failures also affect in-the-wild manipulation systems.OK-Robot reported an 8% error rate from grasp model failures alone; Robo-ABC evaluated as a RAM baseline had below 50% success.Source: §1 · p. 1
  • Existing grasping systems remain difficult to make turnkey, while model-based approaches require precise object poses and multi-view methods can be impractical in clutter.Contact point-based architectures also often generalize poorly across gripper morphologies, particularly beyond symmetric parallel-jaw grippers.Source: §1 · pp. 1–2
  • Prior clutter-grasping methods often require whole-scene simulation or manual real-world data collection, and still rely on instance segmentation for target-driven grasping.The paper argues that recent instance-segmentation advances, including SAM2, make object-centric models worth revisiting for simpler training and inference.Source: §1 · p. 2
  • GraspGen combines a diffusion-based grasp generator with an efficient discriminator and introduces on-generator discriminator training to address false positives from generator errors.The paper also releases a dataset of 53 million grasps and reports flexibility across three gripper types, partial and complete point clouds, single objects and clutter, and simulation and real settings.Source: §1 · p. 2

2. Related Work

Related work frames 6-DOF grasping as grasp sampling and grasp analysis, with generative models commonly paired with discriminators for scoring. GraspGen uses object-centric point clouds and aims to improve both stages.

2. Related Work

  • 6-DOF grasping is commonly divided into grasp sampling and grasp analysis, with generated poses scored and ranked by discriminators in many approaches.Some prior methods use a single model for both stages, while approaches also differ in their point-cloud, implicit, or voxelized inputs.Source: §2 · p. 2
  • GraspGen uses object-centric point-cloud input and supports both single-view and multi-view observation setups.The paper links this design to instance-segmentation advances that can avoid scene modeling during training.Source: §2 · pp. 2–3
  • Closest prior diffusion-grasping work formulated 6-DOF antipodal grasping for known objects, then extended it to dexterous grasping from point clouds with a discriminator.GraspGen positions its contribution as a new multi-gripper dataset and improvements to both grasp analysis and grasp sampling.Source: §2 · p. 3

3. GraspGen

GraspGen generates object-conditioned 6-DOF grasps with diffusion, then scores samples using a separately trained discriminator. Its design combines point-cloud encoding, on-generator discriminator training, and simulated grasp datasets for multiple grippers.

3. GraspGen

  • Grasp generation targets a spatially diverse set of successful poses, which are scored and ranked to return top-K grasps for execution.Diversity matters in clutter because otherwise successful grasps may be unreachable or collide and be filtered by the motion planner.Source: §3 · p. 3

3.1. Grasp Generation with Diffusion

  • The generator models grasp poses as a conditional DDPM in SE(3), factorized into SO(3) and R3, with object shape encoded from a point cloud.The paper chooses DDPM over energy-based score-matching Langevin sampling, describing it as faster to compute and simpler to implement.Source: §3.1 · p. 3
  • PointTransformerV3 encodes object point clouds by serializing them before transformer processing, avoiding a nearest-neighbor ball-query bottleneck.Grasp translations are normalized using a multiplier computed from dataset statistics rather than chosen arbitrarily or by grid search.Source: §3.1 · pp. 3–4
  • Inference samples noise and iteratively denoises grasps in 10 steps, with separate denoising processes for translation and rotation.Training uses a denoising loss on predicted versus actual noise; discriminator scoring is trained separately and is not part of diffusion-model training.Source: §3.1 · p. 4

3.2. Grasp Evaluation with On-Generator Training

  • On-generator training addresses the mismatch between offline grasp data and the generator’s sampled grasps, including slightly colliding poses and distant low-likelihood outliers.The recipe generates samples on training objects, simulates them for success labels, then trains the discriminator on the annotated dataset.Source: §3.2 · pp. 4–5
  • The discriminator reuses the generator’s frozen object encoder and concatenates its embedding with the grasp pose before an MLP predicts a sigmoid success score.Only the final MLP layer is trained from scratch using binary cross-entropy.Source: §3.2 · p. 5

3.3. Dataset

  • The simulated dataset provides 6D gripper transforms and binary success labels for Franka Panda, Robotiq-2f-140, and vacuum grippers.It uses 36,366 Objaverse meshes, with a random 8,515-mesh subset to match ACRONYM’s size; each gripper comprises approximately 17 million grasps.Source: §3.3 · p. 5

4. Experimental Evaluation

GraspGen outperforms prior methods on FetchBench, Franka-ACRONYM simulation, and real-robot evaluation, with reported gains across multiple grippers. On-Generator discriminator training and the tested model components are also evaluated through distribution-shift analysis and ablations.

4.1. Simulation Results

  • On Franka-ACRONYM, GraspGen outperformed baselines by over 48% in Precision-Coverage AUC.The reported evaluation uses complete object point clouds and compares grasp success against coverage of the positive grasp set.Source: §4.1 · p. 6, Figure 3 · p. 6
  • On FetchBench, GraspGen achieves state-of-the-art results, surpassing Contact-GraspNet and M2T2 by 16.9% and 7.8%, respectively; experiments used a Franka Panda in 100 scenes with 60 tasks per scene, totaling 6,000 grasp executions.Source: §4.1 · p. 7

4.2. Analysis of On-Generator Training

  • On-Generator-only training achieved the highest discriminator performance; offline-dataset training had 6.5% lower AUC.The offline and generated grasp datasets showed substantial distribution shift, especially for negative grasps.Source: §4.2 · pp. 7–8, Figure 6 · p. 7

4.3. Ablation Studies

  • The discriminator was 6.7% higher in AUC and used 21× less memory than the compared state-of-the-art architectures.The comparison is to discriminator architectures from Mousavian et al. (2019).Source: §4.3 · p. 8
  • Translation-normalization performance had a convex relationship with its multiplier; the dataset-statistics equation gave a reliable local minimum, while rotation representations performed comparably.For the Franka gripper, the reported multiplier was κ=3.27.Source: §4.3 · p. 8, Figure 7 · p. 8
  • PointTransformerV3 reduced translation error by 5.3 mm and increased recall by 4% over PointNet++.The paper reports that all tested rotation representations performed comparably.Source: §4.3 · p. 8

4.4. Performance on Multiple Grippers

  • Across grippers, GraspGen outperformed M2T2 by 37% in Franka simulation, 44% in Robotiq simulation, and 57% in real-robot experiments.It also outperformed SE3-Diff across all three grippers.Source: §4.4 · p. 8

4.5. Real Robot Evaluation

  • GraspGen achieved an overall real-robot success rate of 81.3%, outperforming M2T2 and AnyGrasp by 28% and 17.6%, respectively.Performance was evaluated on isolated objects, table clutter, baskets, and shelves; shelf and basket settings were more challenging.Source: §4.5 · p. 9, Figure 8 · p. 9

5. Conclusion & Limitations

GraspGen is a 6-DOF grasp-generation framework validated across multiple objects and three gripper embodiments, and it achieves state-of-the-art results on FetchBench grasping in clutter. Its performance depends on depth sensing and instance segmentation, and it struggled with cuboids in practice.

5. Conclusion & Limitations

  • GraspGen achieves state-of-the-art results on FetchBench grasping in clutter and is validated across multiple objects and three gripper embodiments.The authors hope it provides a foundation for future downstream applications.Source: §5 · p. 9
  • Performance depends on depth-sensing and instance-segmentation quality; the authors also report difficulty predicting cuboid grasps and approximately 3K GPU hours for data generation and training.The authors believe more box-like training data may address the cuboid difficulty.Source: §5 · p. 9

6. Appendix

The appendix provides supplementary material on grasp predictions, dataset statistics, evaluations, and ablations. It includes qualitative comparisons of GraspGen with M2T2 and points to a video of real-robot execution.

6. Appendix

  • Supplementary material covers dataset statistics, evaluations, ablations, and other analyses, and the project video shows examples of real-robot execution.The listed topics include FetchBench clutter evaluations, sim-to-real noise augmentation, and guidance on tuning the number of inferred grasps.Source: §6 · p. 15

6.1. Qualitative Visualizations of Grasp Predictions

  • GraspGen’s qualitative predictions are more focused on target objects and provide greater coverage than M2T2, which sometimes misses small targets and generates environmental false positives.The appendix also shows additional predictions on segmented real-object point clouds.Source: §6.1 · p. 15

6.2. Further Dataset Statistics

  • The appendix identifies Table 3 as a detailed comparison of the GraspGen dataset with prior grasping datasets.The supplied passage does not report the comparison’s values.Source: §6.2 · p. 15

6.3. Quantitative Metrics in Radar Chart in Sec. 1

For object-centric simulation, the key metric is the area under the Precision-Coverage curve; for real-robot and FetchBench experiments, the paper reports grasp success rates. On Robotiq-2F-140, GraspGen achieves almost double M2T2’s AUC.

6.3. Quantitative Metrics in Radar Chart in Sec. 1

  • Object-centric simulation uses Precision-Coverage AUC, while FetchBench and real-robot evaluations report grasp success rates.For suction, the paper reports coverage comparisons between GraspGen and SE3-Diff because it did not train an M2T2 model.Source: §6.3 · p. 15

6.4. Baseline Comparisons for Robotiq-2F-140

  • M2T2’s contact-point formulation is designed for symmetric, non-adaptive grippers, so it does not directly transfer to adaptive Robotiq-2F-140.For real-world experiments, the authors reused the Franka Panda M2T2 model with a fixed translation offset of -10 cm along the z axis.Source: §6.4 · p. 15
  • In real-robot qualitative results, GraspGen’s predictions are more focused on the target object and have greater coverage than M2T2’s.Table 2 presents examples from shelf, basket, tabletop, and isolated-object settings.Source: Table 2 · p. 16
  • GraspGen achieves almost double M2T2’s AUC on Robotiq-2F-140, and outperforms both baselines by a substantial margin.The comparison is summarized in Figure 9.Source: §6.4 · p. 15, Figure 9 · p. 17

6.5. Baseline comparisons for Suction

The appendix compares suction-gripper performance and reports ablations on datasets, the discriminator, rotation representation, and point-cloud encoders. The discriminator and PointTransformerV3 results show improvements in classification efficiency and grasp-generation metrics, respectively.

6.5. Baseline comparisons for Suction

  • Suction grasps show large rotation error because of symmetry and slightly higher L2 translation error despite substantially larger coverage than SE3-Diff.Across embodiments, learning difficulty ranks Franka, suction, then Robotiq, from hardest to easiest.Source: §6.5 · p. 17

6.6.1. Ablation on Dataset

  • The ACRONYM-trained model performs slightly better than the GraspGen-Franka-trained model, including on the latter’s test set.The authors hypothesize the difference reflects simulator and shape-dataset mismatches: Flex versus PhysX, and ShapeNet versus Objaverse.Source: §6.6.1 · pp. 17–18, Figure 10 · p. 18

6.6.2. Analysis of the Discriminator

  • GraspGen’s discriminator is 6.7% higher in AUC and 5.87% higher in mAP, while using 21× less memory than the discriminator in (42).Figure 11 shows the discriminator comparison.Source: Figure 11 · p. 18, §6.6.2 · p. 18

6.6.3. Ablation on Rotation representation

  • The 6D rotation representation and Euler angles perform comparably, suggesting normalization is key for diffusion learning on large grasp datasets.The paper uses the Lie algebra representation for its other experiments.Source: §6.6.3 · pp. 18–19, Table 4 · p. 19

6.6.4. Ablation on Pointcloud Encoder

  • PointTransformerV3 reduces translation error by 5.3 mm and increases recall by 4% compared with the tested PointNet++ backbone.The authors hypothesize the performance gap may widen with larger-scale data.Source: Table 5 · p. 19, §6.6.4 · p. 19

6.7. Further Details of FetchBench Experiments

The FetchBench reimplementation uses the ground-truth scene collision mesh to compare grasp-generation methods independently of motion planners, with specified planner configurations for the oracle and Contact-GraspNet.

6.7. Further Details of FetchBench Experiments

  • The experiments use the ground-truth collision mesh of the scene to compare grasp-generation methods independently of motion planners.The reimplementation is in Isaac Sim.Source: §6.7 · p. 19
  • The oracle planner uses “FetchMeshCurobo.”This is one of the two named FetchBench configurations.Source: §6.7 · p. 19
  • Contact-GraspNet uses “FetchMeshCuroboPtdCGNBeta” and makes one grasping attempt without retrials.Source: §6.7 · p. 19

6.8. Computing Earth-Movers-Distance (EMD) in Fig. 6

The EMD procedure compares generated and training grasp distributions for the same object using pairwise grasp distances, one-to-one matching, and repeated subsampling.

6.8. Computing Earth-Movers-Distance (EMD) in Fig. 6

  • EMD measures distribution shift between diffusion-generated grasps and the training dataset for the same object.Source: §6.8 · p. 19
  • The procedure subsamples 500 grasps from each dataset and computes pairwise distances using the cost function introduced in (62).Source: §6.8 · p. 19
  • A Linear Sum Assignment finds the lowest-distance one-to-one matching, repeated over five random subsamples and averaged into a score for each object.Source: §6.8 · p. 20

6.9. Data Augmentation for Sim2Real transfer

The paper uses point-cloud noise and augmentation for sim-to-real transfer, including simulated instance-segmentation errors to make the grasp model robust to boundary outliers.

6.9. Data Augmentation for Sim2Real transfer

  • The authors report that noise and data augmentation were crucial for sim-to-real transfer.Source: §6.9 · p. 20
  • Training randomizations include point-cloud orientation after mean centering, camera viewpoints, point subsampling, and instance-segmentation error.Source: §6.9 · p. 20
  • Segmentation boundary overshoot can create 3D outliers that attract high-confidence grasps and risk collisions; dilation-generated outliers are used in training, and the augmented model is shown to be robust.Figure 13 contrasts predictions without and with segmentation-noise augmentation; the latter ignores outlier points.Source: §6.9 · p. 20, Figure 13 · p. 20

6.10. Inference Parameter Tuning

Inference performance depends on the grasp-quality threshold and the number of diffusion-sampled grasps: a high threshold calls for a larger batch, while very low thresholds hurt precision.

6.10. Inference Parameter Tuning

Loading 2507.13097v1…