Source-linked AI summary
6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Arsalan Mousavian, Clemens Eppner, Dieter Fox
TL;DR
Robot grasp generation must handle diverse, stable 6-DOF poses from partial point-cloud observations when complete object models are unavailable. The paper uses a VAE to sample grasps and a learned evaluator to assess and refine them, achieving an 88% success rate on 17 unknown objects in real-robot experiments. The method is trained purely in simulation and transfers to the real world without extra steps.
Problem
Grasp selection requires choosing stable 3D positions and orientations from noisy partial depth observations, while complete object models are often unavailable.
Method
A VAE samples diverse 6-DOF grasps from partial point clouds, and a grasp evaluator predicts success and uses gradients to refine sampled poses.
Results
88% success rate was achieved for picking up 17 objects with unknown 3D models, with evaluation including real-robot experiments.
Takeaways & Limitations
The approach generates diverse grasp sets while maintaining high success rate and transfers from simulation to a real robot without real-world training data collection.
Takeaways & Limitations
The method samples latent values uniformly and removes grasps through collision checks and kinematic-feasibility filtering rather than directly modeling surrounding objects.
Abstract
from arXiv · showhide
Generating grasp poses is a crucial component for any robot object manipulation task. In this work, we formulate the problem of grasp generation as sampling a set of grasps using a variational autoencoder and assess and refine the sampled grasps using a grasp evaluator model. Both Grasp Sampler and Grasp Refinement networks take 3D point clouds observed by a depth camera as input. We evaluate our approach in simulation and real-world robot experiments. Our approach achieves 88\% success rate on various commonly used objects with diverse appearances, scales, and weights. Our model is trained purely in simulation and works in the real world without any extra steps. The video of our experiments can be found at: https://research.nvidia.com/publication/2019-10_6-DOF-GraspNet\%3A-Variational
1. Introduction
6-DOF GraspNet addresses grasp selection from partial depth-camera point clouds by generating diverse grasp candidates and improving their precision with learned evaluation. The approach is evaluated on a real robot and reports an 88% success rate across 17 unknown objects.
- Motivation: Grasp selection must account for object and gripper geometry, mass distribution, friction, reachability, and collisions, yet complete object models are often unavailable from noisy depth observations.These constraints motivate directly learning from partial point clouds rather than relying solely on complete 3D models.
- Approach: The framework combines a VAE that maps partial object point clouds to diverse grasp sets with an evaluator network that assesses and refines 6D gripper poses.The evaluator’s gradient can improve samples by moving the gripper out of collision or aligning it with the object.
- Approach: The VAE is designed to provide high coverage of functioning grasps while generating only a small number of failing grasps.Diversity matters because some generated grasps may be kinematically infeasible for the robot.
- Evaluation: The paper evaluates grasp sampling, evaluation, and refinement through real-robot experiments and hyperparameter ablations.Its organization includes comparison with related deep-learning grasping approaches and analysis in simulation.
2. Related Work
Prior grasping methods use visual learning, geometric sampling, reinforcement learning, and generative modeling, but many restrict pose diversity or depend on heuristics and incomplete scene information. 6-DOF GraspNet instead targets full 6-DOF grasp generation from partial point clouds with a VAE-based sampler and evaluator.
- 6-DOF Grasping: Many recent grasp-synthesis methods represent grasps as oriented image rectangles, restricting the gripper pose to 3 degrees of freedom and limiting diversity.The paper motivates predicting the full 6-DOF pregrasp pose instead.
- 6-DOF Grasping: Full 6-DOF grasp prediction is difficult because occluded object parts can affect grasp success.Related work addresses this through geometry reconstruction or learned grasp-score functions, often with simulation-only evaluation.
- Alternative Representations: Single-best-pose regression and coarse orientation classification inherently limit the diversity of predicted grasps.The cited classification approach uses 24 predefined orientations, while GPD samples candidates more densely from observed point clouds.
- Generative Approach: 6-DOF GraspNet uses a VAE to sample diverse grasps in SE(3), with an evaluation network accepting or rejecting proposals based on predicted success.Both modules use the observed 3D point cloud as input, connecting generative modeling to grasp assessment.
3. 6-DOF Grasp Pose Generation
The method generates diverse 6-DOF grasp poses from an observed object point cloud, then evaluates and iteratively refines those poses. A VAE models successful-grasp modes, while an evaluator identifies and improves likely failures.
- Problem formulation: Grasp generation targets diverse stable poses in SE(3) conditioned on a partial object point cloud.The successful-grasp distribution may contain multiple separated modes, such as a mug’s rim, handle, and bottom.
- Variational grasp sampler: The grasp sampler generates different poses by decoding different latent samples conditioned on the object point cloud.During inference, latent values are sampled from a standard normal distribution, while the encoder is removed.
- Grasp refinement: Evaluator gradients iteratively refine sampled grasps, including moving grippers out of collision or improving alignment with the object.The refinement update is bounded so the maximum translation change does not exceed 1 cm.
- Variational grasp sampler: The VAE is trained by reconstructing successful grasps while regularizing the encoder’s latent distribution toward a unit-variance normal distribution.The reconstruction loss compares transformed predefined points on the robot gripper.
- Grasp evaluation: The evaluator predicts grasp success from the object point cloud and grasp pose, using hard negatives that are near successful poses but collide or miss the object.Hard negatives are obtained from pre-generated negatives and perturbations of positive grasps.
4. Experiments
The experiments train on physics-simulated grasps, evaluate success and diversity, and study latent dimensionality, refinement, sampling quantity, and sampler design. Simulation and robot tests show that refinement improves grasp quality, more samples improve coverage, and the learned sampler outperforms geometric sampling.
- Training Data: 7,074,038 of 10,816,720 candidate grasps are simulated for 206 objects from six ShapeNet categories.Candidates are sampled from object geometry and filtered for collision-free configurations whose closing volume intersects the object.
- Evaluation Metrics: Success rate measures executed grasp success, while coverage rate measures how well generated grasps cover the space of positive grasps.A positive grasp is covered when a predicted grasp lies within 2 cm in translation.
- Latent Space Dimensionality: A two-dimensional latent space is selected because one dimension lacks capacity, whereas three- and four-dimensional spaces perform worse during inference.Higher-dimensional spaces are harder to cover densely during training, degrading sampled grasp quality at inference.
- Refinement: Refinement increases both success rate and coverage, with the AUC of the success-coverage curves plateauing after 10 iterations.The reported curves are computed over 16,600 grasps.
- Sampled Grasps: 2,000 sampled grasps increase coverage relative to the 200-grasp setting used in the ablations and robot experiments.The larger sample count is generated in 10 batches on the same point clouds.
- Sampler Comparison: The VAE sampler with latent size 2 outperforms geometric sampling in both success and coverage.Geometric sampling rarely generates grasps around rims or thin structures and does not extrapolate to missing depths or occlusions.
- Robot Experiments: In robot experiments, the method outperforms GPD in success rate across all tested objects, with diversity helping find kinematically feasible grasps.A grasp is successful when the robot lifts the object 10 cm without dropping it.
5. Conclusions
6-DOF GraspNet combines a VAE for diverse grasp sampling with an evaluator network for iterative refinement. The approach transfers from synthetic training to real-robot grasping while identifying extensions for scene-aware grasp generation and manipulation feedback.
- The method combines a VAE that samples varied grasps with an evaluator network that assesses and iteratively refines their quality.The evaluator addresses the VAE's insufficient precision for highly robust grasp generation.
- The model is trained with synthetic grasp data from a physics simulator, avoiding the need to collect real-world data for large object sets.This supports scaling to many objects without real-world data collection.
- A stated extension is to account for surrounding objects when generating grasps and to use the evaluator for real-time manipulator feedback.The current direction also includes directly avoiding colliding or infeasible grasps in cluttered scenes.