Source-linked AI summary
Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter
Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Siegwart, Juan Nieto
TL;DR
Robotic grasping in clutter must generalize to unseen objects while handling occlusions, noisy sensing, and physical interactions. VGN maps a TSDF scene representation directly to dense 6 DOF grasp predictions, and achieves 10 ms inference with 92% object clearance in real-world clutter experiments. The approach supports grasp planning without explicit collision checking, but its simulated-data training leaves limitations in real-world friction modeling.
Problem
Robots deployed in unstructured environments need grasps for unseen objects despite clutter, occlusions, noisy sensing, and restricted top-down approaches.
Method
VGN uses a TSDF voxel grid and a fully convolutional network to predict grasp quality, orientation, and opening width for each voxel.
Results
10 ms inference and 92% object clearance were achieved in the reported real-time and real-world clutter experiments, respectively.
Takeaways & Limitations
The dense 6 DOF predictions enable grasp proposals without explicit collision checking and support direct transfer to a physical setup without fine-tuning.
Takeaways & Limitations
A point estimate for orientation cannot capture the full distribution of grasps at a voxel, although it reduces approach complexity.
Abstract
from arXiv · showhide
General robot grasping in clutter requires the ability to synthesize grasps that work for previously unseen objects and that are also robust to physical interactions, such as collisions with other objects in the scene. In this work, we design and train a network that predicts 6 DOF grasps from 3D scene information gathered from an on-board sensor such as a wrist-mounted depth camera. Our proposed Volumetric Grasping Network (VGN) accepts a Truncated Signed Distance Function (TSDF) representation of the scene and directly outputs the predicted grasp quality and the associated gripper orientation and opening width for each voxel in the queried 3D volume. We show that our approach can plan grasps in only 10 ms and is able to clear 92% of the objects in real-world clutter removal experiments without the need for explicit collision checking. The real-time capability opens up the possibility for closed-loop grasp planning, allowing robots to handle disturbances, recover from errors and provide increased robustness. Code is available at https://github.com/ethz-asl/vgn.
1 Introduction
The paper targets flexible grasping in cluttered, unstructured scenes, where robots must handle unseen objects, occlusions, noisy sensing, and physical interactions. It proposes VGN, which predicts real-time 6 DOF grasps directly from volumetric scene information.
- Unstructured robot domains require grasps for previously unseen objects despite clutter, occlusions, and noisy on-board sensor readings.
- Top-down grasping from single depth images restricts approach direction and depends on favorable camera placement.
- VGN takes a 3D voxel grid containing truncated distances to surfaces as input for real-time 6 DOF grasp synthesis.
- The network uses a fully convolutional architecture trained on synthetic cluttered grasp trials to predict grasp quality, orientation, and width throughout the volume.
- VGN enables 6 DOF synthesis in real time and directly learns collision-free grasp proposals from full 3D scene information.
2 Related Work
Related grasping methods differ in both scene representation and how they generate candidates. VGN uses full volumetric information to predict dense grasp proposals directly, avoiding runtime sampling and iterative evaluation.
- Deep learning grasp detectors generally improve generalization to unseen objects and success in clutter compared with manually designed policies.
- Top-down methods typically use image-based inputs, whereas 6 DOF methods use full 3D representations such as point clouds or occupancy grids.
- Using the full TSDF lets VGN avoid separate scene completion and explicit collision-detection processing.
- Several approaches predict grasp quality only after an initial sampling step, creating a trade-off between computational tractability and grasp-space coverage.
- VGN provides grasp orientation, width, and quality for every voxel, then selects high-scoring grasps with non-local maxima suppression.
3 Problem Formulation
The task is to plan parallel-jaw grasps for unknown rigid objects in clutter and remove them successfully. The formulation represents the scene volumetrically and predicts a dense grasp map over discretized positions.
- The goal is to find parallel-jaw gripper configurations that successfully grasp and remove unknown rigid objects from clutter.
- Rather than sampling and evaluating individual candidates, the formulation evaluates many discretized grasp positions in parallel.
- The input is a TSDF voxel grid whose cells contain truncated signed distances to the nearest surface.
- Each grasp is defined by gripper position, orientation, opening width, and a success-probability quality score q ∈[0, 1].
- The dense grasp map predicts quality, orientation, and width at every voxel, while using a point estimate for orientation to reduce model complexity.
4 Volumetric Grasping Network
VGN approximates a dense grasp map with a fully convolutional network that predicts grasp quality, orientation, and width throughout a TSDF volume. It is trained end-to-end on simulated grasp trials and queried to select high-quality grasps without runtime candidate sampling.
- VGN is a deep neural network approximation of a dense grasp map over the scene volume.
- The fully convolutional architecture maps an input volume to voxel-aligned predictions through quality, rotation, and width heads.The rotation head uses quaternions to represent grasp orientation.
- The training loss combines grasp-quality classification with width and rotation losses, while accounting for parallel-jaw gripper rotational symmetry.The symmetry-aware rotation loss considers both equivalent orientations separated by 180° about the wrist axis.
- 4.2 Synthetic Training: Synthetic training uses pile and packed scenes, sampled viewpoints, collision-labeled grasp trials, and approximately 2 million balanced grasps.Both top-down and side grasps are represented in the training distribution.
- At inference, VGN smooths quality predictions, masks geometrically invalid voxels, thresholds low-quality locations, and returns promising grasps from the output volumes.
5 Experiments
The experiments evaluate VGN in simulation and on a physical robot for cluttered grasping, including previously unseen objects and dense scenes. VGN achieves fast GPU inference, strong simulated clutter-removal performance, and 92% real-world object clearing, while failures remain tied to contact, collisions, and CPU-only deployment.
- 5 Experiments: Experiments assess grasp performance, real-time detection, and transfer to a physical system without additional fine-tuning.
- 5.1 Experimental Setup: The physical setup uses a 7 DOF Panda arm, parallel-jaw gripper, wrist-mounted depth sensor, and a 30 × 30 × 30 cm3 TSDF workspace.
- 5.1 Experimental Setup: Simulation uses 343 object meshes split into 303 training and 40 testing objects, with pile and packed clutter-removal scenarios.
- 5.2 Simulated Experiments: A quality threshold of 0.9 provides a reported balance between grasp success rate and percentage of cleared objects in simulation.Using ϵ = 0.95 increases success rates but removes fewer objects, while decreasing ϵ lowers success rates.
- 5.2 Simulated Experiments: 10 ms GPU inference enables substantially faster grasp planning than GPD, which required 1.2 s on the same computer.
- 5.3 Real Robot Experiments: In real clutter removal, 55 of 68 grasp attempts succeeded, yielding an 80% success rate and 92% cleared.Most failures involved insufficient friction on cylindrical objects, with additional collision and unstable-slip failures.
- 5.3 Real Robot Experiments: Additional packed-object trials demonstrated side-grasps and grasps on thin bowl rims, although one box was knocked over during transport.
6 Conclusion
The Volumetric Grasping Network generates 6 DOF grasp proposals end-to-end in a single forward pass and performs well in cluttered scenes without explicit collision checking. Its TSDF-based approach transfers directly to a real robotic setup, while simulation-only training and unverified gripper scaling remain limitations.
- The network generates 6 DOF grasp proposals with a single forward pass of a Fully Convolutional Network.
- Synthetic grasping trials train the network to perform well in highly cluttered scenes without explicit collision checking.
- A modern Graphics Processing Unit supports the approach's efficiency.
- The model transfers directly to a real robotic setup without additional adjustments through its TSDF-based approach.
- Training only on simulated data remains a limitation identified during real-robot experiments.The authors suggest investigating an adversarial test in the physics simulation to address this issue.
- Scaling the approach to different gripper geometries is straightforward in principle but requires further experimental verification.