Source-linked AI summary
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes
Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox
TL;DR
Grasping unknown objects in clutter is difficult because existing full 6-DoF approaches can involve complex pipelines and challenging output spaces. Contact-GraspNet generates grasp distributions directly from scene depth using observed contact points, reducing the representation to 4-DoF. In structured-clutter robot experiments, it achieves over 90% success and reduces failures relative to a recent state-of-the-art method.
Problem
6-DoF grasp generation for unknown objects in clutter involves multimodal, discontinuous, imbalanced, and occlusion-ambiguous grasp distributions.
Method
Contact-GraspNet directly generates collision-aware parallel-jaw grasps from depth by representing grasps through observed contact points and supervising combined pose predictions.
Results
Over 90% success rate is achieved in a robotic study of unseen objects in structured clutter.
Takeaways & Limitations
The method generates diverse grasps covering the graspable surface while avoiding gripper collisions during generation.
Abstract
from arXiv · showhide
Grasping unseen objects in unconstrained, cluttered environments is an essential skill for autonomous robotic manipulation. Despite recent progress in full 6-DoF grasp learning, existing approaches often consist of complex sequential pipelines that possess several potential failure points and run-times unsuitable for closed-loop grasping. Therefore, we propose an end-to-end network that efficiently generates a distribution of 6-DoF parallel-jaw grasps directly from a depth recording of a scene. Our novel grasp representation treats 3D points of the recorded point cloud as potential grasp contacts. By rooting the full 6-DoF grasp pose and width in the observed point cloud, we can reduce the dimensionality of our grasp representation to 4-DoF which greatly facilitates the learning process. Our class-agnostic approach is trained on 17 million simulated grasps and generalizes well to real world sensor data. In a robotic grasping study of unseen objects in structured clutter we achieve over 90% success rate, cutting the failure rate in half compared to a recent state-of-the-art method.
I. INTRODUCTION
Contact-GraspNet addresses 6-DoF grasping of unknown objects in structured clutter, where collision avoidance and grasp diversity must coexist with difficult learning and practical runtime constraints. It introduces an end-to-end method and a 4-DoF contact-based representation, achieving 90% grasp success in real-world clutter.
- I. INTRODUCTION: Existing model-free grasping is challenging because one shared representation must cover varied object shapes and sizes within the large SE(3) pose space.Planar representations simplify learning but restrict three-dimensional reasoning and neglect many possible grasps.
- I. INTRODUCTION: 6-DoF grasping in structured clutter requires successful target grasps, collision avoidance, and diverse proposals compatible with robot kinematics.These requirements are especially important in home robotics and healthcare applications.
- I. INTRODUCTION: Contact-GraspNet directly processes full scene point clouds or local target regions, avoiding dependence on accurate instance masks and incorporating collisions during generation.Instance segmentation can subsequently filter grasps belonging to a target object.
- I. INTRODUCTION: 90% grasp success rate is achieved in cluttered real-world scenes, 10% higher than in equal settings.The method is presented as an end-to-end approach for 6-DoF grasping of unknown objects.
- I. INTRODUCTION: The proposed grasp representation projects 6-DoF grasps to contact points in an observed point cloud and reduces the representation to 4-DoF.This reduction is reported to facilitate learning significantly.
II. RELATED WORK
Related work spans end-to-end policies, discriminative grasp evaluation, and generative methods, but 6-DoF grasping remains difficult because of limited generalization, sampling complexity, and a large multimodal search space. Contact-GraspNet instead generates collision-aware grasp proposals directly from scene depth.
- II. RELATED WORK: End-to-end policies learn actions from raw pixels but commonly combine perception, planning, grasping, and control in a monolithic model.Their generalization to novel environments is limited because perception and control are learned indirectly together.
- II. RELATED WORK: Discriminative methods classify existing grasp candidates, while geometric heuristics are often used to sample grasp locations in higher-dimensional spaces.The cross-entropy method works poorly in the higher-dimensional 6-DoF grasp space.
- II. RELATED WORK: Generative grasp methods aim to produce meaningful 6-DoF grasps, but their central challenge is the large, multimodal search space.These methods often draw experience from physics simulators.
- II. RELATED WORK: The paper takes raw depth images, optionally with object masks, and generates robust, diverse, non-colliding 6-DoF grasp proposals with widths from partially observed clutter.The method does not require a full explicit 3D reconstruction.
- II. RELATED WORK: Learning successful grasp distributions is difficult because they are multimodal, discontinuous, imbalanced, and ambiguous under occlusion.Direct regression in high-dimensional output spaces such as SE(3) has been shown to be difficult.
A. Grasp Representation
The grasp representation anchors each predicted grasp to an observed contact point, converting 6-DoF pose estimation into contact classification plus rotation and width estimation. This reduces the output space while preserving scene geometry and grasp-distribution modes.
- A. Grasp Representation: A visible contact point is used to map successful 6-DoF grasps onto nearby points in the recorded point cloud.Most predictable two-finger grasps expose at least one contact before grasping.
- A. Grasp Representation: Given a suitable contact point, learning reduces to estimating 3-DoF grasp rotation and parallel-jaw grasp width.The contact point roots the remaining grasp parameters in observed scene geometry.
- A. Grasp Representation: The grasp pose is defined from contact point c using rotation vectors a and b, grasp width w, and distance d from the gripper baseline to its base.a is the approach vector, b is the grasp baseline vector, and d is constant.
- A. Grasp Representation: The reduced representation facilitates learning, improves pose accuracy by binding predictions to observed geometry, and avoids axis-angle ambiguities and discontinuities.At test time, sampling contact points across the observable surface represents modes of the 6-DoF grasp distribution.
- A. Grasp Representation: PointNet++ processes point clouds with local hierarchical feature aggregation, allowing predictions to be directly associated with input 3D points.This capability supports the proposed contact-based grasp representation.
B. Data Generation
The training data projects simulated non-colliding grasps onto rendered scene point clouds, assigning points as feasible or infeasible contacts. Dense annotations from ACRONYM provide broad grasp-distribution coverage.
- ACRONYM contains 8,872 ShapeNet meshes and 17.7 million simulated grasps under varying friction.
- Rendered scene point clouds assign point-wise grasp-success labels based on nearby non-colliding mesh contacts.
- Points without a feasible contact within r = 5mm form P−, while points suitable for contacts form P+.
- Sufficient point-cloud coverage enables dense projection of the ground-truth 6-DoF grasp distribution onto the recorded scene.
C. Network
The network predicts grasp quantities at selected scene points, while orthonormalizing rotation components to reduce representation complexity and facilitate 3D rotation regression.
- The network predicts grasp quantities for 2,048 farthest points selected from an input of 20,000 random scene points.
- Four network heads output contact success, two rotation-related vectors, and ten grasp-width-bin values per point.
- Gram–Schmidt orthonormalization couples the predicted rotation vectors, so only the component orthonormal to b̂ is predicted for â.
- The orthonormalized representation reduces dimensionality and facilitates regression of 3D rotations.
D. Target Losses
The loss trains contact confidence and grasp geometry jointly, using positive-contact supervision, symmetry-aware pose distances, and weighted width-bin classification.
- Contact-success predictions use binary cross entropy, with backpropagation restricted to the 512 largest pointwise errors to counter data imbalance.
- Predictions for grasp geometry are evaluated only at positive contact points.
- The grasp loss compares transformed ground-truth and predicted gripper points using a symmetry-aware weighted minimum average distance.
- Distances to the closest ground-truth grasp points are weighted by predicted contact-success confidence ŝ_i.
- The joint loss can capture multiple grasp-distribution modes, couple contact classification with pose accuracy, and penalize grasps far from ground truth.
- Width prediction uses weighted multi-label binary cross entropy, with losses weighted inversely to grasp-width-bin size.
E. Implementation Details
Implementation uses Adam training, scene-centered inference, and a segmentation-aware pipeline that can process full scenes or local target regions. Ablations assess loss weighting, calibration, noise, and region selection.
- Implementation Details: Adam training starts at learning rate 0.001 and decays stepwise to 0.0001.
- Implementation Details: Inference centers the point cloud at its mean in camera coordinates.
- Inference Pipeline: The pipeline segments unknown objects from RGB-D input, processes a full scene or local region, and associates grasps with segments through contact-point filtering.
- Loss Ablations: Removing weighted width-bin loss reduces both success rate and coverage, while the ladd−s loss improves high-confidence success but slightly lowers low-confidence success.
- Data Ablations: Gaussian-noise training preserves simulation performance while improving generalization to noisy sensor data.
- Data Ablations: Full-scene prediction has similar average success but significantly lower grasp coverage than local-region processing.
IV. EXPERIMENTAL EVALUATION
The method is evaluated both on a Franka robot grasping unknown objects in cluttered scenes and in simulation across predicted grasps and method or data variations.
- The physical evaluation uses a Franka robot to pick unknown objects from cluttered scenes.
- The study compares different variations of the method and its training data.
- A large number of predicted grasps are executed in the FleX physics simulator.
A. Inference
Contact-GraspNet can process full-scene depth-derived point clouds or target-centered local regions, selecting diverse, reachable grasps while accounting for scene collisions.
- Inference: Contact-GraspNet processes a raw depth image or a full scene point cloud, with optional local regions around target segments.Instance detection or segmentation may specify a target, but the network itself can operate on raw depth images.
- Inference: 0.28s for a full scene or ∼0.19s for a local region enables applications requiring reactive closed loop grasping.
- Inference: At test time, confidence thresholding and farthest point sampling select broad grasp coverage before choosing a reachable, collision-free grasp.
- Inference: The evaluation limits execution to a maximum of two grasp trials per object without rearrangements and reports single-trial success.
- Inference: Simulator grasp success requires collision-free opening and object retention after grasping and shaking, while coverage measures proximity to ground-truth grasps.
C. Real robot grasp experiments
Real-robot experiments evaluate Contact-GraspNet on unseen objects in replicated cluttered scenes, while ablations examine optimization targets, data, and failure cases.
- Real robot grasp experiments: The robot study uses a 7-DoF Franka Panda, a parallel-jaw gripper, nine cluttered scenes, and 51 unseen objects.
- Real robot grasp experiments: The method achieves a significantly higher grasp success rate than and, with stronger first-trial success and fewer re-grasps.
- Ablations: Weighting grasp-width bins prevents predictions from collapsing into narrow widths, while weighting outperforms oversampling.
- Ablations: Connecting contact confidence with grasp pose improves calibration.
- Ablations: Training on a small dataset with 110 objects from 5 categories is insufficient for out-of-category generalization, irrespective of method.
- Ablations: Thick objects requiring near-maximum-width grasps and small objects can produce lower-confidence predictions.Noise injection during training reduces the effect for thick objects.
V. CONCLUSIONS
Contact-GraspNet simplifies 6-DoF grasp generation by predicting contact points and rotations directly in cluttered scenes. The resulting approach generates diverse grasps, avoids collisions, and does not depend on accurate segmentation masks.
- V. CONCLUSIONS: Contact-GraspNet transforms 6-DoF grasp estimation into contact-point classification and grasp-rotation estimation.
- V. CONCLUSIONS: Tailored optimization targets address multi-modality, imbalance, and sparsity to generate diverse grasps across the graspable surface.
- V. CONCLUSIONS: Predicting grasps directly in scenes and considering gripper collisions during training helps avoid collisions.
- V. CONCLUSIONS: The approach can incorporate segmentation predictions but is not dependent on accurate masks.
- V. CONCLUSIONS: The method is complementary to grasp-ranking methods that use gripper or robot models as input.