Source-linked AI summary
End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGB
Stefan Ainetter, Friedrich Fraundorfer
TL;DR
Robotic grasping in cluttered scenes requires accurate grasp detection together with semantic understanding of overlapping objects. The paper introduces an end-to-end multi-task CNN with grasp refinement and evaluates it on established datasets plus an OCID extension, reporting state-of-the-art grasp accuracy and object-specific grasp assignment.
Problem
Grasp detection in cluttered scenes must handle overlapping objects while identifying which grasp candidates belong to which objects.
Method
An end-to-end CNN jointly predicts parallel-plate grasps and semantic segmentation, then refines grasp candidates using both outputs.
Results
The network achieves state-of-the-art grasp-detection accuracy on Cornell and Jacquard, while refinement improves orientation accuracy under stricter criteria.
Takeaways & Limitations
The OCID grasp extension enables class-specific evaluation and demonstrates that semantic segmentation can assign grasp candidates to specific objects in complex scenes.
Abstract
from arXiv · showhide
In this work, we introduce a novel, end-to-end trainable CNN-based architecture to deliver high quality results for grasp detection suitable for a parallel-plate gripper, and semantic segmentation. Utilizing this, we propose a novel refinement module that takes advantage of previously calculated grasp detection and semantic segmentation and further increases grasp detection accuracy. Our proposed network delivers state-of-the-art accuracy on two popular grasp dataset, namely Cornell and Jacquard. As additional contribution, we provide a novel dataset extension for the OCID dataset, making it possible to evaluate grasp detection in highly challenging scenes. Using this dataset, we show that semantic segmentation can additionally be used to assign grasp candidates to object classes, which can be used to pick specific objects in the scene.
I. INTRODUCTION
The paper addresses grasp detection and semantic understanding in cluttered scenes, where overlapping objects make object assignment difficult. It proposes a joint network, refinement module, and OCID grasping extension to improve grasp selection and evaluation.
- Motivation: Object detection identifies scene objects but provides limited shape information, especially when objects overlap in highly stacked scenes.This motivates semantic segmentation alongside grasp detection.
- Results: The dataset extension indicates high grasp detection and segmentation accuracy in complex scenes with multiple objects.Figure 1 illustrates how combining semantic segmentation with grasp candidates supports object-specific picking.
- Contributions: The proposed network jointly performs parallel-plate grasp detection and dense, pixel-wise semantic segmentation.The architecture is end-to-end trainable and uses a shared representation for both tasks.
- Contributions: The grasp refinement module combines grasp detection and segmentation outputs to further improve grasp accuracy.It uses object-shape information together with previously predicted grasp candidates.
- Contributions: The OCID extension adds oriented grasp annotations and object class information, enabling evaluation of grasp candidates by object class.This supports selecting a specific object in scenes containing multiple graspable objects.
B. Semantic Segmentation
Semantic segmentation assigns class labels to image pixels and supplies object-level shape information that complements grasp detection. The proposed architecture adapts a panoptic-segmentation framework while sharing features across tasks.
- Grasp representation: The grasp representation describes the location, dimensions, and orientation of a parallel-plate gripper before closure.The five-dimensional representation uses x, y, w, h, and θ for a grasp candidate.
- Definition: Semantic segmentation assigns a class label to each pixel in the image.For unseen objects, the paper uses graspable and non-graspable classes; datasets with class information can use additional classes.
- Architecture: The architecture adapts a panoptic-segmentation framework for joint grasp detection and semantic segmentation.The framework overview is presented through the proposed network architecture.
A. Shared Backbone
The model uses a shared feature-extraction backbone feeding separate grasp-detection and segmentation processing. The grasp branch begins with region proposals and predicts grasp candidates from proposal-aligned features.
- Shared Backbone: A modified ResNet-101 with a Feature Pyramid Network serves as the shared feature extractor.The backbone links ResNet modules conv2 through conv5 to the FPN.
- Grasp Detection Branch: The grasp detection branch is based on Faster R-CNN, with a Region Proposal Network followed by a detection stage.The detector is modified to predict robotic grasp candidates.
- Region Proposal Network: The Region Proposal Network outputs axis-aligned rectangular proposals without orientation information.Each proposal is represented by its center coordinates, width, and height.
- Grasp Detection Head: ROIAlign extracts 14 × 14 proposal features, which are average-pooled and processed by fully connected layers before grasp prediction.The grasp head predicts orientation classes and correction factors for each proposal.
- Grasp Detection Head: The grasp candidate is calculated from predicted correction factors and orientation information for each input proposal.The candidate is represented using the paper’s five-dimensional grasp parameterization.
C. Segmentation Branch
The segmentation branch predicts semantic probability maps, while the refinement head combines these maps with grasp candidates to correct grasp geometry. This connects object-shape information to grasp refinement.
- C. Segmentation Branch: The segmentation head processes the first four FPN scales through independent Mini-DeepLab modules.The modules capture global image structures with relatively low memory consumption.
- C. Segmentation Branch: Upsampled segmentation features are concatenated and passed through a 1×1 convolution producing probabilities for all semantic classes.The output represents a pixel-wise semantic class distribution.
- D. Grasp Refinement Head: The refinement head receives previously predicted grasp candidates and a semantic probability map from the segmentation branch.It crops the candidate region, stacks the cropped and original maps, and feeds them to an MLP.
- D. Grasp Refinement Head: The refinement MLP predicts five correction factors that are used to calculate a refined grasp candidate.The factors correspond to grasp position, size, and orientation corrections.
- D. Grasp Refinement Head: The refinement operation is performed simultaneously for N grasp candidates using stacked semantic probability maps.The resulting MLP input has dimension N × 2 × H × W.
V. TRAINING LOSSES
The training objective combines grasp detection, semantic segmentation, and grasp refinement losses, with separate components for proposal, box, and orientation learning.
- The composite loss jointly trains grasp detection, semantic segmentation, and grasp candidate refinement.Each task contributes a separately weighted component to the overall objective.
- A. Grasp Detection Loss: Grasp detection loss combines RPN, bounding-box regression, and grasp-orientation classification losses.The RPN term trains region proposals, the box term regresses coordinates, and the orientation term classifies grasp orientation.
- A. Grasp Detection Loss: Valid and invalid region proposals form the proposal set used for grasp-orientation classification.The set R contains proposals generated by the RPN, partitioned into valid and invalid subsets.
- A. Grasp Detection Loss: Bounding-box regression uses parameters from region proposals and corresponding ground-truth grasp candidates.The loss compares proposal parameters (x, y, w, h) with ground-truth candidate parameters.
B. Segmentation Loss
The segmentation branch uses a weighted per-pixel loss over semantic classes, with hard-negative mining emphasizing the lowest predicted probabilities.
- Semantic segmentation defines classes as Y = {1, ..., Sclasses} and applies a weighted per-pixel loss.The loss is computed over pixel positions and semantic classes.
- Hard-negative mining selects the 25% of pixels with the lowest predicted class probabilities.Selected pixels receive weight 4/(W H), while the remaining pixels receive zero weight.
C. Grasp Refinement Loss
The refinement loss regresses grasp candidates against ground truth while additionally accounting for orientation and the previously predicted grasp.
- The refinement loss extends the box loss by adding smooth L1 regression for the orientation parameter θ.This trains the refinement head to correct both grasp geometry and orientation.
- Refinement correction factors incorporate the orientation of the previously predicted grasp candidate.The correction targets are adapted to account for the candidate’s predicted orientation.
- The orientation offset is constrained to [0, π), matching the possible grasp-orientation interval.The mod operation adjusts the orientation offset into this interval.
VI. EXPERIMENTS AND RESULTS
The experiments evaluate RGB-only grasp detection and semantic segmentation on Cornell, Jacquard, and an OCID extension using progressively richer model configurations.
- The evaluation uses Cornell, Jacquard, and a novel OCID extension for robotic grasping.The OCID extension supports evaluation in complex scenes with multiple objects.
- RGB-only comparisons include detection-only, detection-plus-segmentation, and full detection-segmentation-refinement configurations.These configurations isolate the grasp detection branch, add segmentation, and then add refinement.
- A. Evaluation Metric: A grasp is counted correct when its angle difference is within 30° and its IoU exceeds 0.25.Both criteria must hold for the predicted and ground-truth grasp candidates.
- B. Segmentation Evaluation: Semantic segmentation is evaluated using IoU between predicted and ground-truth segmentations.
B. Cornell dataset
On Cornell, the RGB-only implementation achieves state-of-the-art grasp accuracy and high inference speed, supporting real-time use.
- The Cornell evaluation uses 5-fold cross-validation with an image-wise split and reports mean grasp detection accuracy.Cornell contains 885 RGB-D images of 240 objects, but the reported evaluation concerns grasp detection because ground-truth segmentation is unavailable.
- The RGB-input implementation achieves state-of-the-art grasp accuracy compared with other RGB methods.
- High inference FPS makes the model well suited for real-time applications.
D. OCID Dataset with Grasp-Extension
The OCID grasp extension adds object-level grasp and class annotations for complex scenes, enabling class-specific evaluation and demonstrating accurate grasping and segmentation.
- OCID grasp contains 1,763 images, over 11.4k segmented object masks, more than 75k hand-annotated grasp candidates, and 31 object classes.The extension annotates ARID10 and ARID20 subsets with valid grasps and assigns each candidate to its corresponding object class.
- Semantic segmentation enables grasp candidates to be assigned to object classes for selecting specific objects in complex scenes.The best candidate for each class is selected by confidence, subject to its center lying on that class’s segmentation mask.
- High grasp accuracy and segmentation IoU indicate accurate performance on complex OCID grasp scenes.Table V evaluates grasp accuracy at object-class level and mean IoU for instance segmentation.
VII. CONCLUSIONS
The paper concludes that its shared-backbone CNN jointly produces grasp detection and semantic segmentation, while refinement improves grasp accuracy, especially orientation.
- The proposed CNN jointly produces grasp detection and semantic segmentation using a shared backbone network.
- The grasp refinement module increases the accuracy of previously predicted grasp candidates, especially their orientation accuracy.
- The OCID grasp extension demonstrates accurate grasp detection in complex scenes and assignment of candidates to specific objects through semantic segmentation.