Source-linked AI summary
Fully Convolutional Grasp Detection Network with Oriented Anchor Box
Xinwen Zhou, Xuguang Lan, Hanbo Zhang, Zhiqiang Tian, Yang Zhang, Nanning Zheng
TL;DR
Robotic grasp detection remains challenging because accurate grasp locations are needed for stable manipulation. The paper proposes an oriented-anchor fully convolutional approach with efficient matching, achieving higher Cornell Grasp Dataset accuracy than the current state-of-the-art method.
Problem
Robotic grasping remains challenging because inappropriate grasp locations can make object manipulation unstable, motivating more accurate and rapid grasp-location detection.
Method
The paper uses a fully convolutional network with oriented anchor boxes and an efficient matching strategy to predict grasp configurations for a parallel-plate gripper.
Results
97.74% image-wise and 96.61% object-wise accuracy on the Cornell Grasp Dataset, outperforming the state-of-the-art by 1.74% and 0.51%, respectively.
Takeaways & Limitations
The oriented anchor box mechanism provides a more accurate and efficient grasp-detection approach using fewer predictions, while the model can predict diverse grasps for an object.
Abstract
from arXiv · showhide
In this paper, we present a real-time approach to predict multiple grasping poses for a parallel-plate robotic gripper using RGB images. A model with oriented anchor box mechanism is proposed and a new matching strategy is used during the training process. An end-to-end fully convolutional neural network is employed in our work. The network consists of two parts: the feature extractor and multi-grasp predictor. The feature extractor is a deep convolutional neural network. The multi-grasp predictor regresses grasp rectangles from predefined oriented rectangles, called oriented anchor boxes, and classifies the rectangles into graspable and ungraspable. On the standard Cornell Grasp Dataset, our model achieves an accuracy of 97.74% and 96.61% on image-wise split and object-wise split respectively, and outperforms the latest state-of-the-art approach by 1.74% on image-wise split and 0.51% on object-wise split.
I. INTRODUCTION
The paper targets accurate and rapid grasp-location detection from RGB images for parallel-plate grippers. It introduces oriented anchor boxes, a faster matching strategy, and an end-to-end fully convolutional network.
- Motivation: Robotic grasping remains challenging because inappropriate grasp locations can make object manipulation unsteady.The paper focuses on more accurate and rapid grasp-location detection for household and industrial scenes.
- Motivation: Prior reference rectangles use fixed horizontal orientations, while grasp orientation is a geometric attribute coupled to location.Earlier methods decouple orientation prediction from bounding-box prediction despite orientation being important for feasible grasps.
- Contributions: Oriented anchor boxes tile rectangles with different default rotation angles across the image to improve grasp detection.An example uses orientations of 60°, 0° and -60° at one grid cell.
- Contributions: The model uses an end-to-end fully convolutional network with a feature extractor and multi-grasp predictor.The approach uses a pretrained ResNet feature extractor, convolutional prediction layers, and data augmentation to address the limited Cornell dataset.
- Contributions: The proposed matching strategy combines a point metric with an orientation constraint and is faster than the previous matching strategy.It accounts for location and rotation angle with lower computational cost than computing oriented-rectangle overlap against horizontal reference rectangles.
- Results: 97.74% image-wise accuracy and 96.61% object-wise accuracy exceed Chu et al. by 1.74% and 0.51%, respectively.These results are reported on the Cornell Grasp Dataset.
II. RELATED WORK
Earlier grasp-detection methods progressed from 3-D models and image-based classifiers to one-stage and two-stage detection frameworks. Anchor-box methods generally used horizontal rectangles, while later work incorporated classification of discrete orientations and denser predictions.
- 3-D and image-based methods: Earlier methods relied on complete 3-D object models, which are difficult to acquire in changing real-world environments.RGB images are more convenient to capture than reconstructing full 3-D models.
- 3-D and image-based methods: Lenz et al. transformed 2-D five-dimensional grasp detection into an object-detection-like problem and achieved 73.9% image-wise accuracy at 13.5 seconds per frame.Their sliding-window network classified whether a good grasp existed in a small image patch.
- Deep grasp detection: Redmon et al. introduced locally constrained predictions at each image location, with Direct Regression predicting one grasp and MultiGrasp Detection predicting multiple grasps.The MultiGrasp Detection model divides the image into N×N grid cells.
- Deep grasp detection: Kumra et al. used ResNet-50 to regress grasp configurations from RGB-D or RGB input, obtaining 89.21% with RGB-D and 88.84% with RGB.RGB-D improved accuracy by 0.37% over RGB in their comparison.
- Anchor-box methods: Guo et al. combined graspability, non-oriented bounding boxes, and orientation in a hybrid architecture using reference rectangles with varied scales and aspect ratios.Their approach used visual and tactile sensing and outperformed Redmon et al.'s MultiGrasp Detection model by 5.2%.
- Anchor-box methods: Chu et al. combined graspability and discrete orientation into region detection with orientation classification using 14×14×9 default reference rectangles.Their best model reported 96% image-wise and 96.1% object-wise accuracy.
- Detection frameworks: The reviewed methods include one-stage detectors that directly predict classes and anchor offsets and two-stage detectors requiring second-stage classification and refinement.The paper categorizes methods according to whether proposal-level processing follows initial detection.
III. PROBLEM FORMULATION
The paper represents grasps as oriented rectangles and introduces oriented anchor boxes with predefined angles to model rotation directly. These anchors are tiled across grid cells and parameterized for regression to ground-truth grasps.
- III. PROBLEM FORMULATION: A grasp is represented by an oriented rectangle with center (x, y), height h, width w, and execution angle θ.
- A. Oriented Anchor Box Mechanism: Each grid cell is associated with k oriented anchor boxes, enabling predictions of k grasp types with different default angles.
- A. Oriented Anchor Box Mechanism: The six-angle configuration uses 75°, 45°, 15°, -15°, -45° and -75° as default anchor rotations.
- A. Oriented Anchor Box Mechanism: Anchor offsets encode predicted center, width, height and rotation relative to the oriented anchor box and ground-truth grasp rectangle.
- A. Oriented Anchor Box Mechanism: The anchor-box parameterization uses logarithmic width and height ratios and normalized center and angle differences.
- A. Oriented Anchor Box Mechanism: Anchor boxes are set to 54×54 pixels, exceeding grid-cell size to cover the image seamlessly; the dataset has limited size variation.
- A. Oriented Anchor Box Mechanism: The network diagram includes a fully convolutional architecture with residual-block alternatives of 16 blocks for ResNet-50 and 33 for ResNet-101.
B. Model Architecture
The model combines a ResNet feature extractor with a convolutional multi-grasp predictor. The predictor classifies and refines multiple oriented anchors at every feature-map location.
- The architecture uses a ResNet feature extractor followed by a multi-grasp predictor, with ResNet-50 and ResNet-101 evaluated.
- A 320×320 RGB image produces 2048 feature maps of size 10×10 from the feature extractor.
- The classification and regression components are 3×3 convolutional layers rather than fully connected layers to avoid overfitting.
- At each 10×10 feature-map location, regression outputs 5×k offsets while classification outputs 2×k graspable and ungraspable scores.
- Classification determines whether an anchor is graspable, while regression refines its center, scale and orientation in one stage.
C. Matching Strategy
The matching strategy assigns ground-truth grasps to oriented anchors using both grid-cell location and rotation proximity. It avoids angle-insensitive horizontal-anchor overlap matching while remaining computationally simple.
- Positive anchors must place their centers in the same grid cell as the ground-truth rectangle.
- Positive anchors must have orientation differences below 90°/k relative to the ground-truth rectangle.
- The location condition follows a point metric that associates the nearest anchor center with the ground-truth rectangle center.
- Increasing k makes the matched anchor closer in angle to the ground-truth grasp, which helps regression loss but introduces more examples.
- Matching uses ceil operations on ground-truth center coordinates and orientation to select the grid cell and oriented anchor.
- The strategy accounts for both location and rotation angle at lower computational cost than computing oriented-rectangle intersection over union.
D. Loss Function
Training combines classification and regression losses for oriented-anchor predictions. Classification is selectively propagated because positive anchors are rare relative to negative anchors.
- The regression branch uses smooth L_1 loss for matched oriented-anchor offsets.
- Regression compares predicted offsets with ground-truth offsets for the matched anchor across center, width, height and angle parameters.
- Only positive examples and a subset of negative examples contribute to classification loss because positive anchors are a small fraction of all anchors.
- The selected negative examples are limited to three times the number of positive examples.
- The classification branch uses graspable scores for positive examples and ungraspable scores for negative examples.
- The total loss combines normalized classification and regression losses with balancing weight α set to 10.
A. Dataset
The Cornell Grasp Dataset contains labeled grasp rectangles across 885 images and 240 objects, with image-wise and object-wise splits testing different generalization settings.
- Dataset: The dataset contains 885 images of 240 graspable objects with positive grasp rectangles labeled in each image.For a given object, grasp rectangles vary in location, scale, and orientation.
- Dataset: The data are divided into 708 training images and 177 test images under two split protocols.Both protocols use the same train/test image counts.
- Dataset splits: Image-wise splitting tests generalization to new object positions and orientations for objects seen during training.Images are assigned randomly to training and validation sets.
- Dataset splits: Object-wise splitting tests generalization to entirely new object instances by keeping all images of each instance in one set.This split prevents images of the same instance from appearing across sets.
- Data preparation: Limited labeled robotics data motivate pre-training and data augmentation, both used to expand the effective training data.The paper uses these strategies because large manually labeled datasets are generally unavailable in robotics applications.
- Data preparation: Training augmentation applies random translation, rotation, and horizontal flipping to 320x320-pixel center crops.Translations reach 50 pixels in x and y, while rotations reach 15 degrees in either direction.
B. Implementation Details
The implementation uses ImageNet-pre-trained convolutional feature extractors, standardized training settings, and rectangle-based grasp evaluation with overlap and angle constraints.
- Network initialization: ImageNet-pre-trained convolutional feature extractors help the large network avoid overfitting on the limited dataset.The pre-training is performed on RGB images from ImageNet.
- Training settings: Models are trained end-to-end for 80k iterations with SGD momentum 0.9, learning rate 0.0001, decay 0.0001, and batch size 16.Experiments run on a single NVIDIA TITAN-X GPU using Torch.
- Evaluation settings: Table I reports accuracy under Jaccard thresholds of 20%, 25%, 30%, and 35%.These thresholds vary the overlap requirement used in evaluation.
- Evaluation settings: Table II reports model accuracy under different angle thresholds.The experiment examines sensitivity to the allowed rotation-angle difference.
- Evaluation metric: A predicted grasp is counted as good when its angle difference is within 30° and its Jaccard index exceeds 25%.The Jaccard index compares predicted and ground-truth grasp rectangles using their intersection and union areas.
VI. RESULTS
The model performs strongly on Cornell grasp detection, using relatively sparse oriented-anchor predictions while producing multiple candidate grasps and some failures under the rectangle metric.
- Threshold analysis: Accuracy declines more sharply when the angle threshold reaches 15° or below, potentially relating to k and the matching-strategy angle constraint.At thresholds above 15°, the model's performance remains good.
- Benchmark comparison: Table III compares the performance of different algorithms on the Cornell Grasp Dataset.The table provides the paper's main cross-method benchmark comparison.
- Qualitative results: Figure 4 compares ground-truth rectangles, Top1 detections, and multi-grasp detections in separate columns.Multi-grasp predictions can place rectangles at one center with different rotation angles and cover representative grasp locations with limited overlap.
- Benchmark performance: Using 600 predictions, the model outperforms the state-of-the-art method by 1.74% on image-wise split.The paper attributes this result to a more accurate and efficient oriented anchor box mechanism.
- Failure analysis: The paper reports four unsuccessful image-wise detections, although some metric failures remain feasible because annotations are not exhaustive.Figure 5 places detection results in the first row and ground-truth rectangles in the second.
- Complex scenes: Figure 6 evaluates the ResNet-101 model on overlapping objects and categories absent from the Cornell dataset.Transparent glasses under occlusion are identified as more difficult to detect.
VII. CONCLUSION
The paper introduces an architecture combining oriented anchor boxes with a new matching strategy for grasp detection. It also identifies broader-scene detection, richer data, realistic object diversity, and inference speed as future directions.
- The proposed grasp-detection architecture combines oriented anchor boxes with a new matching strategy.
- The model can predict diverse grasps for an object.
- Future work targets grasp-location detection for all objects and grasp-relationship parsing when objects overlap in piles.
- The authors plan to collect a larger, more detailed dataset and adapt the model to diverse objects in realistic scenes.
- The authors also plan to speed up inference by ameliorating the neural network, including pruning unnecessary channels.