Source-linked AI summary
Real-Time Grasp Detection Using Convolutional Neural Networks
Joseph Redmon, Anelia Angelova
TL;DR
Robotic grasp detection must handle novel objects accurately and quickly. This paper uses CNN-based single-stage regression with global image information, then extends it to classification and multiple locally constrained grasps. The resulting systems achieve state-of-the-art accuracy at real-time speeds and combine recognition with grasp prediction.
Problem
Robotic grasp detection from RGB-D views must identify good grasps for objects, while existing sliding-window approaches are computationally costly.
Method
The paper applies a CNN once to the full RGB-D image to regress grasp coordinates, with extensions for object classification and locally constrained multiple-grasp prediction.
Results
88 percent accuracy and 13 frames per second are reported for MultiGrasp, while the direct regression model achieves around 85 percent accuracy and 13 fps.
Takeaways & Limitations
Grasp detection and object classification can be combined without sacrificing accuracy or performance, and MultiGrasp combines global information with local prediction.
Abstract
from arXiv · showhide
We present an accurate, real-time approach to robotic grasp detection based on convolutional neural networks. Our network performs single-stage regression to graspable bounding boxes without using standard sliding window or region proposal techniques. The model outperforms state-of-the-art approaches by 14 percentage points and runs at 13 frames per second on a GPU. Our network can simultaneously perform classification so that in a single step it recognizes the object and finds a good grasp rectangle. A modification to this model predicts multiple grasps per object by using a locally constrained prediction mechanism. The locally constrained model performs significantly better, especially on objects that can be grasped in a variety of ways.
I. INTRODUCTION
Robotic grasp detection is difficult and slow, motivating a single-pass CNN that predicts grasp coordinates directly from an RGB-D image. The approach uses global image information and achieves higher accuracy at real-time speed.
- Visual perception is a bottleneck in real robotic systems because understanding scenes requires complex recognition and segmentation tasks.
- Robotic grasp detection seeks a good grasp from an RGB-D object view, but it lags behind human ability to grasp novel objects.
- 13.5 seconds per frame was the recent runtime on the Cornell Grasp Detection Dataset, creating a substantial delay before robot motion.
- Sliding-window grasp detection applies a classifier to numerous image patches and treats highly scoring patches as potential grasps.
- A single network pass predicts grasp coordinates directly, avoiding repeated patch classification and providing a performance boost.
- 88 percent accuracy and 13 frames per second are reported for the global-information network, establishing real-time RGB-D grasp detection.
II. RELATED WORK
Prior grasping methods use physical simulation or RGB-D machine learning, while this work changes the CNN processing pipeline to improve accuracy and speed. The problem is represented with five-dimensional grasp rectangles.
- 3-D simulation approaches can find grasps using full object models and physical information, but such models are often unavailable for novel objects.
- RGB-D grasp methods learn grasp features from data and can generalize to novel objects from a single view without a full physical model.
- The paper addresses sliding-window CNN grasp detection with a different architecture and processing pipeline designed for higher accuracy and faster speed.
- The five-dimensional grasp representation specifies the location and orientation of a parallel-plate gripper before closure.
- The representation simplifies a seven-dimensional grasp description by assuming a good 2-D grasp can be projected into 3-D for execution.
- Using five dimensions makes grasp detection analogous to object detection, with an additional term for gripper orientation.
IV. GRASP DETECTION WITH NEURAL NETWORKS
CNNs provide a strong basis for visual grasp detection, but sliding-window classification is computationally expensive. The proposed system uses a large CNN to make global predictions from the full object image.
- CNNs outperform other techniques in major computer-vision tasks and already work for grasp detection in sliding-window pipelines.
- The model avoids repeatedly running a small classifier on image patches by using a large convolutional network for global grasp prediction.
- The architecture derives from AlexNet, a widely adopted convolutional network for object recognition.
- The network contains five convolutional layers and three fully connected layers, with normalization and maxpooling between convolutional stages.
B. Direct Regression To Grasps
Direct regression maps an RGB-D image to grasp coordinates, while extensions combine grasp detection with object recognition and support multiple grasps. The direct model assumes one graspable object per image.
- B. Direct Regression To Grasps: Direct regression maps a raw RGB-D image to grasp coordinates through convolutional feature extraction and fully connected layers.
- B. Direct Regression To Grasps: The model outputs six values: four for location and height, plus sine and cosine of twice the grasp angle.
- B. Direct Regression To Grasps: The direct model assumes each image contains one graspable object and predicts one grasp, requiring segmentation beforehand outside experimental conditions.
- B. Direct Regression To Grasps: Randomly selecting a ground-truth grasp during training prevents fitting to one grasp and makes squared-error regression fit the average possible grasp.
- C. Regression + Classification: Recognition and grasp detection can be combined in one efficient pipeline by extending the model with object-category outputs.
- C. Regression + Classification: The combined model predicts an object category and a good grasp in one pass while retaining the direct regression model's speed.
D. MultiGrasp Detection
MultiGrasp extends direct grasp regression to predict multiple locally constrained grasps across an image grid. It combines grasp coordinates with per-cell feasibility scores and avoids heuristic bounding-box extraction.
- MultiGrasp divides the image into an NxN grid and predicts one grasp plus its feasibility likelihood per cell.Each predicted grasp must have its center within the cell that produces it.
- The model outputs NxNx7 values: one grasp-likelihood heatmap channel and six grasp-coordinate channels for each region.On Cornell, a 7x7 grid produces a 7x7x7 output layer with 343 neurons.
- During training, MultiGrasp samples up to five ground-truth grasps and backpropagates through the full heatmap plus cells containing grasps.Coordinate errors are not propagated through blank grid-cell entries.
- MultiGrasp predicts full bounding boxes directly and uses its heatmap to weight them, rather than extracting boxes through post-processing or heuristics.Its locally constrained cell structure distinguishes the approach from methods that predict unconstrained multiple boxes and confidence scores.
V. EXPERIMENTS AND EVALUATION
The experiments use Cornell’s varied but non-exhaustive grasp annotations and evaluate with cross-validation under image-wise and object-wise splits. The rectangle metric captures grasp geometry more fully than the point metric, while MultiGrasp visualizations select the highest-weight grasp for Cornell evaluation.
- The Cornell dataset contains 885 images of 240 objects, with multiple diverse grasp labels per image that are not exhaustive.Labels vary in orientation, location, and scale and target parallel plate grippers.
- The rectangle metric requires grasp-angle agreement within 30° and a Jaccard index above 25 percent.The experiments use this metric because it evaluates full grasp rectangles.
- The rectangle metric better discriminates good and bad grasps than the point metric, which ignores grasp angle and size.The point metric is also excluded because prior work does not disclose its threshold values.
- The study uses five-fold cross-validation with both image-wise and object-wise splits.Image-wise splits test new positions of known objects, whereas object-wise splits test generalization to novel objects.
- Image-wise and object-wise splitting produce comparable performance in practice.The authors suggest dataset similarity, such as slightly different sunglasses, may contribute to this result.
B. Object Classification
The classification experiment uses 16 manually assigned Cornell object categories and trains a combined model to predict both category and grasp. ImageNet pretraining is retained for RGB-D input by replacing the blue channel with depth information.
- The Cornell images are manually classified into 16 object categories, ranging from bottles and shoes to sporting equipment.Category sizes range from 20 to 156 images.
- The combined regression-plus-classification model simultaneously predicts the object category and its best grasp at test time.Classification accuracy is reported on the same cross-validation splits as the grasp experiments.
- ImageNet pretraining is used before grasp training because it improves training time and helps avoid overfitting.The paper emphasizes its value when labeled RGB-D grasp data are limited.
- RGB-D input is adapted for AlexNet by replacing the blue image channel with depth information.This preserves the ability to pretrain the full network instead of adding another input channel.
D. Training
Training uses fixed optimization and regularization settings across models, GPU execution, minimal RGB-D preprocessing, and extensive geometric data augmentation. The timing results depend on GPU use.
- Each model is trained for 25 epochs per cross-validation fold with a 0.0005 learning rate and 0.001 weight decay.Dropout with probability 0.5 is applied between fully connected hidden layers.
- The models run with cuda-convnet2 on an nVidia Tesla K20 GPU.The paper states that timing results depend on using a GPU.
- Depth values are normalized to 0–255, missing depth pixels are set to 0, and 144 is globally subtracted for approximate mean-centering.Depth is placed in the blue image channel before inference.
- Training augmentation randomly translates and rotates 320x320 crops, resizes them to 224x224, and generates 3000 examples per original image.Test images use a centered crop and resize without translation or rotation.
VI. RESULTS
The models improve grasp detection through direct regression, combined classification, and MultiGrasp prediction. They achieve strong accuracy and real-time speed, while MultiGrasp addresses failures caused by averaging possible grasps.
- Direct regression: Around 85 percent accuracy in both image-wise and object-wise splits exceeds the previous best by ten percentage points.The direct regression model processes at 13 fps, with the reported speedup attributed to single-pass prediction and GPU acceleration.
- Direct regression: The direct regression model can produce incorrect grasps when average ground-truth grasps are not viable for an object.This failure occurs mainly for circular objects such as flying discs; some visually incorrect examples may still be viable.
- Combined regression and classification: The combined regression + classification model predicts object categories and grasps without sacrificing detection accuracy.It correctly classifies previously seen objects 9 out of 10 times and novel objects more than 60 percent of the time.
- MultiGrasp: MultiGrasp outperforms direct regression by addressing errors from bad average grasps while maintaining real-time operation.It achieves 88 percent grasp detection accuracy at 13 frames per second.
VII. DISCUSSION
The discussion contrasts fast, globally informed grasp prediction with its averaging failure mode and introduces MultiGrasp to combine global context with locally constrained predictions. It also highlights pretraining requirements and the unresolved evaluation of multiple grasps per image.
- Efficiency: A CPU could run the model in far less than a second per image, supporting real-time robotics applications.The GPU-based system benefits from CNN throughput, while test-time CPU latency is expected to remain below one second per image.
- Efficiency: Single-pass prediction exploits a data constraint to achieve accurate grasp prediction without repeated image processing.The model applies once to the image rather than using numerous local evaluations.
- Model trade-offs: Global information improves accuracy over local sliding-window reasoning but can average several good grasps into one bad grasp.The direct regression model may predict a grasp between multiple valid alternatives, such as at the center of a circular object.
- MultiGrasp: MultiGrasp combines whole-image context with local predictions, avoiding bad grasps that fall between several good alternatives.Its locally constrained cells preserve the advantages of global models while reducing averaging effects.
- Evaluation scope: MultiGrasp can predict multiple grasps per image, but current datasets lack an appropriate quantitative evaluation for that capability.The authors propose future evaluation in a multiple-grasp detection task or on a standard object-detection dataset.
- Training: Pretraining on ImageNet is important because, without it, the large CNNs quickly overfit instead of learning meaningful grasp representations.Features tuned for the blue image channel also transfer effectively to depth information.
VIII. CONCLUSION
The paper presents a fast, accurate RGB-D grasp-prediction system that improves state-of-the-art performance and combines global information with local prediction. MultiGrasp achieves the best known performance on the Cornell Grasping Dataset while supporting combined grasp detection and object classification.
- Conclusion: The system predicts robotic grasps from RGB-D images while improving state-of-the-art performance and running more than 150 times faster than previous methods.It is designed as a fast and accurate grasp-prediction system.
- Conclusion: MultiGrasp achieves the best known performance on the Cornell Grasping Dataset by combining global information with a local prediction procedure.The conclusion identifies this combination as the basis of its reported performance.
- Conclusion: Grasp detection and object classification can be combined without sacrificing accuracy or performance.