Source-linked AI summary
Robotic Grasp Detection using Deep Convolutional Neural Networks
Sulabh Kumra, Christopher Kanan
TL;DR
Robotic grasp detection still lacks reliable real-time performance for novel objects. This paper predicts parallel-plate gripper poses from RGB-D images with multimodal deep convolutional networks, achieving 89.21% accuracy on the Cornell Grasp Dataset and real-time speeds.
Problem
Robotic grasp detection for novel objects remains an unsolved problem, particularly for real-time detection and planning.
Method
The system uses parallel deep residual networks to extract RGB and depth features, merges them, and predicts a five-dimensional grasp configuration with a convolutional network.
Results
89.21% accuracy was achieved by the multimodal grasp predictor on the standard Cornell Grasp Dataset, with real-time operation reported.
Takeaways & Limitations
The models predict graspability and grasp configuration for previously unseen objects while improving state-of-the-art performance on the Cornell dataset.
Takeaways & Limitations
The depth-processing ResNet models were initialized from RGB-trained ImageNet models because a pre-trained depth-specific ResNet was unavailable.
Abstract
from arXiv · showhide
Deep learning has significantly advanced computer vision and natural language processing. While there have been some successes in robotics using deep learning, it has not been widely adopted. In this paper, we present a novel robotic grasp detection system that predicts the best grasping pose of a parallel-plate robotic gripper for novel objects using the RGB-D image of the scene. The proposed model uses a deep convolutional neural network to extract features from the scene and then uses a shallow convolutional neural network to predict the grasp configuration for the object of interest. Our multi-modal model achieved an accuracy of 89.21% on the standard Cornell Grasp Dataset and runs at real-time speeds. This redefines the state-of-the-art for robotic grasp detection.
I. INTRODUCTION
The paper addresses real-time robotic grasp detection for novel objects from RGB-D imagery, using a five-dimensional grasp representation and deep multimodal convolutional networks. It combines separate RGB and depth feature extraction with a prediction network and evaluates the approach on the Cornell Grasp Dataset.
- Motivation: Robotic grasp detection remains challenging because real-time grasp detection and planning are not yet reliably achieved.The task uses sensor information to detect graspable objects and map image pixels to real-world coordinates needed for subsequent grasping steps.
- Problem: The target is detecting a good grasp for a novel object from an RGB-D scene image.The system predicts the grasp pose of a parallel-plate gripper before execution.
- Representation: The five-dimensional grasp representation specifies gripper position, orientation, and plate geometry while reducing computational cost and supporting RGB-D detection.Its representation can be projected back to a seven-dimensional configuration for execution on a real robot.
- Approach: The proposed model uses parallel ResNet-based networks for RGB and depth, merges their features, and predicts grasp configuration with another convolutional network.The study also compares the multimodal model with other methods and a RGB-only variation on the standard Cornell Grasp Dataset.
- Prior work: Prior approaches included sliding-window classification, but their repeated patch processing could take 13.5 seconds per frame and relied on computationally expensive or narrow settings.Other methods used known object models, few objects, or restricted object categories, limiting general-purpose grasp detection.
III. PROBLEM FORMULATION
The problem is formulated as finding a successful grasp configuration from an object image. The paper uses a five-dimensional planar representation whose fixed gripper dimensions can be projected to a seven-dimensional executable configuration.
- Problem formulation: Grasp detection is formulated as finding a successful grasp configuration g for an object image I.The formulation treats grasp detection as predicting a configuration from visual input.
- Representation: The five-dimensional configuration encodes the grasp rectangle center, plate height, maximum plate separation, and orientation.The center is (x,y), while h and w are typically fixed for a specific robot EOAT.
- Execution: Planar grasps are used because the five-dimensional configuration can be projected back to a seven-dimensional configuration for execution on a real robot.This reduction supports grasp detection from RGB-D images while retaining a path to robot execution.
IV. APPROACH
The approach makes a single grasp prediction directly from the complete RGB-D image rather than classifying many image patches. It uses residual-network depth and a one-object assumption to support the model design.
- Prediction: The model directly feeds the complete RGB-D image into a deep convolutional network for single-step grasp prediction.This replaces repeated classification of small patches and reduces processing overhead.
- Residual architecture: ResNet uses skip-connected residual blocks so convolutional layers learn residual mappings while identity information bypasses them.The architecture addresses optimization difficulties associated with increasing network depth.
- Assumption: The approach assumes that each input image contains only one graspable object and requires one grasp prediction.This permits a global prediction from the complete image, but images with multiple objects would require region division first.
A. Architecture
The architecture uses ResNet-50 residual learning for feature extraction and distinguishes uni-modal from multi-modal grasp predictors.
- A. Architecture: ResNet-50 replaces eight-layer AlexNet with a fifty-layer residual model for grasp detection.Residual learning uses skip connections that bypass convolutional and nonlinear layers while adding learned residuals to block inputs.
- A. Architecture: Residual blocks add a learned mapping to the block input through skip connections.In H_k = F(H_k−1,W_k)+H_k−1, H_k−1 is the input, H_k the output, and W_k the learned weights.
- A. Architecture: The uni-modal predictor uses one image modality, whereas the multi-modal predictor uses RGB and depth information.The paper describes these as 2D and 3-D grasp predictors, respectively.
B. Uni-modal Grasp Predictor
The uni-modal predictor uses a pre-trained ResNet-50 to extract features from a three-channel image, while the multi-modal design processes RGB and depth streams separately before prediction.
- B. Uni-modal Grasp Predictor: The uni-modal model feeds RGB or RGD three-channel images through an ImageNet-pre-trained ResNet-50 feature extractor.A linear SVM is used as a baseline classifier for predicting grasp configuration.
- B. Uni-modal Grasp Predictor: During uni-modal training, ResNet-50 convolutional weights remain fixed while the final two fully connected layers are tuned.The final layers are initialized with Xavier weight initialization.
- B. Uni-modal Grasp Predictor: The multi-modal predictor processes RGB and depth as separate three-channel inputs using two independent pre-trained ResNet-50 models.Their L2-normalized features are concatenated and passed to a shallow convolutional network.
- B. Uni-modal Grasp Predictor: A linear SVM baseline uses the L2-normalized RGB and depth DCNN features to predict grasp configuration.
V. EXPERIMENTS
Experiments evaluate grasp prediction on the Cornell Grasp Dataset using five-fold cross-validation under image-wise and object-wise splits.
- V. EXPERIMENTS: The Cornell Grasp Dataset contains 885 images of 240 objects and 8019 labeled grasps.The labels include 5110 positive and 2909 negative grasps selected for parallel-plate grippers.
- V. EXPERIMENTS: Five-fold cross-validation is used for all experiments.
- V. EXPERIMENTS: Image-wise splitting tests generalization to previously seen objects in different positions and orientations.
- V. EXPERIMENTS: Object-wise splitting tests generalization to objects not seen during training.
B. Data Pre-processing
Preprocessing resizes grasp-centered patches, scales depth values, replaces occluded depth values, and uses ImageNet pre-training when domain-specific data are limited.
- B. Data Pre-processing: Input grasp-centered patches are resized to 224×224 for ResNet-50.
- B. Data Pre-processing: Depth images are rescaled to the range 0 to 255 before entering the DCNN.
- B. Data Pre-processing: NaN pixels caused by stereo-image occlusions are replaced by zeros.
- C. Pre-training: ResNet-50 is pre-trained on ImageNet because the Cornell grasp dataset provides limited domain-specific data.The pre-training process is intended to provide visual features before target-task training.
D. Training
The models are trained and evaluated using staged optimization and the rectangle grasp metric, which incorporates both angle and overlap criteria.
- Implementation: Experiments were implemented with Keras on Theano using a CUDA-enabled NVIDIA GeForce GTX 645 GPU.
- Training: Training uses two stages: shallow-network training followed by end-to-end optimization of the complete network.The supplied training passage specifies SGD and hyperparameters for the first stage.
- Evaluation: The Cornell dataset evaluation compares grasp prediction accuracy using the stated rectangle-metric protocol.
- Evaluation metric: The rectangle metric considers a grasp good when angle error is below 30° and Jaccard similarity exceeds 25%.This evaluates the complete predicted grasp rectangle rather than only its center point.
- Evaluation metric: The experiments select the best-scored grasp rectangle using the rectangle metric.
VI. RESULTS
On the Cornell RGB-D grasp dataset, the proposed models outperform prior methods in accuracy and speed, while the modified multi-modal model also predicts graspability.
- Results: Both proposed models outperform prior robotic grasp detection algorithms in accuracy and speed on Cornell RGB-D data.Image-wise and object-wise splits test generalization to different grasp features.
- Accuracy: 88.84% accuracy was achieved by the RGB uni-modal predictor, while the baseline uni-modal predictor achieved 84.76%.
- Input modalities: Replacing the blue channel with depth did not help because the ResNet features were learned from RGB images rather than RGD inputs.
- Accuracy: 89.21% accuracy makes the multi-modal grasp predictor the reported state of the art for RGB-D robotic grasp detection.The model uses RGB-D data and outperformed the tested VGG16 alternative.
- Speed: The uni-modal grasp predictor runs 800 times faster than Lenz et al.'s two-stage SAE model.The paper attributes the speed increase mainly to replacing sliding-window classification with a single-pass model.
- Graspability: 93.4% accuracy was achieved for graspability prediction after replacing the final fully connected layer with a binary-output dense layer.The modified model used softmax activation and matched the reported state of the art.
VII. DISCUSSION
The discussion reports improved grasp prediction for unseen objects and illustrates cases where multi-modal predictions are more viable, while identifying a depth-pretraining limitation.
- Discussion: Deep convolutional neural networks predict both graspability and grasp configuration for objects unseen during training.
- Discussion: The multi-modal model performs best with RGB-D data and the object-wise dataset split.The uni-modal model performs best with RGB data and the image-wise split.
- Qualitative comparison: In illustrated cases, the multi-modal predictor produces viable grasps where the uni-modal predictor fails to produce one.Some uni-modal grasps may still be feasible, but the multi-modal rectangles are described as clearly successful.
- Limitation: The multimodal depth branch uses ImageNet-pretrained ResNet-50 because no pretrained ResNet model for depth data was available.The authors state that RGB-only pretraining may not provide depth-specific features.
VIII. CONCLUSION
The paper presents a multi-modal robotic grasp detection system for predicting grasp configurations of novel objects with RGB-D images. Its models improve performance on the Cornell Grasping Dataset and operate at real-time speeds, while physical-robot execution remains future work.
- The system predicts graspability and grasp configurations for novel objects using RGB-D images and a parallel-plate robotic gripper.
- Deep residual networks extract features from multi-modal inputs, which fully connected layers use to output the grasp configuration.
- The models improve state-of-the-art performance on the Cornell Grasping Dataset and run at real-time speeds.
- Applying transfer learning to perform grasps with a physical robot is identified as future work.