Source-linked AI summary
Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network
Sulabh Kumra, Shirin Joshi, Ferat Sahin
TL;DR
Robots need fast, robust grasp inference that transfers to unknown objects. This paper presents a modular system using GR-ConvNet to generate antipodal grasps from n-channel scene images, achieving high benchmark accuracy and physical-robot success rates.
Problem
Robotic grasping requires a generalized technique to infer fast and robust grasps for unknown objects and transfer learned knowledge to novel real-world objects.
Method
A modular system uses GR-ConvNet to generate pixel-wise antipodal grasps from n-channel images, then plans and executes them through separate inference and control modules.
Results
97.7% and 94.6% state-of-the-art accuracy were achieved on Cornell and Jacquard, while household and adversarial objects reached 95.4% and 93% grasp success, respectively.
Takeaways & Limitations
The system predicts and performs accurate grasps for previously unseen objects, including objects in clutter, and its low inference time suits closed-loop robotic grasping.
Abstract
from arXiv · showhide
In this paper, we present a modular robotic system to tackle the problem of generating and performing antipodal robotic grasps for unknown objects from n-channel image of the scene. We propose a novel Generative Residual Convolutional Neural Network (GR-ConvNet) model that can generate robust antipodal grasps from n-channel input at real-time speeds (~20ms). We evaluate the proposed model architecture on standard datasets and a diverse set of household objects. We achieved state-of-the-art accuracy of 97.7% and 94.6% on Cornell and Jacquard grasping datasets respectively. We also demonstrate a grasp success rate of 95.4% and 93% on household and adversarial objects respectively using a 7 DoF robotic arm.
I. INTRODUCTION
The paper presents a modular system for generating, planning, and executing antipodal grasps on unknown objects, using GR-ConvNet to infer multiple grasp poses from n-channel images. It reports state-of-the-art dataset accuracy and high success rates on household and adversarial objects.
- Motivation: The approach targets rapid and robust grasp transfer to novel real-world objects encountered by robots.The introduction identifies knowledge transfer to novel objects as the major challenge in generalized robotic grasping.
- Proposed model: GR-ConvNet generates antipodal grasps for every pixel in an n-channel input image.The generative formulation produces quality, angle, and width images used to infer grasp rectangles, including multiple grasps for multiple objects in one shot.
- System overview: The system combines an inference module that predicts grasp poses with a control module that plans and executes robot trajectories.The inference module processes scene images, while the control module uses the resulting poses for antipodal grasp execution.
- Contributions: The paper contributes a modular robotic system that predicts, plans, and performs antipodal grasps for objects in the scene.The implementation of the inference and control modules was open-sourced.
- Evaluation: 97.7% and 94.6% state-of-the-art accuracy were achieved on the Cornell and Jacquard grasping datasets, respectively.These evaluations use publicly available grasping datasets.
- Evaluation: 95.4% and 93% grasp success rates were demonstrated on household and adversarial objects, respectively.The results demonstrate deployment of the model on a robotic arm at real-time speeds.
II. RELATED WORK
Prior robotic grasping research spans engineered features, supervised and deep learning, generative methods, and multimodal RGB-D approaches. The paper positions its method as an advance over these approaches for grasping unknown objects.
- Robotic grasping: Early robotic grasping research studied contact mechanics and used hand-engineered features for stable or multifinger grasps.Hand-engineering was described as tedious and time-consuming but potentially useful for learning multifinger grasps.
- Robotic grasping: Supervised learning approaches for novel objects included synthetic-data training but were limited to office, kitchen, and dishwasher environments.The cited prior work used supervised learning trained on synthetic data.
- Deep learning for grasping: Deep neural networks learned grasping features directly from image data, with larger datasets improving generalization in one cited CNN approach.Affordable RGB-D sensors and GPU-based computation supported these methods.
- Generative grasping: Generative grasping methods predicted pixel-wise grasp poses from depth images rather than selecting only among discrete grasp candidates.The related work describes this approach as reducing shortcomings of discrete grasping methods.
- Multimodal grasping: Multimodal approaches used RGB-D or separate modality features, including two-step methods that narrowed the search before computing an optimal grasp rectangle.Some methods treated multimodal information jointly, while others used separate features or sequential learning stages.
- Positioning: The paper presents its work as advancing related robotic-grasping methods for unknown objects.Table I is identified as a comparison with recent related work.
III. PROBLEM FORMULATION
The paper formulates robotic grasping as predicting and executing antipodal grasps for unknown objects from an n-channel scene image. Grasps are represented in image and robot frames using position, angle, width, quality, and coordinate transformations.
- Robotic grasping is defined as predicting antipodal grasps for unknown objects from an n-channel scene image and executing them on a robot.
- The robot-frame grasp pose uses tool-tip position, rotation around the z-axis, required tool width, and grasp quality.
- The image-space grasp representation uses grasp center, camera-frame rotation, required width, and grasp quality at each image location.The grasp quality score ranges from 0 to 1, with values near 1 indicating greater grasp-success likelihood.
- The grasp angle represents antipodal rotation, while grasp width is measured in pixels up to the antipodal gripper’s maximum width.The angle lies in [−π/2, π/2], and width lies in [0,Wmax] pixels.
- Image-space grasp poses are transformed into robot coordinates through image-to-camera and camera-to-robot transformations using calibration parameters.Tci maps image space to camera 3D space using camera intrinsics, while Trc uses camera-pose calibration to convert camera space into robot space.
- The notation extends to multiple grasps by collecting the per-pixel angle, width, and quality outputs into three images.
IV. APPROACH
The proposed system separates grasp prediction from robotic execution. An inference module generates grasp poses, and a control module plans and executes trajectories for antipodal grasping.
- The dual-module system predicts, plans, and performs antipodal grasps for objects in the scene.
- The inference module predicts suitable grasp poses, while the control module uses them to plan and execute robot trajectories.
- The system architecture separates perception-based grasp inference from downstream robotic control.
A. Inference module
The inference module preprocesses n-channel scene images and uses GR-ConvNet to generate pixel-wise grasp representations. Its residual, lightweight architecture supports generalized input modalities and real-time robotic control.
- Inference module: The inference pipeline crops, resizes, and normalizes input images, inpainting depth when available before forming a 224 × 224 n-channel input.
- Inference module: GR-ConvNet accepts n-channel images without restricting the input to depth-only or RGB-only modalities.This design is intended to generalize across input modalities.
- Control module: The control module requests the highest-quality predicted grasp, transforms it into robot coordinates, and uses it for trajectory planning.
- Model architecture: GR-ConvNet generates pixel-wise grasp outputs representing quality, angle, and end-effector width from an n-channel image.The architecture uses convolutional layers, five residual layers, and transposed convolutions to produce the grasp images.
- Model architecture: The grasp angle is represented with cos2Θ and sin2Θ, whose combined values form the required angle for antipodal grasping.
- Model architecture: Residual layers are used to improve identity-function learning while avoiding degradation associated with excessively deep convolutional networks.
- Model architecture: 1,900,900 parameters make the network comparatively lightweight and suitable for closed-loop control at up to 50 Hz.
D. Training methodology
The model learns an end-to-end mapping from scene images and dataset objects to generated grasps by minimizing negative log-likelihood. Training uses Adam and smooth L1 loss to address exploding gradients.
- The model is trained end-to-end to learn a mapping from input scene images and dataset objects to generated grasps.
- Training minimizes the negative log-likelihood of the generated grasp conditioned on the input scene image.
- The models use Adam optimization, standard backpropagation, mini-batch SGD, a 10^-3 learning rate, and mini-batches of 8.Results are averaged across three random seeds.
- Smooth L1, also called Huber loss, was selected after trials because it handled exploding gradients best.
- The loss compares the grasp generated by the network with the ground-truth grasp.
V. EVALUATION
The evaluation uses Cornell and Jacquard antipodal grasping datasets, with augmentation compensating for Cornell’s smaller scale and rectangle-based validity criteria enabling comparison.
- Cornell contains 1,035 RGB-D images of 240 objects, with 5,110 positive and 2,909 negative grasps.
- Cornell training data was expanded to 51k grasp examples through random crops, zooms, and rotations, using only positive grasps.
- Jacquard contains 54k RGB-D images and 1.1M grasp examples from simulated grasp attempts, so no augmentation was performed.
- A grasp is valid when predicted and ground-truth rectangles have more than 25% IoU and orientation offset below 30°.
- Pixel-wise image-based grasp outputs are mapped to rectangle representations so the model can be evaluated with the rectangle metric.
VI. EXPERIMENTS
Experiments assess the system on standard datasets, household objects, adversarial objects, and clutter using an RGB-D camera and a 7-DoF Baxter robot.
- The evaluation covers two standard datasets, household objects, adversarial objects, and objects arranged in clutter.
- Real-world experiments used an Intel RealSense D435 RGB-D camera, a 7-DoF Baxter robot, and a two-fingered parallel gripper.
- Execution times were measured on an Intel Core i7-7800X system with an NVIDIA GeForce GTX 1080 Ti and CUDA 10.
- Thirty-five household objects were each tested in 10 positions and orientations, producing 350 grasp attempts across varied shapes, sizes, and geometries.
C. Adversarial test objects
The experiments examine generalization across difficult objects and clutter, while qualitative and modality results show how GR-ConvNet produces grasp representations and performs across inputs.
- C. Adversarial test objects: Ten adversarial objects with complex, indefinite geometry were tested in 100 grasp attempts across 10 orientations and positions each.
- C. Adversarial test objects: Ten clutter runs used 60 unseen objects, with each run ending when no objects remained in the camera’s field of view.
- VII. RESULTS: GR-ConvNet generated quality, angle, and width images that were converted into grasp rectangles for unseen objects and cluttered scenes.
- VII. RESULTS: RGB-D input outperformed uni-modal RGB-only and depth-only inputs because multiple modalities enabled better learning of input features.
- VII. RESULTS: Cornell evaluation reached 97.7% accuracy on the image-wise split and 96.6% on the object-wise split using RGB-D data.
- VII. RESULTS: The per-image execution time suggests suitability for real-time closed-loop applications.
B. Jacquard Dataset
On Jacquard, the model is trained without augmentation and achieves state-of-the-art RGB-D accuracy, while physical experiments test generalization to household, adversarial, and cluttered objects.
- B. Jacquard Dataset: Jacquard training used 90% of images and validation used the remaining 10%; no augmentation was required because the dataset was large.
- B. Jacquard Dataset: 94.6% accuracy was obtained on Jacquard using RGB-D input, outperforming other methods on the dataset.
- B. Jacquard Dataset: Household testing achieved 95.4% accuracy from 334 successful grasps out of 350 attempts, while adversarial testing achieved 93 successful grasps out of 100.
- B. Jacquard Dataset: The model generalized to unseen objects and generated grasps for all tested objects except a transparent bottle.
- B. Jacquard Dataset: Clutter experiments achieved 93.5% grasp success while predicting multiple antipodal grasps for multiple objects.
E. Failure case analysis
Failures were uncommon but concentrated in low-quality grasps, slippage, gripper misalignment, and transparent objects affected by unreliable depth data.
- A few failures involved extremely low grasp scores or objects slipping from the closed gripper.
- Depth inaccuracies and collisions with nearby objects could cause gripper misalignment during grasping.
- The transparent bottle was the main reported object for which the model could not produce a good grasp.
- Future work targets improved depth prediction for reflective objects and support for additional gripper types.