Source-linked AI summary
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach
Douglas Morrison, Peter Corke, Jürgen Leitner
TL;DR
Existing grasp-synthesis methods are often too slow and candidate-based for closed-loop operation in dynamic settings. The paper introduces GG-CNN, which directly predicts pixelwise grasp poses and quality from depth images, achieving strong dynamic-scene success rates while remaining lightweight. Its scope is bounded by sensing and gripper limitations, including unreliable depth for black or reflective objects and difficulty grasping objects below 15 mm.
Problem
Candidate-based deep-learning grasp synthesis can require one to tens of seconds, limiting closed-loop grasping in dynamic environments.
Method
GG-CNN is a fully convolutional, object-independent model that directly generates grasp maps from depth images on a pixelwise basis.
Results
83% and 88%: closed-loop grasp success rates for moved adversarial and household objects, with 81% for dynamic clutter.
Takeaways & Limitations
GG-CNN supports closed-loop grasping at up to 50 Hz and achieves state-of-the-art results on unknown dynamic objects, including dynamic clutter.
Takeaways & Limitations
The RealSense camera provides invalid or inaccurate depth for many black or reflective objects, and the gripper cannot grasp objects below 15 mm reliably.
Abstract
from arXiv · showhide
This paper presents a real-time, object-independent grasp synthesis method which can be used for closed-loop grasping. Our proposed Generative Grasping Convolutional Neural Network (GG-CNN) predicts the quality and pose of grasps at every pixel. This one-to-one mapping from a depth image overcomes limitations of current deep-learning grasping techniques by avoiding discrete sampling of grasp candidates and long computation times. Additionally, our GG-CNN is orders of magnitude smaller while detecting stable grasps with equivalent performance to current state-of-the-art techniques. The light-weight and single-pass generative nature of our GG-CNN allows for closed-loop control at up to 50Hz, enabling accurate grasping in non-static environments where objects move and in the presence of robot control inaccuracies. In our real-world tests, we achieve an 83% grasp success rate on a set of previously unseen objects with adversarial geometry and 88% on a set of household objects that are moved during the grasp attempt. We also achieve 81% accuracy when grasping in dynamic clutter.
I. INTRODUCTION
Robotic grasping in unstructured, dynamic environments requires generalizing to unseen objects while tolerating perception and control errors. GG-CNN addresses slow, candidate-based grasp synthesis with direct pixelwise generation for fast closed-loop execution.
- Robots must compute grasps for nearly unlimited objects and act despite workspace changes, perception noise, control inaccuracies, and perturbations.
- Candidate-sampling deep-learning methods typically require about one to tens of seconds, limiting their use in closed-loop grasp execution.
- GG-CNN directly generates antipodal grasp poses and quality measures for every pixel in an input depth image instead of sampling candidates.
- 19 ms: GG-CNN executes on a GPU-equipped desktop computer, enabled by orders-of-magnitude fewer parameters than other grasp-synthesis CNNs.
- 83% and 88%: closed-loop dynamic grasp success rates on adversarial and household object sets, respectively.
- The study also addresses reproducibility by proposing 20 test items spanning adversarial geometries and household benchmark objects.
III. GRASP POINT DEFINITION
The paper represents grasps in image coordinates using a center pixel, rotation, and perceived width, then maps the selected image-space grasp into world coordinates. Each pixel therefore carries the information needed to evaluate and execute a grasp.
- A grasp comprises Cartesian position, rotation around the z-axis, gripper width, and a scalar quality measure representing grasp-success chances.
- For a depth image, the image-space grasp is represented by center pixel s=(u, v), image-frame rotation ˜φ, and image-coordinate width ˜w.
- The image-space grasp is converted into world coordinates using transforms from image coordinates to the camera frame and from the camera frame to the robot frame.
- The grasp map stores rotation, width, and quality values at every image pixel, enabling direct selection of the best visible grasp.
IV. GENERATIVE GRASPING CONVOLUTIONAL NEURAL NETWORK
The paper formulates grasp synthesis as learning a neural approximation from depth images to grasp maps. Training uses paired inputs and outputs with an L2 loss.
- A neural network Mθ approximates the complex mapping M from input images I to grasp maps G.The network parameters are represented by θ.
- Mθ(I) = (Qθ, Φθ, Wθ) is learned from training pairs IT and GT using an L2 loss function.
A. Grasp Representation
Each pixel stores a candidate grasp’s quality, angle, and gripper width. Together, these images represent the grasp map used for synthesis.
- The grasp map G represents grasp parameters for grasps executed at the Cartesian point corresponding to each pixel.
- Q assigns each pixel a grasp quality from 0 to 1, with values closer to 1 indicating a higher chance of success.
- Φ stores the grasp angle at each pixel, using the antipodal grasp’s symmetry around ±π/2.
- W stores the gripper width at each pixel in the depth-invariant range [0, 150] pixels.The pixel width can be converted to a physical measurement using camera parameters and measured depth.
B. Training Dataset
The training data are derived from Cornell grasp annotations and converted into pixelwise grasp-map images. Augmentation expands the dataset, while quality, angle, and width labels encode positive grasps.
- The Cornell Grasping Dataset provides 885 RGB-D images with 5110 positive and 2909 negative human-labelled grasps.
- 8840 augmented depth images and associated grasp maps are created through random crops, zooms, and rotations, incorporating 51,100 grasp examples.
- Ground-truth grasp rectangles are converted into image masks using their centre third, which represents the gripper’s centre position.
- Positive grasps set the corresponding quality-map regions QT to 1, while all other pixels are set to 0.
- Grasp angles are encoded as sin(2ΦT) and cos(2ΦT) to represent antipodal grasps without angle-wrap discontinuities.
- Depth images are inpainted to remove invalid values and mean-centered to provide depth invariance.
C. Network Architecture
The GG-CNN is a fully convolutional, compact network that directly predicts grasp maps from depth images. Its lightweight design supports rapid inference and evaluation across varied experimental objects and configurations.
- Network Architecture: The fully convolutional GG-CNN directly approximates the grasp map from an input depth image.
- Network Architecture: The network represents inputs and outputs as 300×300 images and reconstructs grasp angles from predicted unit-vector components.
- Network Architecture: 62,420 parameters make the final GG-CNN smaller than grasp-candidate classification networks containing hundreds of thousands or millions of parameters.
- Network Architecture: 95 network configurations are trained for 100 epochs each, and evaluated on 1710 augmented images to select the best configuration.
- Physical Components: The grasping experiments use a Kinova Mico 6DOF robot with a two-fingered gripper and a wrist-mounted RGB-D camera.
- Network Architecture: 6 ms is required to compute a single depth image, while the entire grasping pipeline takes 19 ms on the reported platform.
1) Limitations:
The system uses defined object sets and a three-stage grasp pipeline, but its sensing and gripper hardware impose concrete operating boundaries.
- Limitations: The RealSense camera produces inaccurate depth below 150 mm and invalid data on many black or reflective objects.Closed-loop trials therefore stop updating the target grasp pose at approximately 70 mm from the object.
- Limitations: The KG-2 gripper cannot grasp objects shorter than 15 mm, especially thin cylindrical objects such as pens.The gripper’s finger compliance and slight tip splay prevent reliable grasping of these objects.
- Test objects: The benchmark uses eight adversarial 3D-printed objects and twelve household objects selected from existing grasping datasets.The adversarial set emphasizes complex, curved, and inclined geometries; the household set varies object size, shape, and difficulty.
- Test objects: Objects that are too small, too large, or too heavy for the robot are excluded from the experiments.This scope restriction supports comparison with related work using similar object classes.
- Pipeline: The grasp detection pipeline crops and rescales depth images, inpaints invalid values, evaluates GG-CNN, filters quality maps, and computes the best pose.The selected pose uses the maximum quality pixel and the corresponding predicted rotation and width.
D. Grasp Execution
The system evaluates both open-loop execution and closed-loop visual servoing, with the latter processing depth images in real time for dynamic grasping.
- Grasp Execution: The experiments compare single-view open-loop grasp execution with a closed-loop Position Based Visual Servoing controller.The closed-loop method is used to evaluate grasping in dynamic environments.
- Open-loop execution: The open-loop procedure computes one grasp from a captured depth image, moves to a pre-grasp pose, descends, closes the gripper, and lifts the object.Execution stops when the grasp pose is reached or a collision is detected through force feedback.
- Closed-loop execution: Depth images are generated at 30 Hz and processed by the grasp pipeline to generate grasp poses during closed-loop execution.The controller is initialized with the camera approximately 400 mm above the table.
- Closed-loop execution: The controller represents the grasp and gripper poses as 6D Cartesian position and Euler-angle vectors and generates a 6D end-effector velocity.The velocity scale λ causes the gripper to converge toward the grasp pose.
- Grasp completion: The gripper fingers are velocity-controlled toward the computed width, and success requires lifting the object back to its starting position.Control stops when the grasp pose is reached or a collision is detected.
- Trial setup: Objects are shaken in a box before each attempt to reduce bias from object pose, within an approximately 250×300 mm reachable workspace.The workspace supports vertical grasps permitted by the robot’s kinematics.
VI. EXPERIMENTS
The experiments cover static, dynamic, and cluttered grasping with more than 2000 attempts, using reproducible object sets and defined object motions.
- Experiments: More than 2000 grasp attempts evaluate static, dynamic, and cluttered scenarios while aiming to reproduce comparable prior experiments.The study also defines its own object sets and dynamic motions for reproducibility.
- Experiments: The evaluation proceeds from singulated static objects to moved objects and dynamic clutter, with comparisons summarized in Table II.The experiments are designed to assess both conventional grasping performance and the method’s dynamic-scene capability.
- A. Static Grasping: 84% and 81% grasp success rates were obtained on adversarial objects with open-loop and closed-loop methods under static conditions.The corresponding trial counts were 67/80 and 65/80.
- A. Static Grasping: 92% and 91% grasp success rates were obtained on household objects with open-loop and closed-loop methods under static conditions.The corresponding trial counts were 110/120 and 109/120.
- A. Static Grasping: Table II comparisons are indicative because the studies use different item sets and experimental protocols.The table notes that the compared network in [21] was trained on the adversarial objects, unlike the authors’ approach.
- B. Dynamic Grasping: 83% and 88% closed-loop success rates were achieved when adversarial and household objects were moved during grasp attempts.These dynamic results were not significantly different from the corresponding static results and fell within their 95% confidence bounds.
- B. Dynamic Grasping: The dynamic experiments do not directly compare against an open-loop method because object motion moves objects too far from their original positions for successful grasps.This comparison boundary applies specifically to the moving-object condition.
C. Dynamic Grasping in Clutter
The method is evaluated in cluttered scenes, including scenes with object motion, and is compared with a closely related clutter-grasping controller.
- C. Dynamic Grasping in Clutter: The clutter experiments recreate prior work using objects selected to be as close as possible to its test set.The GG-CNN was not trained on cluttered environments.
- Cluttered objects: 100% success was achieved on ten isolated test objects, compared with 98% (39/40) in [32].Four grasps were performed on each object.
- Cluttered objects: 87% success was achieved in static clutter, compared with 89% (66/74) in [32].The experiment ran for ten trials while removing successfully grasped objects from the pile.
- Dynamic cluttered objects: 81% success was achieved in dynamic clutter, compared with 77% (58/75) in [32].The dynamic condition added random object motion of at least 100 mm translation and 25° rotation during each grasp attempt.
- Dynamic cluttered objects: Only a 5% performance decrease occurred from static to dynamic clutter, compared with 12% in [32].In the other eight runs, all available objects were grasped with two or fewer failed grasps.
- Dynamic cluttered objects: An open-loop baseline in [32] fell from 95% success in static clutter to 23% in dynamic clutter.The cited explanation is that the baseline cannot react to changed item locations.
D. Robustness to Control Errors
The study evaluates closed-loop grasping under simulated kinematic errors and finds it more robust than open-loop control. Performance remains limited by the camera’s inability to update grasp poses near the object.
- Experimental setup: The simulated error model introduces cross-correlations between Cartesian x, y, and z velocities, with coefficients sampled independently at each grasp attempt.The model approximates kinematic error independently of the robot’s specific kinematic model, making its effect on end-effector positioning deterministic and easier to reproduce.
- Experimental setup: The experiment compares open- and closed-loop grasping across adversarial and household objects under four simulated control-error levels.Each object receives 10 grasp attempts for σ = 0.0, 0.05, 0.1, and 0.15; the corresponding open-loop positioning standard deviations are 0.0, 8.5, 17.0, and 25.5 mm.
- Results: Closed-loop control outperforms open-loop control whenever kinematic errors are present.The comparison uses 10 trials per object in both the adversarial and household sets.
- Results: 68% and 73% are the closed-loop worst-case grasp success rates for adversarial and household objects, respectively.The open-loop method achieves only 38% in the worst case.
- Limitations: The closed-loop performance decrease is caused by the camera’s inability to update the grasp pose within 70 mm of the object.Consequently, errors occurring within that range cannot be corrected.