PaperlayerView exact revision on arXiv

Source-linked AI summary

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg

arXiv:1703.09312v3cs.RO

TL;DR

Collecting real grasp labels or physical outcomes is costly, motivating a way to learn robust grasp plans from synthetic data. Dex-Net 2.0 trains a GQ-CNN on 6.7 million synthetic point clouds, grasps, and analytic metrics to predict grasp success from depth images. In more than 1,000 ABB YuMi trials, the planner was 3× faster than registration-based planning and achieved strong performance on known and novel objects.

  • Problem

    Real grasp-planning datasets require either tedious human labeling or months of physical execution time.

  • Method

    Dex-Net 2.0 trains a GQ-CNN on 6.7 million synthetic point clouds, parallel-jaw grasps, and robust analytic grasp metrics to rank grasp candidates from depth images.

  • Results

    3× faster than registration-based planning, the planner achieved 93% success on known objects, 80% success on ten novel objects, and 99% precision on 40 novel household objects.

  • Takeaways & Limitations

    The results show that a grasp planner trained entirely on synthetic data can perform robust grasp planning across known and novel objects.

Abstract

from arXiv · show

To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net .

I. INTRODUCTION

Dex-Net 2.0 addresses the cost of collecting real grasping data by training a deep model on synthetic point clouds and analytic grasp metrics. Its planner predicts robust grasps directly from depth images and performs strongly across known and novel objects.

  • Motivation and approach: GQ-CNN predicts grasp robustness from depth images, avoiding reliance on human labels or months of physical execution data.The approach trains a CNN on rendered point clouds, grasps, and analytic grasp metrics.
  • Related approaches: Analytic registration-based methods can be slow and error-prone, while empirical methods require costly human labels or physical grasp trials.Dex-Net 2.0 combines analytic supervision with direct depth-image prediction.
  • Results: 3× faster than registration-based planning, Dex-Net 2.0 achieved 93% success on known objects and the best performance on novel objects.The comparison used more than 1,000 physical trials with an ABB YuMi robot.
  • Results: 99% precision was achieved on 40 novel household objects despite training entirely on synthetic data.The objects included some articulated or deformable household objects.

III. PROBLEM STATEMENT

The paper formulates robust planar parallel-jaw grasping for a singulated rigid object on a table from a depth-camera point cloud. It models uncertainty in object, camera, grasp, sensing, and friction variables and estimates grasp success robustness.

  • Problem statement: The task is to plan a robust planar parallel-jaw grasp for a singulated rigid object resting on a table.The input is a point cloud from a depth camera.
  • State representation: The state includes object shape and pose, camera pose, and the gripper-object friction coefficient.The state is represented as x = (O, To, Tc, γ).
  • Grasp representation: A grasp is represented by its 3D center and table-plane angle, u = (p, ϕ).The center p has coordinates (x, y, z), while ϕ is the angle in the table plane.
  • Observation model: The observation is a 2.5D point cloud represented as a depth image with height H and width W from a camera with known intrinsics.The formulation uses a single-view depth observation.
  • Robustness metric: Grasp robustness is the expected success metric, or probability of success, conditioned on the grasp and observed point cloud.Formally, Q(u, y) = E[S | u, y].

C. Objective

The objective is to learn a grasp robustness function that predicts binary grasp success from candidate grasps and depth-image observations. Dex-Net 2.0 addresses the sampling challenge with a generative model producing synthetic training examples across objects, poses, grasps, observations, and analytic success metrics.

  • C. Objective: The GQ-CNN robustness function Qθ*(u, y) ∈ [0, 1] classifies candidate grasps using a binary success metric.The function takes grasp and image inputs and estimates grasp robustness.
  • C. Objective: 6.7 million synthetic point clouds, parallel-jaw grasps, and robust analytic metrics across 1,500 3D models address the need for many samples.The dataset is generated from a graphical model for objects, poses, grasps, images, and success metrics.
  • C. Objective: The sample approximation estimates Qθ* from i.i.d. samples drawn from the generative model for images, grasps, and success metrics.The objective is approximated using sampled tuples of states, grasps, observations, and outcomes.
  • C. Objective: The graphical model factors the data distribution into state, observation, grasp-candidate, and analytic grasp-success models.This structure specifies how synthetic training examples are generated.
  • C. Objective: The analytic success model uses robust epsilon quality under uncertainty and requires grasps to be collision-free with the object and table.EQ includes uncertainty in friction and gripper pose.

2) Database:

Dex-Net 2.0 combines 1,500 mesh models, sampled parallel-jaw grasps, robust quality labels, and rendered noisy depth images. The GQ-CNN aligns each grasp with a centered image representation and predicts grasp robustness from image and gripper-depth inputs.

  • 2) Database:: 6.7 million datapoints are generated through the Dex-Net 2.0 pipeline.The dataset is built from object models, grasps, rendered point clouds, and associated grasp metrics.
  • 2) Database:: 1,500 mesh models are aligned, rescaled to a 5.0cm gripper width, assigned mass, and associated with stable poses.The collection includes 1,371 synthetic models and 129 laser scans.
  • 2) Database:: Each object receives up to 100 parallel-jaw grasps sampled from antipodal surface points with constraints for object-surface coverage.Expected epsilon quality is evaluated under uncertainty using Monte-Carlo sampling.
  • 2) Database:: Rendered 2.5D point clouds pair each object stable pose with sampled camera and planar object poses using a pinhole camera model.Training images include added noise and are centered on the object through pixel transformations.
  • 2) Database:: The GQ-CNN takes a centered, orientation-aligned depth image and gripper distance from the camera to estimate grasp robustness.Alignment removes the need to learn rotational invariances and supports arbitrary grasp orientations.
  • 2) Database:: The network uses convolutional and fully connected layers with a separate input for gripper distance, and its first filters appear to detect oriented depth gradients.The architecture contains approximately 18 million parameters.

2) Training Dataset:

The study trains and evaluates GQ-CNN grasp models from transformed depth-image and grasp configurations, then benchmarks planners on an ABB YuMi tabletop setup using standardized success criteria.

  • Training Dataset: GQ-CNN training examples associate each grasp with pixel position, orientation, and depth relative to a rendered depth image.Grasps are transformed into the camera frame and projected onto the imaging plane.
  • Optimization: Training uses backpropagation with stochastic gradient descent and momentum, plus geometric augmentation and adaptive image-noise sampling.Images are reflected and rotated, while Gaussian-process noise is approximated through upsampled interpolated noise.
  • Grasp Planning: The planner ranks randomly sampled antipodal candidates with the GQ-CNN and executes the highest-quality candidate that is reachable and collision-free.The policy selects argmax_u∈C Qθ(u,y).
  • Physical Benchmark Description: The benchmark reports success rate, precision, robust grasp rate, and planning time for tabletop single-object grasping.Success requires lifting, transporting, and holding the object after shaking.
  • Physical Benchmark Description: Physical trials use an RGB-D image, inpainting, color-background segmentation, and a bounding box before the YuMi executes the planned gripper pose.A grasp succeeds if the gripper retains the object after lifting, transporting, and shaking.

B. Datasets

The evaluation combines physical object benchmarks, synthetic and physical training datasets, and several grasp-planning baselines to test classification, efficiency, and generalization.

  • B. Datasets: The Train set contains eight 3D-printed objects with adversarial geometry, while the Test set contains ten unseen household objects for evaluating generalization.Test objects vary in material, geometric, and specular properties.
  • B. Datasets: The four GQ-CNN datasets range from 400 physical trials and 189k adversarial synthetic datapoints to Dex-Net-Small with 670k and Dex-Net-Large with 6.7m datapoints.Each dataset uses an 80–20 image-wise training-validation split.
  • C. Grasp Planning Methods Used for Comparison: Compared planners include image-based grasp heuristics, point-cloud registration, random forests, support vector machines, and GQ-CNN variants.Non-registration methods use the same antipodal grasp sampling procedure and execute their highest-ranked candidate.
  • E. Performance Comparison on Known Objects: On Train, point-cloud registration reaches 95% success, whereas GQ-L-Adv reaches 93% and is 3× faster.The table caption reports 80 trials per method and notes lower performance for the physical-data-trained GQ-Adv-Phys variant.
  • C. Grasp Planning Methods Used for Comparison: GQ-CNN models are trained with thresholded robust epsilon quality, while alternative variants use force-closure probabilities or physical-robot grasp labels.The main GQ model is trained on Dex-Net-Large with Gaussian-process image noise.

D. Classification of Synthetic Data

GQ-CNNs classify synthetic grasp data more accurately than the random-forest and SVM baselines, while physical planning achieves strong performance on both known and novel objects.

  • D. Classification of Synthetic Data: 85.7% accuracy was achieved by the GQ-CNN trained on all Dex-Net 2.0 on a held-out validation set of approximately 1.3 million datapoints.On Adv-Synth, GQ-CNNs achieved near-perfect validation accuracy and outperformed ML-RF and ML-SVM.
  • E. Performance Comparison on Known Objects: GQ planned grasps 3× faster than registration and achieved 93% success and 94% precision on known objects.The results indicate that the full Dex-Net 2.0 dataset was needed to exceed 90% success.
  • F. Performance Comparison on Novel Objects: On ten novel objects, GQ achieved the best performance with 80% success and 100% precision, with zero false positives among 29 robust classifications.The parameters were set using Train performance without access to Test performance.

G. Generalization Ability of the Dex-Net 2.0 Grasp Planner

The Dex-Net 2.0 planner generalizes to novel household objects, including articulated and deformable items, while maintaining high success and precision. Its failures mainly arise from missing depth data and collisions misclassified as robust grasps.

  • Generalization performance: 100% precision was reported on a separate test set of 10 household objects, with zero false positives among 29 positive classifications.The table states that GQ performed best in success rate and precision on this benchmark.
  • Generalization performance: 94% success and 99% precision were achieved on 40 household objects, including articulated and deformable objects, with CEM-augmented Dex-Net 2.0.Precision was 68 successes out of 69 grasps classified as robust, with 2.5 seconds average planning time.
  • Order-fulfillment application: The order-fulfillment system successfully placed the correct objects in the shipping box on 4 of 5 attempts and grasped objects successfully in 93% of 27 attempts.The application used an ABB YuMi and three target objects in the presence of distractors.
  • Failure modes: Common failures occurred when the RGB-D sensor missed thin object geometry, making those regions appear accessible to the planner.The failure mode was observed among examples of failed GQ-CNN-planned grasps.
  • Failure modes: Collisions with objects were sometimes misclassified as robust grasps, especially in narrow parts of object geometry.The paper suggests improving performance with more accurate depth sensing and analytic collision pruning.
  • Overall evaluation: Across more than 1,000 physical evaluations, the planner was 3× faster than registration-based planning and achieved 99% precision on 40 novel objects.The paper reports that these results were obtained despite training entirely on synthetic data.

APPENDIX A PARAMETERS OF GRAPHICAL MODEL

The graphical model factors state, observations, grasp candidates, and grasp success, then specifies distributions for object, pose, camera, friction, rendering, and measurement noise.

  • Graphical model: The joint model p(S, g, x, y) factors into state, observation, grasp candidate, and grasp success distributions.The factors are p(x), p(y|x), p(g|x), and p(S|g, x).
  • State parameters: The state distribution combines friction, object identity, object pose conditioned on the object, and camera pose as p(x) = p(γ)p(O)p(To|O)p(Tc).Object poses are modeled through stable poses and randomized planar poses, while camera pose is sampled on spherical coordinates.
  • Grasp model: Grasp candidates are sampled uniformly from pairs of antipodal surface points parallel to the table plane with friction coefficient µ = 0.6.Samples outside the friction cone or nonparallel to the surface are rejected.
  • Observation model: Rendered depth images are modeled as y = α ∗ˆy + ϵ, combining a rendered image with Gamma-distributed scaling and Gaussian Process measurement noise.The noise model uses measurement noise σ = 0.005 and kernel bandwidth ℓ = √2px.
  • Robustness metrics: Grasp robustness metrics are computed using the graphical model and the specified noise parameters.The metrics are based on the modeled state, observations, grasps, and success process.

APPENDIX B GRASP SAMPLING METHODS

The appendix formulates grasp-policy learning as maximizing success over possible point-cloud distributions and implements it with a greedy GQ-CNN policy over constrained candidates.

  • Policy objective: The objective is to learn policy parameters θ that maximize the success rate of planned grasps over point clouds generated from possible objects D.The objective is presented as equation B.1.
  • Greedy grasp policy: The greedy policy selects the candidate u in constrained set C with the highest predicted robustness Qθ(u, y).Constraints can include kinematic feasibility and other restrictions on available grasps.
  • Policy implementations: Two implementations are explored: ranking a fixed set of sampled antipodal grasps and optimizing for the most robust grasp with derivative-free optimization.Both implementations use the GQ-CNN robustness function to choose grasps.

A. Antipodal Grasp Sampling

The antipodal sampler extracts high-gradient regions from a depth image, samples pixel pairs satisfying antipodal constraints, and converts accepted candidates into 3D grasps with discretized heights.

  • Grasp representation: The algorithm samples antipodal grasps represented by planar pose, angle, and height relative to the table.These grasps are generated from depth-image evidence and later converted into 3D coordinates.
  • Candidate generation: Depth-image regions of high gradient are found by thresholding the image before candidate-pair sampling.The edge set is defined by Gx(u)^2 + Gy(u)^2 > g.
  • Candidate generation: Pairs of pixels are generated by rejection sampling and retained when they satisfy the antipodal condition under the friction coefficient.The friction coefficient is incremented until the desired number of grasps is reached when necessary.
  • 3D conversion: Accepted image-space grasps are converted to 3D by discretizing gripper height between the grasp-center height and the table surface.The height resolution is δh = 0.01m in the reported implementation.
  • Implementation parameters: The implementation uses M = 1000 candidate grasps and N = 1000 maximum samples per friction coefficient, with camera intrinsics from a Primesense Carmine 1.08.Other settings include g = 0.0025m, µℓ = 0.4, δµ = 0.2, and δh = 0.01m.

B. Derivative Free Optimization

The paper addresses failures caused by fixed candidate sets by optimizing grasp robustness with derivative-free cross entropy search over a learned grasp distribution. The method iteratively resamples promising grasps and returns an estimate of the most robust grasp.

  • Fixed candidate sets can fail when all candidates have low success probability, especially for objects requiring precise grasp configurations.The example in Fig. 9 illustrates this failure mode.
  • The cross entropy method optimizes for robust grasps by iteratively resampling from a learned distribution and updating that distribution.The distribution is modeled with a Gaussian Mixture Model.
  • The method seeds its initial grasps with antipodal point pairs before optimizing the learned robustness distribution.
  • The algorithm takes CEM iterations, initial and resampled grasp counts, GMM components, friction coefficient, elite percentage, and GQ-CNN robustness function as inputs, returning the most robust grasp estimate.The reported experiment used m = 3, n = 100, c = 50, µ = 0.8, k = 3, and γ = 25%.
  • Algorithm 2 specifies a robust grasping policy that applies the cross entropy method to a learned GQ-CNN.
Loading 1703.09312v3…