PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty

Edward Johns, Stefan Leutenegger, Andrew J. Davison

arXiv:1608.02239v1cs.ROcs.CVcs.LG

TL;DR

The paper addresses grasping isolated objects from depth images when gripper pose is uncertain, a setting where targeting one predicted optimum can be fragile. It learns a grasp function scoring all poses, smooths that function with the pose-uncertainty distribution, and selects the maximum of the result. Synthetic and real experiments show the smoothed method is more robust to pose uncertainty than ignoring it.

  • Problem

    Gripper pose uncertainty makes precise execution of a single image-derived grasp pose challenging, especially near unstable or collision-prone regions.

  • Method

    A CNN predicts grasp quality for every pose from one depth image, using simulated training data, then smooths the grasp function with the gripper pose-uncertainty distribution.

  • Results

    The maximum of the smoothed grasp function is superior to the maximum of the original function in both synthetic and real-world experiments.

  • Takeaways & Limitations

    Accounting for pose uncertainty during grasp selection improves robustness compared with choosing only the original maximum-score pose.

  • Takeaways & Limitations

    Extending simulation to more gripper-pose degrees of freedom would require more selective data generation because current data generation already takes a week.

Abstract

from arXiv · show

This paper presents a new method for parallel-jaw grasping of isolated objects from depth images, under large gripper pose uncertainty. Whilst most approaches aim to predict the single best grasp pose from an image, our method first predicts a score for every possible grasp pose, which we denote the grasp function. With this, it is possible to achieve grasping robust to the gripper's pose uncertainty, by smoothing the grasp function with the pose uncertainty function. Therefore, if the single best pose is adjacent to a region of poor grasp quality, that pose will no longer be chosen, and instead a pose will be chosen which is surrounded by a region of high grasp quality. To learn this function, we train a Convolutional Neural Network which takes as input a single depth image of an object, and outputs a score for each grasp pose across the image. Training data for this is generated by use of physics simulation and depth image simulation with 3D object meshes, to enable acquisition of sufficient data without requiring exhaustive real-world experiments. We evaluate with both synthetic and real experiments, and show that the learned grasp score is more robust to gripper pose uncertainty than when this uncertainty is not accounted for.

I. INTRODUCTION

Gripper pose uncertainty makes targeting the single best grasp risky because nearby poses may be unstable or collision-prone. The paper instead learns grasp quality across poses and smooths it with the uncertainty distribution to select a more robust target.

  • Pose uncertainty arises from noisy encoders, kinematic deformation, and inaccurate camera–robot calibration, making precise alignment difficult.
  • A single best grasp can become very poor when small targeting errors reach unstable regions or collision-blocked parts of the object.
  • The grasp function scores all possible grasp poses, enabling pose uncertainty to be marginalised by smoothing scores with an uncertainty function.
  • A CNN predicts grasp scores for every parallel-jaw pose from a single depth image, with synthetic depth images and physics simulation supplying training data.
  • Selecting the maximum of the smoothed grasp function makes it less likely that the robot will grasp in a region of poor quality.

II. RELATED WORK

Prior parallel-jaw grasping methods regress an optimum pose or score local patches, while this paper predicts grasp quality across an entire image from one depth image. Related work also motivates simulation because manual labels and real-robot data are difficult to scale.

  • Deep learning has achieved state-of-the-art grasp-pose detection in simpler settings, although hand-engineered features remain effective for clutter and multi-fingered grasping.
  • Manually labelled images are unsuitable for larger-scale training because deep learning requires very large training volumes.
  • Static grasp metrics omit motion during lifting, whereas dynamic physics simulations provide more accurate grasp-quality predictions.
  • Physics-simulation predictions of good grasps have been reported to correlate highly with human-labeller predictions.
  • Real-robot data collection can require weeks or months, limiting scalability even when reinforcement learning or multiple robots are used.
  • The paper predicts grasp-quality scores over an entire image from a single image, enabling real-time operation and online incorporation of gripper pose uncertainty.

III. DEFINING A TARGET POSE

The task defines a parallel-jaw gripper pose from a single-view depth image for grasping an isolated object on a flat table. Image-coordinate poses are calibrated into the robot frame and discretised into candidate poses whose scores form the grasp function.

  • The setup grasps one isolated object from a single-view depth image while constraining the gripper perpendicular to a flat table at constant height.
  • The target pose is expressed in image coordinates as p = {u, v, θ}, representing two translations and rotation about the image z-axis.
  • A wrist-mounted depth camera and camera–robot calibration allow the image-coordinate target pose to be transformed into the robot frame.
  • The image-coordinate pose is transformed to the robot frame through calibrated transformations between the robot, gripper, camera, and image coordinates.
  • Discretisation produces 8712 candidate poses, each receiving a grasp-quality score that together defines f(q).The chosen grid uses 1cm translation increments and 30° rotation increments.

IV. GENERATING TRAINING DATA

Training data are generated synthetically by rendering depth images of 3D meshes and executing simulated parallel-jaw grasps with dynamic physics. Simulator parameters are tuned for realistic relative grasp-quality rankings rather than accurate absolute forces.

  • A CNN maps a single depth image to a grasp-quality score for each candidate grasp pose.
  • Synthetic training data combine OpenGL-rendered depth images with physics-engine simulations because real-world experiments cannot scale to the required volume.
  • DART is selected over static simulators because dynamic modelling matters for predicting grasp behaviour after execution.
  • Each simulated grasp positions a kinematic hand, closes dynamically controlled fingers, and lifts the hand 20cm at 0.1m/s.
  • The simulator's finger torque, friction coefficients, and object density are manually tuned to obtain acceptable grasping realism across objects.
  • Simulation quality is judged by relative grasp quality across poses rather than the absolute torque magnitude required.

B. Image Simulation

Training depth images are simulated from randomly oriented 3D object meshes, then augmented with sensor-like noise. Physics simulation supplies graded grasp-quality labels for every discretized pose.

  • Image generation: Synthetic depth images are rendered with OpenGL’s depth buffer and corrupted using pixel-localization and depth-measurement noise.The noise model uses two Gaussian components to approximate the experimental camera.
  • Noise model: Pixel-localization noise has standard deviation 1 pixel, while depth-estimation noise has standard deviation 1.5mm.
  • Grasp labels: Physics simulation computes a grasp-quality score f(q) for every discrete pose by attempting grasps on each training object.Successfully lifting an object 20cm off the surface receives score 1.
  • Grasp labels: Scores are assigned from 0 to 1 in 0.2 intervals, producing more informative labels than binary success or failure.Higher scores correspond to image regions associated with stable grasps.
  • Object variation: ModelNet objects include varied categories and random orientations so the network learns object shape rather than object identity or semantics.

V. LEARNING THE GRASP FUNCTION

The CNN predicts a discretized grasp-quality distribution for every possible gripper pose from one depth image, rather than predicting only the single highest-scoring pose. It is trained as a six-class classification problem over pose-specific grasp scores.

  • Network objective: The CNN maps one depth image to grasp scores for every pose, enabling prediction of a grasp-score distribution across the image.
  • Network objective: Each pose is classified into one of six scores: 0, 0.2, 0.4, 0.6, 0.8, or 1.0.The network uses an AlexNet-like structure with five convolutional layers, max pooling, and two fully connected layers.
  • Training loss: Training minimizes the difference between ground-truth and predicted score classes across all images and poses.
  • Output representation: The grasp-function visualization represents each pose with six score nodes and displays pose strength using the maximum score level.
  • Output representation: Softmax is used because relative output values matter, while their absolute magnitudes do not.Training uses gradient descent on mini-batches with TensorFlow.

VI. GRASP EXECUTION

During execution, the system repairs missing depth pixels, predicts a grasp function, and accounts for pose uncertainty by Gaussian smoothing in pose space. It then interpolates the smoothed function and sends the highest-valued pose.

  • Depth preprocessing: Zero-valued depth pixels are replaced by the nearest non-zero pixel or by an average of nearby non-zero pixels.These zeros arise from imperfect sensing, including depth shadows and poorly infrared-reflective bodies.
  • Grasp prediction: The processed depth image is passed to the network to predict grasp quality for each gripper pose.
  • Uncertainty modeling: The achieved gripper pose is modeled as a Gaussian around the target pose with covariance matrix Σ.The uncertainty is defined in the image-plane (u, v) coordinates used for the grasp function.
  • Robust planning: The grasp function is smoothed in three-dimensional pose space (u, v, θ) using a Gaussian kernel whose covariance matches gripper pose uncertainty.
  • Robust planning: After smoothing, trilinear interpolation provides precision beyond discretization, and the maximum of the final distribution determines the commanded pose.

VII. EXPERIMENTS

The approach is trained from simulated data generated with 1,000 ModelNet objects and evaluated in both simulation and real-world robot grasping. Simulation enables broader datasets and parameter ranges, while real experiments test transfer to real data.

  • Training data: Training data were generated from 1,000 randomly selected ModelNet objects, requiring roughly one week at about 10 minutes per object.
  • Training data: Training augmentation randomly rotates images to improve robustness to camera orientation.
  • Evaluation: Validation combines synthetic simulation experiments with real-world grasping on a robot platform.
  • Evaluation: Simulation evaluates the method over a much larger dataset and parameter range than feasible in real-world experiments.Real-world experiments assess how well synthetic training adapts to real data.

A. Simulation

Simulation evaluates grasp-selection methods under increasing translational and orientation uncertainty. Robust Best Grasp consistently outperforms the baselines, especially at high uncertainty, while Best Grasp can fall below Centroids.

  • Experimental setup: The experiment tests 1000 ModelNet objects with synthetic noisy depth images and simulated pose uncertainty.Each grasp is successful when the object remains in the gripper after lifting 20cm.
  • Methods: The comparison includes Centroids, Best Grasp, and Robust Best Grasp, which selects the maximum of the uncertainty-smoothed grasp function.
  • Results: At σθ = 10 degrees, Robust Best Grasp performs similarly to Best Grasp at low translational uncertainty but increasingly surpasses it as uncertainty grows.Best Grasp produces poor grasps when it selects only the score-function maximum.
  • Results: As translational and orientation uncertainty increase, all methods degrade, but Robust Best Grasp consistently outperforms the alternatives.The advantage is particularly prominent at high uncertainty.
  • Results: At very high uncertainty, Best Grasp performs worse than the Centroid baseline.This suggests that pinpointing an optimum grasp is less effective than targeting the centroid when pose control is highly unreliable.

B. Real-World Grasping

Real-world experiments evaluate the method on 20 varied everyday objects using a wrist-mounted depth camera and a Kinova MICO arm. Robust Best Grasp outperforms competing methods at measured uncertainty, with a larger relative advantage after uncertainty is increased.

  • Experimental setup: Real-world validation uses 20 everyday objects spanning varied sizes, shapes, and frictional coefficients.Each object is placed at five random positions and orientations within a graspable table area.
  • Experimental setup: The robot’s measured uncertainty is σuv = 6.2 pixels and σθ = 4.7 degrees, then increased to σuv = 20 and σθ = 15.
  • Results: Robust Best Grasp outperforms the two competing methods at the robot’s measured pose uncertainty.The real-world results are reported in Table II.
  • Results: After increasing pose uncertainty, Robust Best Grasp’s relative performance increases further.
  • Results: At very high pose uncertainty, Centroid performs better than selecting the maximum of the grasp function.
  • Grasp behavior: As uncertainty increases, the robust method shifts targeted grasps away from collision-prone mass regions and, at very large uncertainty, toward the middle of long thin object parts.Figure 11 provides examples executed using the robot’s measured uncertainty.

VIII. CONCLUSIONS

The paper learns a grasp function from depth images using synthetic data and selects grasps by smoothing that function with gripper pose uncertainty. The smoothed-function maximum is superior to the original-function maximum in both synthetic and real-world experiments, while broader extensions face scalability constraints.

  • Contribution: The method predicts grasp quality over all grasp poses, calling this output the grasp function.
  • Contribution: Synthetic physics and depth-image simulation provide training data for a CNN that maps depth images to the grasp function.
  • Conclusion: Convolving the grasp function with gripper pose uncertainty produces a smoothed function whose maximum is superior to the original-function maximum in synthetic and real experiments.
  • Limitations and future work: Extending the approach to more complex gripper poses would require more selective simulation because current data generation already takes a week.The paper identifies active learning as future work for scalability.
Loading 1608.02239v1…