PaperlayerView exact revision on arXiv

Source-linked AI summary

Grasp Pose Detection in Point Clouds

Andreas ten Pas, Marcus Gualtieri, Kate Saenko, Robert Platt

arXiv:1706.09911v1cs.RO

TL;DR

Existing grasp detection methods generalize to new objects but have not been shown reliable enough for practical use in realistic dense clutter. This paper introduces algorithmic improvements and evaluates them systematically, reporting 93% average end-to-end grasp success for novel objects in dense clutter.

  • Problem

    Existing grasp detection methods achieve 75%–95% success for novel objects in isolation or light clutter, while those scenarios often do not reflect real-world grasping.

  • Method

    The paper combines segmentation-free grasp-hypothesis generation, a multi-view surface-normal descriptor, object-category knowledge, dense-clutter benchmarking, and high-precision recall evaluation.

  • Results

    The proposed innovations are reported to yield a 93% average end-to-end grasp success rate for novel objects in dense clutter.

  • Takeaways & Limitations

    The paper demonstrates grasp detection evaluated systematically on real robot hardware in dense clutter, including ablations and a high-precision performance measure.

  • Takeaways & Limitations

    The method’s reported sampling advantages may diminish for hands with more complex closing regions, beyond the evaluated two-fingered setting.

Abstract

from arXiv · show

Recently, a number of grasp detection methods have been proposed that can be used to localize robotic grasp configurations directly from sensor data without estimating object pose. The underlying idea is to treat grasp perception analogously to object detection in computer vision. These methods take as input a noisy and partially occluded RGBD image or point cloud and produce as output pose estimates of viable grasps, without assuming a known CAD model of the object. Although these methods generalize grasp knowledge to new objects well, they have not yet been demonstrated to be reliable enough for wide use. Many grasp detection methods achieve grasp success rates (grasp successes as a fraction of the total number of grasp attempts) between 75% and 95% for novel objects presented in isolation or in light clutter. Not only are these success rates too low for practical grasping applications, but the light clutter scenarios that are evaluated often do not reflect the realities of real world grasping. This paper proposes a number of innovations that together result in a significant improvement in grasp detection performance. The specific improvement in performance due to each of our contributions is quantitatively measured either in simulation or on robotic hardware. Ultimately, we report a series of robotic experiments that average a 93% end-to-end grasp success rate for novel objects presented in dense clutter.

Introduction

Grasp detection localizes viable grasp poses directly from noisy, partially occluded sensor data, avoiding explicit object-pose estimation and CAD models. The paper targets insufficient reliability in realistic clutter with algorithmic improvements and systematic evaluation.

  • Motivation: Traditional CAD-model registration and pose estimation can fail with noisy partial clouds and assume an accurate model exists for every grasped object.The proposed grasp-detection framing instead detects local grasp surfaces directly from sensor data.
  • Motivation: Grasp detection produces viable grasp pose estimates directly from noisy, partially occluded RGBD images or point clouds without estimating object pose.These methods treat grasp perception analogously to object detection and generally transfer grasp knowledge to new objects.
  • Motivation: 75%–95% grasp success rates were reported for novel objects in isolation or light clutter, scenarios that often do not reflect real-world grasping.Grasp success is defined as successes divided by total grasp attempts.
  • Contributions: Three algorithmic contributions improve accuracy and applicability in dense clutter: segmentation-free hypothesis generation, multi-view surface-normal descriptors, and object-category knowledge.The descriptor improves classification accuracy by approximately 10%, while category knowledge adds roughly 2%.
  • Contributions: The paper introduces a dense-clutter benchmark, recall at specified high precision, and a method combining object and grasp detection.The benchmark evaluates algorithmic ablations on a real robot.
  • Evaluation: Additional experiments compare ablations, quantify runtime, evaluate category-knowledge gains, and assess combined object-and-grasp detection.The current paper extends two prior conference publications with these analyses and comparisons.

Problem Statement

The paper formulates grasp pose detection as finding hand configurations that produce force-closure grasps from a partial, noisy viewpoint cloud. The formulation assumes a two-finger hand and searches poses within a specified region of interest.

  • Problem formulation: A viewpoint cloud consists of point-cloud data paired with at least one camera viewpoint for each observed point.It represents partial and noisy observations from one or more depth sensors registered to the robot workspace.
  • Hand model: A two-finger hand is modeled by two contact surfaces moving toward and away from each other along a one-dimensional manifold.A parallel-jaw gripper is a trivial example, while opposing fingers of a Barrett hand can form a virtual finger.
  • Hand model: For two-finger hands, a hand’s 6-DOF pose relative to an object suffices to calculate closing-finger contact locations and evaluate whether a grasp exists.This follows from the hand’s two opposing contact surfaces.
  • Grasp definition: Force closure requires contact wrenches to span the origin, and the task is to identify such grasps from the viewpoint cloud.The observed cloud encodes partial and noisy information about multiple underlying objects.
  • Problem formulation: Given a viewpoint cloud, region R ⊂ SE(3), and two-finger hand, the detector seeks one or more 6-DOF poses h ∈ R that form force-closure grasps when closed.The grasp may be formed with respect to some object in the observed object set.

Overview of the Grasp Pose Detection Algorithm

The algorithm preprocesses a viewpoint cloud, defines a region of interest, samples thousands of 6-DOF grasp candidates, encodes them, scores them with a CNN, and selects one for execution.

  • Algorithm overview: Algorithm 1 takes a viewpoint cloud, region of interest, hand description, and positive integer N, and outputs a set of 6-DOF grasp candidates.The output set is H ⊂ R.
  • Preprocessing and ROI: Preprocessing denoises the viewpoint cloud, and ROI identification locates where grasp detection will occur.Preprocessing includes voxelization, outlier removal, and other standard noise-reduction steps.
  • Encoding and scoring: Each candidate is encoded as a stacked multi-channel image and scored by a four-layer convolutional neural network for grasp likelihood.The score indicates how likely each candidate is to constitute a grasp.
  • Grasp selection: The selector chooses a grasp for execution using the CNN score and other suitability considerations.The algorithm’s final stage operates after candidate scoring.
  • Candidate sampling: The sampler generates N, typically several thousand, grasp candidates from the ROI, with each candidate represented as a 6-DOF hand pose.The ROI may contain multiple objects or all objects in the scene rather than a segmented target.

Sampling Grasp Candidates

The method samples 6-DOF grasp hypotheses around randomly selected visible points, using local surface frames and a grid of translations and rotations. It filters candidates for collision avoidance and contact, aiming for broad coverage of graspable surfaces.

  • Candidate hands must avoid point-cloud collisions and include at least one observed point in their closing region.These conditions are useful filters, although noisy point clouds make them neither strictly necessary nor sufficient.
  • The algorithm outputs 6-DOF grasp configurations by sampling points in the region of interest and testing hand placements.It samples from C ∩R and returns a set H of grasp configurations.
  • For each sampled point, the method computes a local Darboux frame from eigenvectors of a neighborhood matrix.The frame aligns with smoothed surface normals and principal curvature directions.
  • The local search evaluates a grid of 10 translations along y and 8 rotations about z, then pushes each hand along negative x until contact.This produces candidate configurations relative to each sampled surface point.
  • The method generates more numerous and uniformly distributed force-closure grasps than the compared x-axis orientation variant.The comparison covers box-like and cylindrical objects viewed from 20 perspectives each.
  • The sampling advantage is hypothesized to depend on two-fingered hands and may disappear for hands with more complex closing regions.Its densest samples align the closing region orthogonally to the local axis of minor principal curvature.

Classifying Grasp Candidates

Grasp candidates are classified with a CNN using voxelized geometry, observed and unobserved volumes, surface normals, and multiple views in the hand frame.

  • Grasp detection is formulated as binary classification with a four-layer CNN that predicts whether each candidate is a grasp.The network receives local observed and occluded object-surface information around the candidate.
  • The candidate representation encodes geometry inside the gripper closing region in the hand reference frame.It includes observed surfaces and volumes unobserved by every sensor.
  • The closing region is voxelized into an M × M × M grid with occupancy, observation, and outward surface-normal information.Occupied voxels carry unit surface normals.
  • The method projects the voxelized closing region onto three planes orthogonal to the hand-frame axes before CNN classification.The projections use the negative x, y, and z directions of the hand reference frame.
  • Each projection supplies averaged occupied heights, unobserved-region heights, and surface normals as classifier inputs.Together, the three projections provide 15 channels.

A Grasp Dataset for Training the CNN

The training dataset pairs sensor-derived point clouds and reconstructed meshes across object geometries and viewpoints, labeling sampled candidates by force-closure criteria.

  • The dataset is designed to pair grasp candidates with force-closure labels across varied grasp geometries and sensor viewpoints.This matches the representation’s dependence on both surface geometry and viewing perspective.
  • Simulated point clouds transfer less effectively to real sensor point clouds, motivating heavy reliance on BigBird data.BigBird pairs reconstructed meshes with real Asus Xtion Pro depth-sensor clouds.
  • For each of 55 objects, the authors generated 20 registered clouds by combining views 53 degrees apart.The configuration reflects the robot’s two-view sensor arrangement.
  • Each sampled two-finger hand pose is labeled by evaluating whether closing the fingers would form a force-closure grasp.The labeling uses the frictionless antipodal special case because only two contact surfaces are considered.
  • Frictionless antipodal grasps are conservative force-closure labels and are robust to small contact perturbations under the stated assumptions.Any such grasp is force closure for nonzero Coulomb friction with soft contacts.
  • Mesh-noise uncertainty is handled by allowing each vertex a 1 mm positional perturbation when evaluating equilibrium grasps.This addresses noise in meshes reconstructed from sensor data.

Improving Classification Accuracy

The paper improves grasp classification by introducing representations, training strategies, and selection heuristics that balance detection accuracy, computation time, and grasp reliability. Results show that the 15-channel representation maximizes accuracy, while a faster 3-channel version supports practical robotic operation and dense-clutter experiments achieve high success rates.

  • Performance Metric: Recall-at-high-precision measures grasp detection when false positives are tightly constrained, addressing the limitations of accuracy alone for grasping.Because false positives can cause failed grasp attempts, the classifier threshold is adjusted to trade recall for very high precision.
  • Representation Accuracy: 89% recall at 99% precision is achieved with the 15-channel representation, compared with 82% for the 12-channel version without occlusion channels.Adding occlusion information improves accuracy by approximately 2% but can double candidate-encoding time.
  • Representation Accuracy: The 3-channel representation performs about as well as the 12-channel version while computing descriptors four times faster than the 15-channel representation.This makes the 3-channel representation suitable for time-sensitive grasp detection despite lower accuracy than the full representation.
  • Representation Accuracy: The 3-channel representation obtains at least 10% lower accuracy than the other evaluated representations when using the representation from prior work.The comparison uses the paper’s own grasp candidates for all representations.
  • Training Data: Training on simulated data from 400 CAD models yields 83% accuracy on the BigBird test set, below the approximately 90% achieved by the best representation on real-data evaluation.The paper attributes the gap to subtle differences between simulated and real sensor depth images.
  • Category Knowledge: 93.5% accuracy and 35% recall-at-99%-precision are obtained for cylindrical objects, while box-like objects reach roughly 93% accuracy and 60% recall-at-99%-precision under leave-one-object-out training.Training uses data from the other objects in the same shape class, providing prior category knowledge without training on the target instance.
  • Runtime and Robot Evaluation: The algorithm typically requires one to eight seconds of runtime, and the 3-channel version processes a 66k-point cloud in 2.1 seconds versus 8 seconds for the 15-channel version.The robotic experiments use the faster version and compensate for its lower accuracy by raising the grasp acceptance threshold.
  • Runtime and Robot Evaluation: 93% grasp success is achieved with active point clouds in dense clutter, compared with 84% using passive point clouds.The active evaluation reports 20 failures in 288 attempts; the passive evaluation reports 22 failures in 138 attempts.

Combined Grasp and Object Detection

The paper combines grasp detection with object detection to identify and grasp specific objects. It uses detected grasps to generate object proposals, then classifies those proposals in cluttered scenes.

  • Grasp detection must be combined with object detection when the robot needs to grasp a particular object rather than any object.
  • Detected grasps serve as object proposals by projecting each grasp’s inter-finger point into the RGB image and placing a 120 × 120 pixel bounding box around it.
  • 68 grasps were detected in the illustrated scene, but detections occurred on only three of seven objects because other objects were occluded or unreachable.
  • The method produced high-confidence labels for the middle flashlight and the white-and-red box, with corresponding grasps selected for those detections.
  • The object detector was trained and evaluated on 11 objects using cluttered scenes containing nine of them across 27 point clouds.
  • 85% average object detection accuracy was achieved in relatively cluttered settings, although segmentation could potentially improve performance.

Discussion

The paper argues that direct grasp detection avoids difficult object-pose registration requirements and performs strongly in dense clutter. It also identifies failure modes involving multi-object grasps and reliable recognition of specific targets.

  • Grasp detection avoids object segmentation, object identity or shape requirements, and CAD-model registration to noisy or incomplete point clouds.
  • 723 dense-clutter grasp experiments, including 288 best-case contingency attempts, support repeatability for the tested objects.
  • Representation, simulated pretraining, object-category priors, point-cloud quality, and grasp-selection heuristics each contribute to grasp-detection performance.The reported hardware effects include 9% higher success with active sensing and nearly 20% improvement from simple grasp-selection heuristics.
  • Because the algorithm does not segment objects, it can grasp multiple objects as though they were one, especially with hands capable of larger grasps.
  • Combined grasp and object detection achieved 85% object-classification accuracy in the laboratory, but improving this result remains important for real-world deployment.
Loading 1706.09911v1…