Source-linked AI summary
High precision grasp pose detection in dense clutter
Marcus Gualtieri, Andreas ten Pas, Kate Saenko, Robert Platt
TL;DR
Grasp pose detection must reliably identify robot grasps in dense clutter despite limited performance and challenging partial point clouds. This paper improves candidate classification using richer representations, object prior knowledge, and simulated-depth pretraining, achieving 93% grasp success in dense clutter on a Baxter robot.
Problem
Existing grasp pose detection methods often achieve only 75%–95% success and are commonly evaluated in light clutter that does not reflect real-world grasping challenges.
Method
The paper generates many 6-DOF grasp candidates, classifies them with a CNN, and evaluates richer candidate representations, object knowledge, and simulated CAD-depth pretraining.
Results
93% grasp success was achieved in dense clutter with the active wrist-mounted sensor, compared with 84% using passive statically placed sensors.
Takeaways & Limitations
More informative representations, pretraining, and prior knowledge significantly improve grasp detection within the evaluated GPD framework.
Takeaways & Limitations
GPD cannot infer non-geometric properties such as center of mass, inertia, and weight, and cannot directly distinguish adjacent objects without segmentation.
Abstract
from arXiv · showhide
This paper considers the problem of grasp pose detection in point clouds. We follow a general algorithmic structure that first generates a large set of 6-DOF grasp candidates and then classifies each of them as a good or a bad grasp. Our focus in this paper is on improving the second step by using depth sensor scans from large online datasets to train a convolutional neural network. We propose two new representations of grasp candidates, and we quantify the effect of using prior knowledge of two forms: instance or category knowledge of the object to be grasped, and pretraining the network on simulated depth data obtained from idealized CAD models. Our analysis shows that a more informative grasp candidate representation as well as pretraining and prior knowledge significantly improve grasp detection. We evaluate our approach on a Baxter Research Robot and demonstrate an average grasp success rate of 93% in dense clutter. This is a 20% improvement compared to our prior work.
I. INTRODUCTION
The paper addresses unreliable grasp pose detection in dense clutter by improving candidate classification with richer representations, prior object knowledge, and simulated-data pretraining.
- The approach characterizes graspable local surfaces with machine learning instead of fitting complete CAD models, enabling generalization across object identities.
- 75%–95% grasp success rates on novel objects remain insufficient for practical grasping, while light-clutter evaluations underrepresent real-world difficulty.
- The paper evaluates two grasp representations, category or instance knowledge, and simulated CAD-depth pretraining using BigBird depth scans and RGB images.
- 93% grasp success was achieved in dense clutter on a Baxter Research Robot, a 20% improvement over prior work.The experiment comprised 288 grasp attempts, with 20 failures.
- Unlike several prior approaches, the method does not require human demonstrations and supports grasp approach directions aligned with object surfaces rather than one fixed sensor direction.
III. GRASP POSE DETECTION
The algorithm takes a point cloud, viewpoints, grasp-region points, and gripper geometry, and outputs 6-DOF hand poses expected to achieve force closure.
- The inputs are a 3-D point cloud with observation viewpoints, a subset identifying graspable regions, and the robot gripper’s geometric parameters.
- The output is a set of robot hand poses in SE(3) for which closing the 1-DOF parallel-jaw gripper is expected to achieve force closure.
- Although the current algorithm uses point clouds, it could be modified to accept a truncated signed distance function.
B. Outline of the GPD algorithm
GPD first samples thousands of 6-DOF grasp candidates and then classifies them with a convolutional neural network.
- The algorithm follows two steps: sample grasp candidates from the input cloud, then classify them using the cloud and viewpoints.
- Several thousand 6-DOF candidates are generated by sampling points, estimating local surface geometry, orienting the hand, and advancing it until contact.Candidates without points between the fingers are discarded, and sampling continues until the desired count is reached.
- Each candidate is classified as a grasp or non-grasp by a four-layer convolutional neural network.
- Figure 1 illustrates grippers placed at randomly sampled candidate configurations.
C. Grasp representation
Each grasp candidate is encoded within the cuboid swept by closing parallel-jaw fingers, combining observed geometry, occluded volume, and surface normals through voxel projections.
- The representation describes observed surfaces and unobserved volumes inside the rectangular cuboid swept by the closing parallel-jaw fingers.The method ignores RGB information.
- The cuboid is scaled to a unit cube and voxelized into a 60 × 60 × 60 grid, with occupancy, observation status, and outward surface normals assigned to voxels.
- Figure 2 depicts a candidate from partial point-cloud data, the local voxel-grid frame, and example grasp images used by the classifier.
- The full 3-D voxel geometry is projected onto planes orthogonal to the voxel-grid basis axes before being passed to the CNN.
- For each of three projections, the system computes averaged occupied-point heights, unobserved-region heights, and surface normals.
- The representation contains 15 channels: five information channels for each of the three projections.
D. Generating training labels
The paper automatically generates labeled grasp examples by pairing local geometry around 6-DOF candidates with antipodal-grasp labels derived from registered meshes and point clouds. These labels support evaluating grasp detection design choices.
- Training exemplars pair local object-surface geometry relative to a 6-DOF hand pose with a grasp-existence label.The data-generation process begins by sampling grasp candidates and labeling whether each candidate forms a valid grasp.
- A frictionless antipodal grasp requires anti-parallel contact normals aligned with the finger-closing direction and contact line.This condition provides the geometric basis for identifying positive grasp labels.
- The labeling pipeline uses object meshes registered to point clouds or TSDFs from multiple viewpoints.These data can come from simulated CAD-model depth images or real depth scans paired with reconstructed meshes.
- The section frames high-precision grasp detection as the central challenge motivating improved training labels and representations.
A. Measuring recall-at-high-precision for grasp pose detection
Recall-at-high-precision measures how many true grasps a detector retrieves while maintaining a required precision, emphasizing false-positive control. The paper compares this metric across grasp representations using BigBird-derived candidates and data.
- Recall-at-high-precision asks what recall is achieved at a specified minimum precision, such as 99%.Increasing the acceptance threshold raises precision but lowers recall, making this metric especially relevant when false positives are costly.
- The evaluation uses BigBird-derived grasp candidates from partial views and compares representation accuracy on this dataset.The dataset contains 216k candidates from selected objects and simulated two-sensor viewpoints.
- Comparison between different representations: 89% recall at 99% precision was achieved with the 15-channel representation, compared with 82% for the 12-channel version without occlusion channels.The full representation gained approximately 2% accuracy from occlusion information, despite the added computational cost.
- Comparison between different representations: The prior three-channel representation performs about as well as the 12-channel no-occlusion representation, suggesting curvature-axis projection carries much of the useful information.
- Comparison between different representations: The Kappler–Herzog representation obtains at least 10% lower accuracy, potentially because hand-approach projection or absent surface normals loses information.
C. Pretraining on simulated data
The paper evaluates simulated-data pretraining followed by fine-tuning on real BigBird depth data. Pretraining accelerates early learning, although its advantage diminishes with continued training.
- 83% accuracy resulted from training only on simulated 3DNET data, versus approximately 90% when evaluated with the best representation on BigBird data.The comparison highlights a persistent domain difference between simulated and real depth images.
- Simulated depth data are attractive because online CAD resources provide many more models than available real sensor data, but simulation-to-real differences can hurt performance.The paper cites 3DNET’s thousands of CAD models across 200 categories as an example.
- The evaluated strategy initializes the CNN with simulated-data weights and fine-tunes it using real BigBird examples.
- Pretraining on simulated 3DNET data lets the network reach the accuracy obtained after 20,000 iterations by 4,000 iterations without pretraining.The pretrained weights have their strongest effect early, and the contribution decreases over time.
D. Using prior knowledge about the object
The paper tests whether object prior knowledge improves grasp classification by training with object-specific, category-specific, or broad-object data. More specific knowledge corresponds to stronger reported accuracy and recall in the cylindrical-object experiment.
- The prior-knowledge experiment compares no prior knowledge, category knowledge, and precise object geometry as distinct training conditions.The figure reports both classification accuracy and recall-at-high-precision at 99% precision.
- Approximately 97% accuracy and 83% recall-at-99%-precision were obtained when training separately for each target object.The experiment averaged results over 16 cylindrical-like BigBird objects.
- Approximately 92% accuracy and 27% recall-at-99%-precision were obtained when training on all other objects.This scenario represents detection without prior knowledge of the particular target object.
- Robot validation used a fast three-channel representation in two dense-clutter scenarios with statically mounted and wrist-mounted depth sensors.
A. Hardware setup
The experiments use a Baxter Research Robot and a dense-clutter setup in which randomly selected household objects are poured into a tray for grasping.
- Hardware: The setup uses Baxter’s right 7-DOF arm with a modified stock parallel-jaw gripper.The gripper has black rubber coverings and outward-bending metal extensions to tolerate small registration or kinematic errors.
- Experimental protocol: The study runs 45 clutter rounds: 15 with two static Kinect sensors and 30 with a wrist-mounted sensor streaming images.The same selected objects are used for one passive round and two active rounds.
- Experimental protocol: The experiment protocol is documented as a one-round clutter-removal procedure in Table I.The figure shows the preparation stage, including pouring the box contents into the tray.
- Experimental protocol: The objects are mixed in a box, poured into a tray, and then removed by the robot until the tray is cleared or the experiment stops.Two objects requiring an upright orientation were manually repositioned when they landed upside down.
C. Clutter removal algorithm
The clutter-removal pipeline combines sensor reconstruction, grasp-candidate filtering, and utility-based selection before executing grasps; performance differs with sensor placement and visibility.
- Scene reconstruction: Passive sensing combines two Kinect images in the robot base frame and voxelizes them, while active sensing reconstructs the scene with a wrist-mounted sensor.The active sensor trajectory keeps at least 40cm between the sensor origin and a fixed point centered in the tray.
- Candidate filtering: The algorithm generates hundreds of grasp candidates, then removes candidates without collision-free IK solutions or acceptable collision geometry.IKFast finds solutions, OpenRAVE performs collision checking, and point-cloud obstacles are modeled as 2cm cubes.
- Grasp selection: The utility function weighs grasp height, gripper width, approach angle, and arm travel distance to rank feasible grasps.The empirically preferred strategy targets the highest object and approaches from above to reduce tray collisions.
- Results: Some objects remained uncleared because they were knocked out, pushed beyond sensor visibility, or grouped too closely for a finger-clear grasp.These failure modes define practical boundaries of the clutter-removal setup.
- Results: 93% grasp success was achieved with the active wrist-mounted sensor, compared with 84% using passive static sensors.The active condition cleared 90% of objects, while the passive condition cleared 77%.
- Results: The active condition’s 9% success-rate improvement is attributed to a more complete view of objects in the tray.Unseen or highly partial point clouds increase collision risk and make grasp classification more difficult.
VI. CONCLUSION
The paper improves point-cloud grasp classification through richer representations, simulated-depth pretraining, and object prior information, while identifying remaining limitations in geometric reasoning.
- VI. CONCLUSION: A 15-channel grasp representation improves classification over less informative representations, but only slightly outperforms the cheaper 12-channel representation.The 15-channel representation is substantially more expensive to compute.
- VI. CONCLUSION: Pretraining on simulated depth data reduces the iterations needed to reach high classification accuracy compared with an untrained network.The simulated data are obtained from idealized CAD models and online datasets.
- VI. CONCLUSION: Prior instance or category information about the target object improves grasp-classification performance.The paper evaluates both forms of object knowledge as additions to the classification step.
- VI. CONCLUSION: 73% to 93% average grasp success was achieved in the robot experiments compared with the authors’ previous work.The conclusion presents this increase as evidence that the proposed classification enhancements improve real-world robot grasping.
- VI. CONCLUSION: GPD algorithms cannot currently infer non-geometric properties such as center of mass, inertia, and weight.The paper also notes that avoiding segmentation prevents directly selecting and distinguishing adjacent objects.