Source-linked AI summary
PointNetGPD: Detecting Grasp Configurations from Point Sets
Hongzhuo Liang, Xiaojian Ma, Shuang Li, Michael Görner, Song Tang, Bin Fang, Fuchun Sun, Jianwei Zhang
TL;DR
Robot grasp planning must handle imprecise and sparse sensing while evaluating grasp quality from point clouds. PointNetGPD directly analyzes local 3D point sets and trains on 350k real examples with fine-grained analytic labels. It outperforms the considered GPD baselines, including by 4.79% on the most difficult 1-viewed point cloud.
Problem
Grasp planning under uncertainty requires grasp-quality evaluation from imprecise and deficient sensor point clouds.
Method
PointNetGPD directly analyzes the 3D point cloud within each candidate gripper configuration and trains on 350k real point clouds and grasps with analytic quality scores.
Results
4.79% improvement over the best GPD baseline was achieved on the most difficult 1-viewed point cloud, while PointNetGPD performed significantly better on grasp-quality classification.
Takeaways & Limitations
PointNetGPD generalizes to novel objects and outperforms state-of-the-art grasp-detection methods in the reported experiments.
Abstract
from arXiv · showhide
In this paper, we propose an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud. Compared to recent grasp evaluation metrics that are based on handcrafted depth features and a convolutional neural network (CNN), our proposed PointNetGPD is lightweight and can directly process the 3D point cloud that locates within the gripper for grasp evaluation. Taking the raw point cloud as input, our proposed grasp evaluation network can capture the complex geometric structure of the contact area between the gripper and the object even if the point cloud is very sparse. To further improve our proposed model, we generate a larger-scale grasp dataset with 350k real point cloud and grasps with the YCB object set for training. The performance of the proposed model is quantitatively measured both in simulation and on robotic hardware. Experiments on object grasping and clutter removal show that our proposed model generalizes well to novel objects and outperforms state-of-the-art methods. Code and video are available at \href{https://lianghongzhuo.github.io/PointNetGPD}{https://lianghongzhuo.github.io/PointNetGPD}
I. INTRODUCTION
PointNetGPD addresses grasp planning under sensing uncertainty by evaluating candidate grasps directly from sparse 3D point clouds. It combines PointNet-based geometry analysis with a 350k-example real grasp dataset using fine-grained quality scores.
- Grasp planning is difficult because real-world sensing is imprecise and incomplete.
- PointNetGPD evaluates grasp configurations directly from 3D point clouds without projecting them into multiple 2D images or dense 3D volumes.This preserves more of the original point-cloud geometry and improves evaluation efficiency.
- The authors built a dataset containing 350k real point clouds and parallel-jaw grasps from the YCB object set.The dataset provides meticulous scores combining force-closure and GWS analysis.
- Fine-grained grasp quality scores and labels are intended to improve training beyond binary supervision.
- The method performs geometry analysis without hand-crafted features and remains effective when point clouds are very sparse.The authors describe this as supporting grasp planning under imprecise and deficient sensing.
II. RELATED WORK
Prior grasp-detection methods use model databases, image or depth projections, voxelized 3D representations, or handcrafted features. PointNetGPD is motivated by preserving sparse 3D geometry while avoiding these limitations.
- Model-based grasping relies on pre-built object models and labeled grasp databases, limiting direct use of raw sensor point clouds.
- Some clutter-grasping methods use multi-view depth but depend strongly on accurate CAD models, hindering generalization to novel objects.
- 3D-CNN methods voxelize point clouds, but their runtime and memory complexity grows cubically with voxel resolution.This forces the input toward relatively low resolution.
- GPD uses projected depth features and a CNN, but the authors report overfitting and reduced performance on overall sparse point clouds.The issue is especially relevant when clutter causes severe occlusion.
- The paper argues that predominantly 2D or 2.5D approaches are insufficient for geometry analysis.PointNet-based 3D representation learning and meticulous quality labels are presented as alternatives.
III. PROBLEM FORMULATION
The problem formulation represents object state, grasp configuration, sensor point cloud, and grasp quality explicitly. The learning task is to predict quality from the observed point cloud and candidate grasp.
- A. Definitions: The object state includes friction coefficient, geometry and mass properties, and a 6-DOF pose.
- A. Definitions: The coefficient of friction is denoted γ, while object geometry and mass properties are denoted M_o.
- A. Definitions: The object pose is represented as W_o ∈ R^6.
- A. Definitions: A grasp configuration is g = (p, r) ∈ R^6, combining gripper position and orientation.
- A. Definitions: The position p and orientation r each contain three spatial components.
- A. Definitions: The paper considers parallel-jaw grippers and represents the camera-derived point cloud with N points as P ∈ R^3×N.
- A. Definitions: The ground-truth quality metric Q(s, g) maps an accurate object state and grasp to a continuous real-valued quality.
B. Objective
The paper trains a point-cloud grasp-quality classifier using large-scale labeled data and combines force-closure with grasp-wrench-space analysis for supervision. Its force-closure score converts friction requirements into a quantitative robustness signal.
- Objective: The learned metric Q_θ(P, g) predicts grasp-quality labels from a sensor point cloud and gripper configuration.
- Objective: The method addresses sparse, imprecise point clouds by directly analyzing them with PointNet.
- Objective: Training uses 350k real point clouds and grasps from objects in the YCB set.
- Objective: The dataset samples candidate grasps over precise object meshes before assigning robust analytic quality metrics.Mesh-based sampling avoids unfeasible candidates such as grasps colliding with the object.
- Objective: The force-closure score records 1/γ after gradually increasing friction until a grasp becomes antipodal.Lower required friction is treated as indicating a more robust and feasible grasp.
- Objective: Figure 2 contrasts grasps labeled γ = 0.4 and γ = 2.0, showing a significant robustness difference for a box-like object.
- Objective: The GWS metric uses the radius of grasp wrench space as a quantitative grasp-quality score.The implementation uses a simplified friction-less GWS in R^3.
- Objective: The final quality score is a weighted sum of force-closure and GWS metrics, with α = 1.0 and β = 0.01 in the experiments.
B. Learning a Grasp Quality Metric from Point Cloud
PointNetGPD evaluates grasp quality from the point cloud inside the gripper’s closing area. It uses local gripper coordinates, PointNet-style processing, and a lightweight network.
- The network takes the point cloud within the gripper closing area as the grasp representation.
- Points are transformed into a unified local gripper coordinate system before feature extraction.This reduces ambiguity from differing experiment and camera settings.
- Spatial transformations and feature extraction produce a global feature for classifying grasp quality.
- The model has approximately 1.6 million parameters.
Z Gripper Closing Area
A grasp is represented by the point cloud inside the gripper closing area, transformed into local gripper coordinates. The pipeline also includes candidate generation and collision-reduction modifications.
- Z Gripper Closing Area: The grasp representation consists of the point cloud within the gripper closing area.The transformed points serve as the network input representation.
- Z Gripper Closing Area: Point clouds are transformed into unified local gripper coordinates before grasp-quality evaluation.
- Z Gripper Closing Area: GPG samples heuristic grasp candidates from the point cloud before grasp planning.
- Z Gripper Closing Area: Sampled points near the support surface are discarded, and approaching-away or colliding grasps are removed or pulled clear.
VI. EXPERIMENT
The experiments evaluate PointNetGPD in simulation and on robotic hardware, including grasp-quality classification and real-world grasping tasks. Simulation compares models across single-view and full-point-cloud inputs.
- Experiment Details: PointNetGPD is evaluated in both simulation and robotic hardware experiments.Simulation focuses on grasp-quality classification, while hardware experiments test generalization to real-world settings.
- Experiment Details: Simulation compares grasp-quality classification against GPD baselines using 3- and 12-channel configurations.The 15-channel GPD version is reserved for robotic experiments because camera location is unavailable in the dataset.
- Experiment Details: The experiments include a 3-class classification task to verify the validity of the dataset’s grasp scores.
- Experiment Details: Full point cloud input yields better performance than a single view for all considered models.
2) Results Analysis:
PointNetGPD outperforms GPD baselines in grasp-quality classification, including sparse one-view inputs. The 3-class experiment also reports stronger accuracy for the best-quality class than the best class in the 2-class experiment.
- 4.79% improvement is achieved by PointNetGPD over the best GPD baseline on the most difficult 1-viewed point cloud.
- PointNetGPD performs significantly better than all GPD baselines on grasp-quality classification.
- GPD can overfit the training set, while PointNetGPD uses fewer parameters and maintains a performance advantage.The authors associate these results partly with the difference in parameter count.
- The best-quality class in the 3-class experiment has higher accuracy than the best class in the 2-class experiment.The authors suggest that higher-score grasps may be easier to identify.
B. Robotic Experiments
Robotic experiments evaluate PointNetGPD on isolated objects using one-view depth sensing, comparing class formulations with a GPD baseline. The PointNetGPD variants achieve higher average single-object success rates.
- Experimental setup: The experiments use a UR5 arm with a Robotiq 3-finger gripper operating in pinch mode and a single Kinect2 depth view.The gripper’s two contact surfaces move along a 1-D manifold.
- Experimental setup: Twenty-two YCB objects are tested, including 11 previously seen in training and 11 novel objects.Sixteen objects are additionally selected for clutter-removal experiments.
- Compared models: The comparison includes 2-class and 3-class PointNetGPD models alongside a 15-channel GPD baseline.The best and second classes of the 3-class model are also compared to assess multi-class quality labels.
- Evaluation: Single-object trials test each object for ten random orientations, recording success when grasping succeeds or marking failures after five minutes without a valid grasp pose.Performance is evaluated only by success rate.
- Results: The two PointNetGPD methods show higher average success rates than the compared baseline in single-object grasping.Objects with universally 100% or 0% success rates are omitted from Table II.
2) Objects Presented in Dense Clutter:
Dense-clutter experiments compare grasping models on two object sets using success and completion rates. The best class of 3-class PointNetGPD performs best, with a 13.5% average completion-rate improvement over GPD.
- Objects Presented in Dense Clutter: Sixteen objects are divided into Set 1 and Set 2, and each set is tested for five rounds in dense clutter.Set 1 contains six objects with universally perfect isolated-condition success, whereas Set 2 contains two.
- Objects Presented in Dense Clutter: Success rate measures the percentage of successful grasps, while completion rate measures the percentage of objects removed from clutter.The second predicted class of 3-class PointNetGPD is evaluated in addition to the isolated-condition models.
- Results: 13.5% is the averaged completion-rate improvement of the best 3-class PointNetGPD class over GPD.The best class also shows the strongest overall grasping outcomes.
- Results: All models perform better on Set 1 than Set 2, which the authors associate with gripper fit and higher object roughness in Set 1.The best 3-class class is reported as hugely superior to the second class.
- Experimental setup: The robotic setup uses a UR5 arm, Robotiq gripper, and one fixed depth-sensor view for clutter experiments.Figure 6 distinguishes the two clutter object sets with green and blue polygons.
VII. CONCLUSION AND FUTURE WORK
The paper concludes that PointNetGPD detects grasp configurations directly from point sets and outperforms state-of-the-art methods after training on a 350k-example YCB dataset. Future work targets end-to-end candidate generation and simultaneous clutter segmentation.
- VII. CONCLUSION AND FUTURE WORK: PointNetGPD uses PointNet to evaluate grasp quality from imprecise and deficient point clouds.The method is presented as the core module of the grasp pipeline.
- VII. CONCLUSION AND FUTURE WORK: The authors generate a 350k real point-cloud-and-grasp dataset using the YCB object set for training.
- VII. CONCLUSION AND FUTURE WORK: Experiments show that the proposed model outperforms state-of-the-art grasp-detection methods.
- VII. CONCLUSION AND FUTURE WORK: Future work will integrate grasp-candidate generation into the network for end-to-end grasp planning.
- VII. CONCLUSION AND FUTURE WORK: The authors also plan simultaneous clutter segmentation to prevent grasps crossing more than one object.