PaperlayerView exact revision on arXiv

Source-linked AI summary

PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds

Peiyuan Ni, Wenguang Zhang, Xiaoxiao Zhu, Qixin Cao

arXiv:2003.09644v1cs.RO

TL;DR

The paper addresses the cost of sampling and local feature extraction for grasping novel objects from sparse point clouds. It uses an end-to-end PointNet++ network with multi-mask loss and Ferrari Canny-based training-data generation, reporting 71.43% success and 91.60% completion rates.

  • Problem

    Existing grasping pipelines require candidate sampling and local feature extraction, which is costly when grasp points are sparse and difficult for sparse 3D data.

  • Method

    A PointNet++ network with multi-mask loss directly predicts grasp poses, categories, and qualities from whole sparse point clouds, supported by Ferrari Canny-based multi-object data generation.

  • Results

    The experiments report 71.43% success rate and 91.60% completion rate, performing better than current state-of-the-art works.

  • Takeaways & Limitations

    The approach provides fast end-to-end grasp prediction without a sampling or search process for sparse point-cloud inputs.

Abstract

from arXiv · show

Grasping for novel objects is important for robot manipulation in unstructured environments. Most of current works require a grasp sampling process to obtain grasp candidates, combined with local feature extractor using deep learning. This pipeline is time-costly, expecially when grasp points are sparse such as at the edge of a bowl. In this paper, we propose an end-to-end approach to directly predict the poses, categories and scores (qualities) of all the grasps. It takes the whole sparse point clouds as the input and requires no sampling or search process. Moreover, to generate training data of multi-object scene, we propose a fast multi-object grasp detection algorithm based on Ferrari Canny metrics. A single-object dataset (79 objects from YCB object set, 23.7k grasps) and a multi-object dataset (20k point clouds with annotations and masks) are generated. A PointNet++ based network combined with multi-mask loss is introduced to deal with different training points. The whole weight size of our network is only about 11.6M, which takes about 102ms for a whole prediction process using a GeForce 840M GPU. Our experiment shows our work get 71.43% success rate and 91.60% completion rate, which performs better than current state-of-art works.

I. INTRODUCTION

The paper targets costly grasp search and sparse, noisy sensing in novel-object manipulation by directly predicting spatial grasps from whole point clouds. It also introduces a fast multi-object data-generation procedure and multi-mask training strategy.

  • I. INTRODUCTION: The approach addresses difficulties from stacked objects, sparse noisy camera data, and selecting high-quality grasps among candidates.These challenges motivate global point-cloud processing and explicit quality prediction.
  • I. INTRODUCTION: The network has about 11.6M weights and performs a whole prediction in about 102ms on a GeForce 840M GPU.The reported timing is for the complete prediction process.
  • I. INTRODUCTION: The PointNet++ network directly predicts grasp poses, categories, and quality scores from whole sparse point clouds without sampling or search.Every point can predict a result, reducing sampling costs when grasp points are sparse.
  • I. INTRODUCTION: A Ferrari Canny-based multi-object grasp detection algorithm constructs cluttered training data by matching single-object grasp candidates.The procedure avoids exhaustive search in large scenes by requiring only collision detection after candidate matching.
  • I. INTRODUCTION: The training resources include 79 YCB objects with 23.7k grasps and a multi-object dataset containing 20k point clouds with annotations and masks.The multi-object dataset is generated using domain randomization.

II. RELATED WORK

Prior grasping methods depend on object knowledge, sampling, local feature extraction, or reinforcement learning. The paper positions 3D point-cloud processing as a way to avoid sampling-heavy pipelines for sparse spatial grasps.

  • II. RELATED WORK: Algorithm choice in grasp perception depends strongly on the amount of prior object knowledge available.Model-based methods use poses and pre-built grasp databases, while learning methods target novel objects.
  • II. RELATED WORK: Many learning pipelines still sample grasps, label them, and extract local features before prediction, making sparse grasp points costly to find.The paper specifically identifies this issue for grasp points that are sparse.
  • II. RELATED WORK: Two-dimensional feature extraction is described as unsuitable for sparse 3D point clouds and spatial grasps.The related-work discussion motivates learning directly from 3D geometric representations.
  • II. RELATED WORK: The grasping model assumes quasi-static Coulomb-friction physics, a two-finger parallel-jaw gripper, and one depth camera with known intrinsics.These assumptions define the physical and sensing setting of the method.
  • II. RELATED WORK: The model additionally assumes that a grasp approaches along the surface normal of the object.This assumption simplifies the spatial grasping model.

B. Definitions

The paper formalizes scenes, cameras, point clouds, grasps, and grasp quality, then defines a learned mapping from observed points to candidate grasps whose highest-quality prediction is executed.

  • B. Definitions: A scene state x contains m object substates, each describing an object model, mass, centroid, pose, and friction coefficient.The camera state separately contains its pose and intrinsic parameters.
  • B. Definitions: The input point cloud y is a 3×N array of N points captured by the depth camera.The point cloud is defined in the camera-observation setting used by the model.
  • B. Definitions: Each spatial grasp specifies a point, approaching vector, opening vector, and finger approaching distance.Under the model assumption, the approaching vector is the grasp-point surface normal.
  • B. Definitions: Continuous Ferrari Canny quality Qfc is paired with a binary quality Qb because regression is harder to train than classification.The binary metric is defined from the continuous metric in the training stage.
  • B. Definitions: The learned function maps point clouds y to all grasps and their combined metrics Qc, and the robot executes the grasp with maximum Qc.Figure 2 distinguishes major and supplementary grasps and visualizes quality from low to high.

A. Single-object Grasp Planning

Single-object planning samples surface points, generates and filters candidate grasps through collision and Ferrari Canny quality checks, and retains major and supplementary grasps for later multi-object use.

  • A. Single-object Grasp Planning: Ferrari Canny quality Qfc is the distance from the wrench-space origin to the nearest boundary point of the primitive-wrench convex hull.The hull contains the origin when the grasp is force-closure.
  • A. Single-object Grasp Planning: For each sampled surface point, the planner generates grasps along the surface normal and tests multiple rotations for collision-free reachability.A rotation is valid when the deepest approach exceeds Dmin.
  • A. Single-object Grasp Planning: The planner computes combined grasp metrics and selects the highest-scoring valid candidate at each point as the major grasp.Points with no generated grasp are recorded as negative grasp points and normals.
  • A. Single-object Grasp Planning: Supplementary grasps are sampled every 1 mm between Dmin and the approach distance to provide alternatives when major grasps are occluded or colliding.Zero-quality supplementary grasps are filtered, and the remainder are sorted by descending quality.
  • A. Single-object Grasp Planning: The reported experiment sets friction to 0.2, samples 300 points, tests 36 rotations, and uses Dmin=20 mm.Object mass is computed from volume using density 0.5 g/cm3.

B. Multi-object grasp planning

The paper generates multi-object grasp-training data by simulating cluttered scenes, checking grasp collisions, and projecting labeled grasp points onto randomized camera point clouds. Multi-mask labels distinguish valid, negative, and unlabeled training points.

  • B. Multi-object grasp planning: Randomly selected object models are thrown into an invisible 200mm×200mm totebox, and the objects plus ground plane form one scene model.The camera optical axis is constrained to pass through the totebox center during randomized capture.
  • B. Multi-object grasp planning: Collision checking retains collision-free grasps or selects the highest-quality collision-free supplementary grasp using Ferrari Canny quality values.If no supplementary grasp is collision-free, the grasp point and normal become negative examples.
  • B. Multi-object grasp planning: Ground points train only the binary grasp-quality output, while object points initially receive no active supervision until matched with generated grasp points.This masking scheme separates supervision for different point types in the sparse scene.
  • B. Multi-object grasp planning: The algorithm generates training samples by matching positive and negative grasp points to camera point clouds with a kd-tree query radius R, assigning labels and masks.Positive points receive pose, binary quality, and continuous quality labels; negative points receive normal labels and masked-out pose and quality targets.

V. TRAINING

The training pipeline normalizes, perturbs, and resamples simulated point clouds before processing them with a PointNet++ network. Its output separates grasp selection from pose prediction through score, category, normal, and rotation blocks.

  • V. TRAINING: The processed point clouds are centralized, scaled to [0, 1], jittered with 1mm Gaussian noise, and resampled to 8192 points before network input.The scaling factor is M=0.33 in the experiment.
  • V. TRAINING: A PointNet++ feature extractor followed by two one-dimensional convolution filters processes the normalized point clouds.The architecture uses successive set-abstraction and feature-propagation levels.
  • V. TRAINING: The network produces an 8192×9 feature block divided into score, category, normal, and rotation outputs.The score and category blocks identify grasp-capable points, while normal and rotation predict approach and opening vectors.
  • V. TRAINING: The approaching distance is clipped at dmax: it equals ddeepest below the threshold and dmax otherwise.The threshold dmax is set to 40mm in the experiment.
  • V. TRAINING: Training data comprise 20k cluttered multi-object point clouds with labels and masks, split into 80% training and 20% testing sets.The scenes are constructed from YCB object models using single-object and multi-object grasp planning.

A. Simulation experiments

The simulation compares grasp-quality recall and computation time for three algorithms, showing advantages for the proposed end-to-end method on sparse point clouds.

  • Simulation results: The proposed algorithm performs better than other algorithms in RAHP evaluation on sparse, randomly posed point clouds.The authors attribute this to domain randomization and global feature extraction.
  • Simulation results: The proposed method’s prediction time is independent of sampling-point count because it uses no sampling or search process.The compared methods become slower as sampling points increase, with much of their time spent searching and sampling.
  • Prediction visualization: Figure 7 visualizes predicted grasps and scores above corresponding point-cloud labels, with grasp colors ranging from red for low scores to green for high scores.The predictions also regress poses and scores while learning object-size information.

B. Robot experiments

Robot experiments evaluate clutter removal with known and novel objects using a UR10, showing higher performance for some challenging objects with sparse or noisy grasp points.

  • Experimental setup: The clutter-removal experiment selects 15 YCB objects and 15 novel objects for testing with a UR10 and a 60 mm parallel-jaw gripper.The real scene uses a Realsense SR300 camera and MoveIt! Cartesian planning.
  • Experimental protocol: Each algorithm is tested for 20 trials with six randomly chosen objects per scene and at most 10 action attempts per trial.Baseline methods use their maximum sampling points, while the proposed method executes the grasp with maximum Qc.
  • Metrics: Success rate measures successful grasps, whereas completion rate measures the percentage of objects removed from clutter.The reported table is titled “Results of clutter removal experiments.”
  • Robot results: The proposed method achieves higher success for heavy objects and for bowls and cups with sparse, noise-affected grasp points.The authors associate these results with global prediction and training on sparse points.

VII. FUTURE WORK

The paper identifies improved network structure as future work for increasing performance.

  • VII. FUTURE WORK: Future work will consider a better network structure to improve performance.This is the paper’s stated next direction after the reported experiments.
  • VII. FUTURE WORK: The proposed approach directly predicts grasp poses, categories, and qualities from whole sparse point clouds without sampling or search.This summarizes the system whose network structure is targeted for future improvement.
  • VII. FUTURE WORK: The reported experiments obtain a 71.43% success rate and 91.60% completion rate before the proposed future network improvements.These are the paper’s stated overall experimental results.
Loading 2003.09644v1…