PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images

Minghao Gou, Hao-Shu Fang, Zhanda Zhu, Sheng Xu, Chenxi Wang, Cewu Lu

arXiv:2103.02184v1cs.ROcs.CV

TL;DR

General grasping remains challenging because existing approaches may constrain grasp poses or rely on unstable depth data. RGBD-Grasp predicts gripper orientations from RGB and uses depth-guided analytic search for other grasp parameters. It reports state-of-the-art benchmark results and high success rates in single-object and cluttered robot scenes.

  • Problem

    Existing methods may restrict grasp poses to few degrees of freedom or rely on unstable depth images or point clouds.

    Source: §I · p. 1

  • Method

    RGBD-Grasp predicts pixel-wise gripper orientations from RGB, then uses heatmaps and depth in Fast Analytic Searching to calculate gripper width and distance.

    Source: §I · p. 1

  • Results

    RGBD-Grasp achieves state-of-the-art GraspNet-1Billion results and reports 91.7% average success in single-object scenes and 91.1% in cluttered scenes.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors report robustness to depth sensor noise and generalization to novel objects, with experiments also testing domain variation.

    Source: §I · p. 2, 2) Results and Analysis · p. 5

Abstract

from arXiv · show

General object grasping is an important yet unsolved problem in the field of robotics. Most of the current methods either generate grasp poses with few DoF that fail to cover most of the success grasps, or only take the unstable depth image or point cloud as input which may lead to poor results in some cases. In this paper, we propose RGBD-Grasp, a pipeline that solves this problem by decoupling 7-DoF grasp detection into two sub-tasks where RGB and depth information are processed separately. In the first stage, an encoder-decoder like convolutional neural network Angle-View Net(AVN) is proposed to predict the SO(3) orientation of the gripper at every location of the image. Consequently, a Fast Analytic Searching(FAS) module calculates the opening width and the distance of the gripper to the grasp point. By decoupling the grasp detection problem and introducing the stable RGB modality, our pipeline alleviates the requirement for the high-quality depth image and is robust to depth sensor noise. We achieve state-of-the-art results on GraspNet-1Billion dataset compared with several baselines. Real robot experiments on a UR5 robot with an Intel Realsense camera and a Robotiq two-finger gripper show high success rates for both single object scenes and cluttered scenes. Our code and trained model will be made publicly available.

I. INTRODUCTION

RGBD-Grasp separates 7-DoF grasp detection into an RGB orientation-prediction stage and a depth-guided search stage. It reports state-of-the-art GraspNet-1Billion results and high success rates in single-object and cluttered robot scenes.

I. INTRODUCTION

  • Prior 2D planar methods constrain grippers to top-down approaches, whereas 6-DoF methods using RGB-D partial-view point clouds rely on depth data that is less stable than RGB because of potential sensor failure.Traditional physical-analysis methods require accurate object models, which may not always be available, and are usually time-consuming and computationally expensive.Source: §I · p. 1
  • RGBD-Grasp uses RGB to predict pixel-wise gripper orientations, then uses heatmaps and depth in Fast Analytic Searching to calculate opening width and distance to the grasp point.The pipeline takes a monocular RGBD image as input and outputs a 6-DoF pose plus gripper width.Source: §I · p. 1
  • The authors report 10.5 FPS, robustness to depth sensor noise, and generalization to novel objects.They attribute stability to using RGB and describe high-DoF poses as addressing planar restrictions.Source: §I · p. 2
  • AP improves by 2.20, 2.56, and 1.57 for seen, unseen, and novel objects with Kinect, and by 0.42, 1.12, and 1.70 with RealSense; robot success averages 91.7% in single-object and 91.1% in cluttered scenes.The authors report state-of-the-art performance on GraspNet-1Billion and conduct the robot experiments with a UR5 and RealSense camera.Source: §I · p. 2

II. RELATED WORKS

Prior grasping work is grouped into 2D planar and 6-DoF approaches. RGBD-Grasp combines RGB and depth information to address limitations associated with these approaches.

II. RELATED WORKS

  • Prior work is divided into 2D planar grasping and 6-DoF grasping methods.Several review papers provide detailed analyses of the field.Source: §II · p. 2

A. 2D Planar Grasping

  • 2D planar methods represent top-down grasps but cannot express many suitable grasps because of their strong pose constraint.The section reviews methods that search, rank, or predict planar grasp poses.Source: A. 2D Planar Grasping · p. 2

B. 6-DoF Grasping

  • 6-DoF methods allow arbitrary approach directions, but many rely on partial-view point clouds whose depth can be inaccurate for dark, transparent, or reflective materials.RGBD-Grasp incorporates RGB and depth to alleviate the demand for high-quality depth; experiments demonstrate robustness to depth noise and domain variation.Source: B. 6-DoF Grasping · p. 2

III. PROBLEM STATEMENT

The paper defines grasp poses, RGBD images, and a parallel-jaw gripper configuration. A grasp pose has seven variable components, while the gripper’s length and height are fixed by its configuration.

A. Definitions

  • A grasp pose is represented by three translations, three rotations, and gripper width.The schematic specifies that the gripper length and height are fixed according to its configuration.Source: A. Definitions · p. 2, Figure 2 · p. 3
  • An RGBD image consists of an RGB image and a depth map.The RGB image is represented as C and the depth map as D.Source: A. Definitions · p. 2
  • The paper considers a parallel-jaw gripper described by its height, length, and maximum width.These configuration dimensions are shown in Figure 2.Source: A. Definitions · p. 2

B. Problem Statement

Given an RGBD image and gripper configuration, the problem is to choose a fixed-size set of grasp poses that maximizes grasp success rate. Success is defined by whether the object is lifted successfully.

B. Problem Statement

  • Given an RGBD image and gripper configuration, choose k grasp poses to maximize grasp success rate.The objective is stated for a fixed k.Source: B. Problem Statement · p. 3
  • A grasp counts as successful when the object is lifted successfully.The success indicator is defined over the environment, image, gripper configuration, and grasp pose.Source: B. Problem Statement · p. 3
  • The algorithm should predict enough grasp poses to cover the scene and provide alternative candidates for execution.The candidate set contains k poses.Source: B. Problem Statement · p. 3

A. Overview

RGBD-Grasp separates image-plane orientation prediction from width and distance estimation. AVN predicts orientation classes at image locations as a heatmap, using an encoder-decoder architecture.

A. Overview

  • AVN generates gripper orientations at image positions, while FAS uses high-confidence predictions, camera intrinsics, and depth to calculate widths and distances.This division assigns image-based orientation prediction to AVN and depth-guided geometric estimation to FAS.Source: A. Overview · p. 3

B. Angle-View Net

  • Rather than directly regress rotations, AVN treats orientation as classification over sampled approach views and in-plane angles.The paper notes that a location may support several feasible rotations, making direct regression impractical.Source: B. Angle-View Net · p. 3
  • AVN outputs an Angle-View Heatmap (AVH), a 3D tensor of orientation-class confidence scores for each image grid.The encoder-decoder uses ResNet features followed by pixel shuffle and dense upsampling convolution layers to generate the AVH.Source: B. Angle-View Net · p. 3

C. Fast Analytic Searching

Fast Analytic Searching estimates gripper width and distance by filtering sampled grasp poses. This avoids learning dimensional quantities directly from RGB values.

C. Fast Analytic Searching

  • FAS addresses width and distance estimation by filtering poses because these dimensional quantities are difficult to learn from dimensionless RGB values and depend on camera intrinsics.Images at different scales can have the same RGB values, and a trained estimator may not generalize across camera intrinsic parameters.Source: C. Fast Analytic Searching · p. 4
  • FAS samples candidate widths and distances, then rejects poses with points in the gripper-occupied space or no points in the grasping space.Widths are sampled from 0 to a maximum; distances span locations below and above the corresponding point on the depth-reconstructed partial point cloud.Source: C. Fast Analytic Searching · p. 4
  • Among the remaining good poses, grasp-pose non-maximum suppression selects the candidate with the largest distance and smallest width.The figure depicts depth-reconstructed point clouds and sampled candidates, distinguishing collisions, empty grasp spaces, and remaining poses.Source: Figure 6 · p. 4

V. IMPLEMENTATION DETAILS

The authors use GraspNet-1Billion to train AVN because its dense annotations support ground-truth heatmaps for grasps in multiple directions.

V. IMPLEMENTATION DETAILS

  • The 2D planar grasp datasets considered have top-down grasps and sparse annotations, limiting their use for defining grasps in all directions and generating ground-truth AVHs.The cited examples are the datasets of Jiang et al. and Jacquard [10].Source: A. Dataset · p. 4
  • GraspNet-1Billion provides dense 6-DoF grasp annotations used to generate ground-truth AVHs.The authors describe it as the only publicly available 6-DoF grasping dataset they know of.Source: A. Dataset · p. 4
  • Ground-truth AVHs label all angle-view-place combinations negative initially, then mark the closest angle, view, and place for each dataset grasp as positive.Across 100 training scenes, the passage reports about 110 million positive and 64 billion negative Kinect samples, and 143 million positive and 64 billion negative RealSense samples.Source: A. Dataset · p. 4

B. Network and Training

The implementation specifies image and heatmap resolutions, sampled grasp candidates, and network training settings. The experiments compare results on GraspNet-1Billion and a UR-5 robot.

B. Network and Training

  • The network takes 384×288 RGB images and outputs AVHs at 96×72 resolution.Source: B. Network and Training · p. 4
  • Training uses 360 orientation heatmaps, widths sampled from 1 cm to 10 cm, and depths spanning 2 cm below to 2 cm above the origin.Views and angles are sampled as V = 60 and A = 6; width and depth steps are 1 cm.Source: B. Network and Training · p. 5
  • The authors train for 60,000 batches on Kinect and RealSense images using Adam, ImageNet-pretrained weights, and heavy crop and color-jitter augmentation.Training uses two NVIDIA 2080Ti GPUs, batch size 16, and learning rate 10^-3, decayed by 10 every 20,000 batches.Source: B. Network and Training · p. 5

VI. EXPERIMENTS

  • The experiments evaluate the method on GraspNet-1Billion and a UR-5 robot, comparing it with baselines and analyzing differences.Source: §VI · p. 5

A. Evaluation Metric

The section defines GraspNet-1Billion’s AP evaluation and compares RGBD-Grasp with baseline methods. It reports benchmark performance, robustness to depth noise, and cross-camera testing.

A. Evaluation Metric

  • AP averages precision across friction coefficients and candidate counts from 1 to 50 after non-maximum suppression and collision rejection.Remaining grasps are labeled using a force-closure metric.Source: A. Evaluation Metric · p. 5

B. Results on GraspNet-1Billion

  • The baselines are GG-CNN [26], Chu et al., and Fang et al.; GG-CNN uses its Cornell-pretrained model after retraining performed poorly.Chu et al. is retrained on converted GraspNet-1Billion data; an ablation removes AVN and randomly samples candidates for FAS.Source: 1) Baselines · p. 5

2) Results and Analysis:

  • Planar methods [26] are harmed by their DoF restrictions, while randomly sampled orientations yield few reasonable grasps despite FAS filtering.The section points to Table I for scores and Figure 8 for visual examples.Source: 2) Results and Analysis · p. 5
  • RGBD-Grasp slightly outperforms Fang et al. on the original dataset, with a smaller score decrease under depth noise.Fang et al.'s score drops significantly when depth images are noisy.Source: 2) Results and Analysis · p. 5
  • In cross-camera testing, RGBD-Grasp performs well when trained on a different camera, while Fang et al. receives low scores.The passage attributes Fang et al.'s result to its reliance on depth images.Source: 2) Results and Analysis · p. 5

C. Real Robot Experiments

The real-robot experiments use a UR-5 setup and test grasping on single objects and cluttered scenes. The paper reports satisfactory cluttered-scene success and points to results tables for both settings.

C. Real Robot Experiments

  • The robot setup uses a UR-5, RealSense camera, and Robotiq two-finger gripper, with MoveIt planning collision-avoiding trajectories.The trajectory planning accounts for the robot, gripper, camera, and objects.Source: C. Real Robot Experiments · p. 5
  • Single-object trials randomly place nine GraspNet-1Billion objects and run 20 experiments per object; results are reported in Table III.Objects with different difficulties are selected to examine generalization.Source: 2) Experiment on Cluttered Scenes · p. 5, Table III · p. 6

2) Experiment on Cluttered Scenes:

  • Figure 8 visualizes four algorithms on two GraspNet-1Billion scenes; redder grippers indicate more confident grasp poses.The examples use Kinect-captured images.Source: Figure 8 · p. 6
  • RGBD-Grasp achieves a satisfactory success rate in cluttered-scene experiments, with results reported in Table IV.The setup uses scenes with 4–8 randomly selected and placed objects, repeatedly executing the best grasp candidate until objects are taken away.Source: 2) Experiment on Cluttered Scenes · pp. 5–6, Table IV · p. 6

VII. CONCLUSION

RGBD-Grasp separates 7-DoF grasp detection into RGB-based orientation and location prediction and depth-based width and distance calculation. It reports state-of-the-art GraspNet-1Billion results and high success rates in single-object and cluttered robot experiments.

VII. CONCLUSION

  • RGBD-Grasp decouples grasp detection into AVN heatmaps from a monocular RGB image and FAS calculation of gripper widths and distances using heatmaps and depth.The pipeline produces 7-DoF grasp poses.Source: §VII · p. 6
  • The method achieves state-of-the-art results on the GraspNet-1Billion benchmark.Source: §VII · p. 6
  • UR-5 experiments with a Robotiq two-finger gripper report high success rates in both single-object and cluttered scenes.Source: §VII · p. 6
Loading 2103.02184v1…