PaperlayerView exact revision on arXiv

Source-linked AI summary

ROI-based Robotic Grasp Detection for Object Overlapping Scenes

Hanbo Zhang, Xuguang Lan, Site Bai, Xinwen Zhou, Zhiqiang Tian, Nanning Zheng

arXiv:1808.10313v6cs.RO

TL;DR

Grasp detection in overlapping multi-object scenes must account for which object owns each grasp, a capability not adequately handled by existing approaches. The paper proposes ROI-GD, which detects grasps from object-region features and uses an expanded multi-object grasp dataset. ROI-GD performs much better in multi-object scenes while remaining comparable in single-object scenes, with reported robotic success rates of 92.5% and 83.8% in single-object and multi-object scenes.

  • Problem

    Overlapping and occluded objects make grasp detection difficult, while existing cluttered-scene methods may not identify which object each grasp belongs to.

  • Method

    ROI-GD uses a two-stage ROI-based pipeline that proposes object regions and detects grasps from ROI features instead of whole-image features, supported by an expanded VMRD grasp dataset.

  • Results

    ROI-GD performs much better in multi-object scenes and remains comparable with state-of-the-art methods in single-object scenes, with robotic success rates of 92.5% and 83.8% respectively.

  • Takeaways & Limitations

    ROI-based grasp detection supports target-specific grasping by associating detected grasps with their owner objects in overlapping scenes.

Abstract

from arXiv · show

Grasp detection with consideration of the affiliations between grasps and their owner in object overlapping scenes is a necessary and challenging task for the practical use of the robotic grasping approach. In this paper, a robotic grasp detection algorithm named ROI-GD is proposed to provide a feasible solution to this problem based on Region of Interest (ROI), which is the region proposal for objects. ROI-GD uses features from ROIs to detect grasps instead of the whole scene. It has two stages: the first stage is to provide ROIs in the input image and the second-stage is the grasp detector based on ROI features. We also contribute a multi-object grasp dataset, which is much larger than Cornell Grasp Dataset, by labeling Visual Manipulation Relationship Dataset. Experimental results demonstrate that ROI-GD performs much better in object overlapping scenes and at the meantime, remains comparable with state-of-the-art grasp detection algorithms on Cornell Grasp Dataset and Jacquard Dataset. Robotic experiments demonstrate that ROI-GD can help robots grasp the target in single-object and multi-object scenes with the overall success rates of 92.5% and 83.8% respectively.

I. INTRODUCTION

Robotic grasping in overlapping multi-object scenes must identify both feasible grasps and the objects they belong to. ROI-GD addresses this by detecting grasps from object-region features, supported by a new annotated multi-object dataset and evaluations across multi-object and single-object settings.

  • Problem: Overlapping objects make grasp localization difficult and make it difficult to determine which object each detected grasp belongs to.The paper focuses on grasp-owner affiliation while identifying object overlap and occlusion as central challenges.
  • Problem: Existing cluttered-scene methods do not identify grasp-owner affiliations, while one-by-one detection limits efficiency in multi-object scenes.These limitations matter when a robot is asked to grasp a specified object, such as an apple.
  • Approach: ROI-GD uses a two-stage pipeline that first proposes Regions of Interest and then detects grasps from ROI features rather than whole-image features.The first stage provides ROIs in the input image; the second applies a grasp detector to ROI features.
  • Dataset: The paper extends VMRD with manually labeled grasps and grasp-owner affiliations to construct a multi-object grasp dataset.The dataset labels grasp locations and associates each grasp with its owner object index.
  • Results: ROI-GD performs much better in multi-object scenes while remaining comparable with state-of-the-art methods in single-object scenes.The method is evaluated on the multi-object setting and on Cornell and Jacquard single-object datasets.

II. RELATED WORK

Prior grasp detectors largely target single-object or cluttered scenes without preserving which object owns each grasp. ROI-GD addresses this gap while retaining comparable single-object performance.

  • Related approaches: Recent deep-learning grasp detectors achieve state-of-the-art results mainly on single-object datasets.They treat grasps as a specific kind of object and transfer object-detection algorithms to grasp detection.
  • Proposed direction: ROI-GD detects grasps in object regions rather than across the entire image, targeting affiliation-aware grasp detection.The approach is motivated by the observation that human grasp detection is generally object-based.

IV. PROPOSED APPROACH

ROI-GD uses a CNN and region proposals to extract ROI-specific features, then predicts grasp rectangles and graspability for each region. Nonmaximum suppression retains legal ROIs and their best grasp candidates.

  • Network architecture: ROI-GD extracts CNN features from RGB images and uses an RPN to generate object bounding-box proposals called ROIs.The architecture uses ResNet-101 as its backbone, with ROI features cropped from C4 and pooled to W × H.
  • Network architecture: The ROI grasp detector combines pooled features, residual blocks, a grasp rectangle regressor, and a classifier.Each grid cell has k oriented anchors, and the regressor outputs 5 × k offsets for grasp rectangles represented by (x, y, w, h, θ).
  • Grasp prediction: The classifier predicts 2 × k confidence scores for anchors being graspable or ungraspable.For each ROI, the detector produces W × H × k grasp candidates.
  • Grasp prediction: Nonmaximum suppression preserves legal ROIs together with their best grasp candidates.This follows candidate generation and graspability classification for each ROI.

B. Grasp Detection on Regions of Interest

ROI-based detection restricts grasp prediction to features pooled from a specific object region, helping distinguish grasps belonging to different regions. Training assigns each ROI the grasps of its matched object.

  • ROI-based detection: ROI-GD detects grasps on ROIs instead of the whole image to distinguish grasps belonging to different regions.ROIs are object region proposals whose convolutional features are pooled into a common W × H size.
  • ROI matching: Each training ROI is matched to the ground-truth object bounding box with the largest IoU among candidates exceeding 0.5.ROIs without a matching ground-truth box are excluded from grasp-detector training.
  • ROI matching: Only grasp rectangles belonging to the matched object are assigned to that ROI during training.Grasps belonging to other objects are ignored even when they fall within the ROI’s area.

C. Affiliation Construction between Grasps and Objects

The multi-object extension adds an object-detector branch after ROI pooling so each ROI jointly yields object information and grasp candidates. Its multitask loss combines ROI detection with grasp detection.

  • Affiliation construction: An object-detector branch is added after ROI pooling to construct affiliations between specified objects and their grasps.Both object and grasp detectors use ROI features; the object branch classifies ROIs and refines their locations.
  • Affiliation construction: Each ROI produces grasp candidates belonging to that ROI, and NMS preserves results from ROIs most likely to contain objects.This design enables grasping a specified object in multi-object scenes.
  • ROI-specific grasp detection: Unlike whole-image detectors, ROI-GD uses object-region features as input to detect grasps belonging to particular objects.The paper presents this as a direct extension to multi-object scenes while retaining existing grasp-detection advantages.
  • Loss function: The end-to-end network defines one total loss per image while separately modeling ROI and grasp predictions.ROI detection trains the RPN, and grasp detection includes regression and classification components.
  • Loss function: The total objective is a multitask loss combining ROI detection loss and grasp detection losses across ROIs.The balancing coefficient is set to λ = 1/5NROI, where NROI is the number of ROIs used in grasp-detection loss.

A. Dataset

The paper evaluates ROI-GD on VMRD, Cornell Grasp Dataset, and Jacquard Dataset, expanding VMRD with object-affiliated grasp labels for multi-object scenes.

  • VMRD contains 31 categories, 5,185 images, over 17,000 object instances, and 51,000 manipulation relationships.
  • The VMRD dataset uses unique object indices so grasps sharing an index can be assigned to the corresponding object.
  • 4,683 VMRD images were labeled with over 105 grasps, including affiliations between each grasp and its owning object.
  • The VMRD data are divided into 4,233 training images and 450 testing images.
  • Cornell Grasp Dataset has 885 images, while Jacquard Dataset contains 54k images for single-object evaluation.

B. Implementation Details

The evaluation uses VMRD-specific object–grasp criteria alongside single-object dataset splits, while implementation details include augmentation and a multi-object validation setup.

  • Implementation: The implementation applies photometric distortion, horizontal flips, and rotations of ±90° or 180° for data augmentation.
  • VMRD evaluation: VMRD evaluation defines a true positive as a correctly classified object whose box and Top-1 grasp satisfy IoU, Jaccard Index, and angle thresholds.
  • VMRD evaluation: A detected grasp is correct when its Jaccard Index exceeds 0.25 and its angle difference is below 30° from a ground-truth grasp belonging to the object.
  • Cornell evaluation: Cornell evaluation uses image-wise splits for seen objects and object-wise splits to test generalization to unseen object instances.
  • Detection visualization: Fig. 5 presents object detections in the top row and grasp detections in the second row.

D. Validation Results

ROI-GD improves grasp detection in multi-object scenes while remaining competitive on single-object benchmarks. Its performance is affected by anchor configuration, and dataset labeling bias complicates Cornell benchmark interpretation.

  • ROI-GD significantly improves grasp detection in multi-object scenes compared with the cascading Faster-RCNN and FCGN baseline.The baseline assigns grasps to objects using center distance and a confidence threshold above 0.25.
  • Anchor size is an important hyperparameter for network training, with 12×12 and 24 × 24 settings evaluated.
  • Proper k balances easier grasp-rectangle regression against more hard negatives for graspable classification.
  • 93.6% image-wise and 93.5% object-wise accuracy on Cornell Grasp Dataset was achieved at 25.16 FPS.These results were 4.1% and 3.1% lower than the cited state-of-the-art algorithm, respectively.
  • 93.6% accuracy was achieved on Jacquard Dataset using RGD images, a 19.4% gain over the cited comparison method.
  • Cornell performance is biased because many incorrect detections appear feasible for robotic grasping, whereas Jacquard has more grasp labels per object.

E. Robot Experiment

The robot experiments evaluate grasping specified targets alone and amid clutter, using prediction and execution success rates across eight objects. ROI-GD performs well in both settings, but depth-based grasp estimation constrains targets beneath other objects.

  • The evaluation measures success in grasping a specified target alone or among several disturbance objects, using more cluttered scenes than VMRD.
  • Depth information is indispensable for estimating the grasp point and approaching vector from the image-based 5-d grasp configuration.
  • The depth-based method cannot accurately estimate grasp height for targets at the bottom of a pile.Targets therefore had to be placed on or next to other objects rather than below them.
  • 97.5% prediction and 92.5% execution success rates were achieved in single-object scenes across all 8 targets.
  • 88.8% prediction and 83.8% execution success rates were achieved in multi-object scenes across all 8 targets.The grasp closest to the target center with a graspable score above 0.5 was selected for execution.

VI. CONCLUSIONS

ROI-GD detects grasps using ROI features while considering each grasp’s object affiliation, improving performance in multi-object scenes while remaining comparable in single-object scenes.

  • ROI-GD detects grasps on ROI features while considering the affiliations between grasps and their owners.
  • ROI-GD performs much better in multi-object scenes and remains comparable with state-of-the-art algorithms in single-object scenes.
  • 92.5% and 83.8% are the overall success rates for single-object and multi-object grasping tasks, respectively.
Loading 1808.10313v6…