PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Detecting Robotic Grasps

Ian Lenz, Honglak Lee, Ashutosh Saxena

arXiv:1301.3592v6cs.LGcs.CVcs.RO

TL;DR

Robotic grasp detection from noisy RGB-D views requires evaluating many candidates quickly while learning useful multimodal features without hand-engineering. The paper addresses this with a two-stage deep-learning cascade and structured multimodal regularization, reporting improved detection and successful grasp execution on Baxter and PR2.

  • Problem

    Robotic grasping requires fast, robust selection of effective grasps from noisy partial views, while hand-designed features are cumbersome and difficult to adapt to RGB-D inputs.

  • Method

    The method combines a fast first deep network that prunes candidate grasps, a more accurate second network, and structured regularization for multimodal feature learning.

  • Results

    The method outperforms previous state-of-the-art grasp-detection methods and achieves 84% and 89% execution success rates on Baxter and PR2, respectively.

  • Takeaways & Limitations

    The two-stage deep-learning system with group regularization robustly detects grasps for a wide range of objects, including objects previously unseen by the system.

  • Takeaways & Limitations

    The system relies on local features, so some regions that are preferable for grasping may not be readily distinguishable from local information alone.

Abstract

from arXiv · show

We consider the problem of detecting robotic grasps in an RGB-D view of a scene containing objects. In this work, we apply a deep learning approach to solve this problem, which avoids time-consuming hand-design of features. This presents two main challenges. First, we need to evaluate a huge number of candidate grasps. In order to make detection fast, as well as robust, we present a two-step cascaded structure with two deep networks, where the top detections from the first are re-evaluated by the second. The first network has fewer features, is faster to run, and can effectively prune out unlikely candidate grasps. The second, with more features, is slower but has to run only on the top few detections. Second, we need to handle multimodal inputs well, for which we present a method to apply structured regularization on the weights based on multimodal group regularization. We demonstrate that our method outperforms the previous state-of-the-art methods in robotic grasp detection, and can be used to successfully execute grasps on two different robotic platforms.

I. INTRODUCTION

The paper applies deep learning to robotic grasp detection from RGB-D views, addressing the need for fast evaluation of many candidate grasps and effective multimodal feature learning. It introduces a cascaded detector and structured multimodal regularization, reporting improved detection performance and successful execution on two robot platforms.

  • Motivation: Robotic grasp detection must infer effective gripper placements from noisy, partial camera views while meeting stringent speed requirements in closed-loop control.Hand-designed features are cumbersome and time-consuming, particularly when incorporating RGB-D modalities.
  • Problem and contribution: The work applies deep learning to grasp detection rather than recognition, targeting the grasp with the highest chance of success rather than merely locating an object.Robotic grasping differs from generic object detection because multiple grasps may be valid and orientation is critical.
  • Method: A structured multimodal regularization method encourages learned features to use subsets of RGB-D modalities without forcing fixed modality assignments.This provides a middle ground between concatenating modalities and training separate features for each modality.
  • Method: A two-stage cascade uses a faster first network to prune unlikely grasps before a slower, more accurate second network re-evaluates top-ranked candidates.The cascade reduces computational cost while providing robust early rejection of false positives.
  • Results: The method improves recognition and detection performance on grasp rectangles and outperforms prior state-of-the-art and deep-learning methods.The paper evaluates the approach on a challenging grasping dataset.
  • Results: 84% and 89% grasp-execution success rates are reported for Baxter and PR2, respectively, on a highly varied object set.The algorithm was implemented on both robotic platforms.

A. Robotic Grasping

Robotic grasping from sensory data must handle incomplete, noisy observations and many possible configurations. Prior approaches reduce this challenge through learning, 3D data, or structured regularization, but detection differs from ordinary object recognition.

  • A grasp is commonly defined as an end-effector configuration achieving partial or complete form- or force-closure of an object.
  • Known-model grasp synthesis can optimize grasps using complete 3D object shape and physical information, but is inapplicable when the full model is unavailable.
  • Sensor-based grasping uses limited and potentially noisy color, depth, or tactile information rather than complete object models.
  • Learning methods improve grasping novel objects, but earlier systems still rely substantially on hand-engineered input features.
  • Robotic grasp detection differs from generic object detection because multiple grasps may be valid and the system must select one with the highest chance of success, including its orientation.
  • Multimodal deep-learning systems have commonly either learned separate low-level modality features or concatenated modalities, while structured regularization offers another approach.

III. DEEP LEARNING FOR GRASP DETECTION: SYSTEM AND MODEL

The paper detects grasps from a single RGB-D view by learning features and grasp rankings with deep networks. Its system uses a fast first pass to generate candidates and a larger second pass to select the best grasp.

  • The approach learns both RGB-D features and the weights used to rank prospective grasps, replacing previously hand-engineered features.
  • Structured multimodal regularization improves learned RGB-D features without constraining the network structure.
  • A small deep network scores image candidates first, while a larger network re-ranks the top candidates to produce one best-ranked grasp.
  • Potential grasps are represented as oriented image-plane rectangles parameterized by position, width, height, and orientation.
  • Near-exhaustive evaluation of oriented rectangles is computationally expensive for real-time robotic grasping.
  • The two-stage design uses a reduced feature set for fast candidate generation and a larger feature set for more robust candidate ranking.
  • The graspability model uses two sigmoid hidden layers followed by a logistic classifier that predicts the probability of a feasible grasp.

A. Inference and Learning

Learning initializes hidden representations unsupervised and then fine-tunes the full network for grasp recognition. Inference uses separate smaller and larger models in a cascaded re-ranking process.

  • Training uses paired feature vectors and binary graspability labels to optimize recognition accuracy.
  • Sparse auto-encoder pretraining initializes hidden-layer weights before supervised learning.
  • The first auto-encoder reconstructs the input, after which the second reconstructs the first hidden-layer representation.
  • Supervised training jointly learns classifier weights and fine-tunes both hidden layers using likelihood maximization with weight regularization.
  • During two-stage inference, a smaller network selects the top T rectangles and a larger network re-ranks them to obtain one best grasp.

IV. SYSTEM DETAILS

The system extracts aligned multimodal features from each candidate rectangle and normalizes them for learning. Preserving aspect ratio and accounting for masked padding are important for accurate graspability predictions.

  • Features are extracted locally from each RGB-D rectangle after rotating it to align its edges with the gripper plates.
  • Each aligned 24x24 image yields seven channels and 4032 input features.
  • Preserving aspect ratio prevents image distortion from making non-graspable rectangles appear graspable.
  • The channels contain YUV color, depth, and X, Y, and Z surface-normal components aligned relative to the gripper.
  • Multimodal normalization scales each channel using statistics across all its features so modalities have more closely matched statistics.

B. Preserving Aspect Ratio.

The method preserves grasp-feature aspect ratios while correcting padding-induced bias by scaling masked multimodal inputs before network processing. This improves detection substantially and reduces preference for square rectangles.

  • Motivation: Padding and distortion can make non-graspable rectangles appear graspable or favor rectangles with less padding.The problem is especially relevant for thin grasp rectangles, whose receptive fields may contain many padded zero values.
  • Mask-based scaling: The method scales each modality’s input magnitude according to the fraction of that modality masked out.Each modality can use a different mask, so scaling is computed separately by mode.
  • Mask-based scaling: The scaled input is used by the network, while reconstruction error is computed against the original input after scaling.This avoids quadratically increasing the squared-error contribution of heavily masked cases.
  • Results: Mask-based scaling improves grasping results by over 25% on both point and rectangle metrics.It also removes an inherent bias toward square rectangles and produces a wider aspect-ratio range closer to the ground-truth data.

V. STRUCTURED REGULARIZATION FOR FEATURE LEARNING

The paper introduces structured multimodal regularization so learned features can combine selected input modalities without requiring every feature to use all modes. The approach addresses limitations of dense, modality-separated, and conventional regularized feature learning.

  • Motivation: Concatenating modalities can overfit by learning features that include all modes or miss associations between modalities with different statistics.Modality-specific first-layer features instead assume ideal low-level features are purely unimodal.
  • Motivation: The ideal feature set may use some but not all modalities, a case not directly handled by dense or modality-specific models.Choosing among these approaches can also require time-consuming tuning and comparison.
  • Proposed regularization: Structured multimodal regularization lets features correlate across modalities while penalizing the number of modalities used by each hidden unit.The penalty discourages weak cross-modal correlations and is applied separately to modality-based weight groups.
  • Limitations of standard regularization: A simple L1 penalty degraded the quality of remaining-mode features when made strong enough to induce multimodal sparsity.The method therefore uses group-wise regularization rather than relying on standard L1 alone.
  • Regularization design: Max-based group regularization concentrated weights in fewer modes but could saturate other weights and produce duplicate features.These effects reduced the effective diversity of the learned feature set.
  • Regularization design: The proposed group penalty directly targets the number of modes with nonzero maxima without further constraining their nonzero weights.A differentiable approximation is used for gradient-based optimization.

VI. EXPERIMENTS

Experiments use an extended Cornell grasping dataset with image- and object-disjoint validation, evaluating a deep network and a two-pass variant. The two-pass system uses a smaller network for ranking and a larger network for reranking.

  • Evaluation protocol: Five-fold cross-validation evaluates both per-image and per-object splits, with hyperparameters selected on 300 separate grasps.Per-object splits prevent training and validation images from sharing an object.
  • Networks: The main network uses 200 hidden units at each of two layers, while the first-pass network uses 50 hidden units at each layer.Both networks receive seven 24x24 channels, corresponding to 4032 input features.
  • Inference: During inference, the smaller network exhaustively searches candidate rectangles and the larger network reranks its top 100.This implements the two-pass detection procedure.

B. Baselines

The experiments compare learned features and multimodal regularization against hand-engineered and alternative deep-learning baselines. Results favor the proposed system across recognition and detection, while showing that evaluation metric choice and input masking materially affect conclusions.

  • Baselines: Recognition comparisons include hand-engineered features, FPFH combinations, standard L1 regularization, and modality-separated first-layer features.A linear SVM was used for classification, with chance performance also reported.
  • Metrics: The detection evaluation compares top-ranked rectangles using point and rectangle metrics.The point metric ignores orientation, whereas the rectangle metric rejects orientation errors above 30° and uses intersection over union.
  • Recognition results: 9% improvement over features from and 4.1% over those features with FPFH are reported for recognition.L1 and group regularization perform similarly, while separate first-layer features reduce performance slightly.
  • Recognition results: Using all modalities gives the highest recognition accuracy, while depth alone is the strongest single-modality input.Combining depth with surface normals improves over either alone, indicating non-redundant information.
  • Detection results: Deep-learning approaches outperform hand-engineered features by up to 13% on the point metric and 17% on the rectangle metric after mask-based scaling.Training separate first-layer features is the exception among the deep-learning approaches.
  • Detection results: Structured multimodal regularization improves results over standard L1 regularization by up to 1.8%.The two-pass system improves efficiency and accuracy, outperforming the second-pass network alone by up to 2.4%.
  • Detection results: The system outperforms all baselines except on the point metric in the object-wise split, where the point metric can overstate performance.The rectangle metric is presented as a better indicator of grasp-detection accuracy.
  • Adaptability: The detector can flexibly constrain grasp predictions to different Baxter gripper settings.Figure 11 illustrates results for wide and thin gripper configurations.

B. Multimodal Group Regularization

Group regularization improves grasp detection by integrating RGB-D modalities more robustly than simple L1 regularization, especially when evaluating unseen objects.

  • Group regularization improves detection accuracy over simple L1 regularization, particularly for object-wise splits.The larger improvement on unseen objects is attributed to reduced overfitting.
  • The figures compare viable grasps from group-regularized networks with failures from networks trained using L1 regularization or a single-stage system.RGB appears above depth, with colored edges indicating the gripper.
  • Group regularization finds valid grasps in cases where L1 regularization does not.The cited examples include sunglasses, nail polish, and scissors.
  • The group-regularized network combines modalities more effectively and is more robust to depth noise and missing data.L1-selected grasps can appear valid in individual RGB or depth channels while being invalid when all modalities are considered.

C. Two-stage Detection System

The two-stage detector uses a fast first pass to prune candidates before a more accurate second pass, improving both efficiency and detection performance in robotic experiments.

  • The two-pass system reduced the number of rectangles evaluated by the large network by roughly 1000×.The first-stage network prunes weak candidates before second-stage evaluation.
  • Detection performance increased by up to 2.4% versus a single pass with the large network.The small network alone significantly underperforms the larger network.
  • In example cases, the two-stage system selected clearly successful grasps after pruning weak grasps that the single-stage system might choose.The compared grasp rectangles are shown over RGB and depth images.
  • Average MATLAB inference time fell from 24.6s/image for exhaustive large-network search to 13.5s/image with the two-stage system.The efficiency gain accompanies the reported accuracy improvement.
  • On Baxter, the algorithm successfully executed 84% of 100 grasp trials and detected valid grasps in 92%.Eight percent involved valid detections that Baxter failed to execute correctly.
  • PR2 succeeded in 89% of trials, aided by its wider gripper span, complete closure, and greater gripping force.PR2 achieved 100% success on the plush cat and umbrella cases that challenged Baxter’s gripper setting.
  • Performance varied with object geometry, sensor quality, gripper constraints, and approach direction.Reported successes included irregular objects, rims, handles, and nonvertical approach vectors, while glossy objects caused depth-related failures.

IX. DISCUSSION AND FUTURE WORK

The discussion identifies scope boundaries for gripper configurations, pose outputs, semantic grasp preferences, and visual servoing, while outlining extensions to broader robotics problems and modalities.

  • The system directly targets two-fingered parallel-plate grippers, requiring new labeled training data for other fixed gripper configurations.The authors suggest applicability to jamming grippers and differently shaped two-fingered contact surfaces.
  • The detector does not directly solve full 3D gripper orientation, which is determined after rectangle detection from object surface normals.A proposed extension would search the full 6-DoF pose using 3D windows and voxels.
  • The system outputs a gripper pose but not the finger configuration required by multifingered reconfigurable hands.The authors propose using it as a heuristic to reduce the search space for configurations.
  • Local features may miss semantically preferred grasp regions, such as handles rather than blades or barrels.A region-level semantic graspability term is proposed as an efficient addition to local detection scores.
  • Visual servoing was implemented heuristically, while feature learning is proposed for future hand-camera refinement.The existing procedure attempts to center the segmented object under the hand camera.
  • The approach could extend to object detection, obstacle detection, tracking, visual servoing, and other multimodal robotics problems.The structured multimodal regularizer is described as applicable beyond color, depth, and surface normals.

X. CONCLUSIONS

The paper presents deep learning for RGB-D robotic grasp detection, combining learned features with group regularization and a two-stage system. Experiments report stronger performance than hand-engineered and existing multimodal learning approaches across offline and real-robot settings.

  • Deep learning avoids hand-engineering features and significantly outperforms well-designed hand-engineered features from previous work.This conclusion concerns robotic grasp detection from RGB-D data.
  • Group regularization produces better multimodal features than existing deep learning approaches to multimodal data.The paper reports evidence from extensive offline and real-robot experiments.
  • The two-stage deep learning system with group regularization robustly detects grasps for a wide range of objects, including previously unseen objects.The conclusion covers both offline evaluation and real robotic platforms.
Loading 1301.3592v6…