PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Approaches to Grasp Synthesis: A Review

Rhys Newbury, Morris Gu, Lachlan Chumbley, Arsalan Mousavian, Clemens Eppner, Jürgen Leitner, Jeannette Bohg, Antonio Morales, Tamim Asfour, Danica Kragic, Dieter Fox, Akansel Cosgun

arXiv:2207.02556v3cs.RO

TL;DR

The paper reviews deep-learning approaches for robotic grasp synthesis, focusing on six-degree-of-freedom end-effector poses. It analyzes 85 publications, organizes them into common methodologies and supporting methods, and distills ten key takeaways for future research.

  • Problem

    Grasp synthesis is a high-dimensional search or optimization problem involving gripper-object poses, joint configurations, contact conditions, and grasp-quality criteria.

  • Method

    The authors conduct a systematic review of 85 deep-learning publications, clustering them by methodologies, datasets, object sets, and supporting methods.

  • Results

    The review identifies four grasping methodologies—sampling, direct regression, reinforcement learning, and exemplar methods—plus shape completion and affordances as supporting methods.

  • Takeaways & Limitations

    The survey distills ten key takeaways intended to inform future robotic grasping and manipulation research.

  • Takeaways & Limitations

    The reviewed literature lacks consistent performance metrics and sufficient real-world grasping datasets.

Abstract

from arXiv · show

Grasping is the process of picking up an object by applying forces and torques at a set of contacts. Recent advances in deep-learning methods have allowed rapid progress in robotic object grasping. In this systematic review, we surveyed the publications over the last decade, with a particular interest in grasping an object using all 6 degrees of freedom of the end-effector pose. Our review found four common methodologies for robotic grasping: sampling-based approaches, direct regression, reinforcement learning, and exemplar approaches. Additionally, we found two `supporting methods` around grasping that use deep-learning to support the grasping process, shape approximation, and affordances. We have distilled the publications found in this systematic review (85 papers) into ten key takeaways we consider crucial for future robotic grasping and manipulation research. An online version of the survey is available at https://rhys-newbury.github.io/projects/6dof/

I. INTRODUCTION

This survey examines deep-learning methods for synthesizing 6-DoF grasp poses, motivated by the complexity and uncertainty of real-world grasping. It organizes 85 publications into four main methodologies and two supporting methods, and distills ten key takeaways.

  • Grasping controls an object's motion by applying forces and torques at contacts, making it central to most robot manipulation tasks.Finding suitable grasps is a high-dimensional search over gripper-object poses, joint configurations, and contact conditions.
  • Analytical approaches require knowledge of object geometry, material properties, and dynamics that are rarely directly observable from real-world sensory data.
  • 6-DoF grasp synthesis considers the full position and orientation of the gripper coordinate frame, rather than a constrained top-down pose.The review covers both parallel grippers and dexterous hands.
  • Deep learning supports 6-DoF grasp synthesis through models that adapt to variations in object shape, pose, and other grasping factors.The review associates these methods with accuracy, flexibility, reduced manual engineering, and robustness to variability.
  • The survey differs from broader manipulation reviews by focusing specifically on deep learning for grasp synthesis and providing a detailed 6-DoF discussion.
  • The review analyzes 85 publications using four method categories: sampling, direct regression, reinforcement learning, and exemplar approaches.It also identifies shape completion and affordances as supporting methods around grasping.
  • The authors synthesize the reviewed literature into ten key takeaways for future robotics and manipulation research.

II. NOTATIONS AND ANALYTICAL GRASPING

The paper defines grasping concepts and distinguishes 4-DoF and 6-DoF pose representations before reviewing analytical grasping. Analytical methods offer mathematical guarantees but rely on simplified models and assumptions that limit real-world applicability.

  • A grasp pose specifies the robot hand's position and orientation, while most reviewed formulations aim to learn the final robot pose for successful grasping.
  • A 4-DoF grasp uses 3D position and rotation about a commonly gravity-aligned approach vector, often representing top-down grasping.
  • A 6-DoF grasp specifies 3D position and full orientation, using a non-fixed approach vector for greater flexibility but increased complexity.
  • Affordances describe the different tasks that can be achieved with an object.
  • The approach vector defines the gripper's line of approach, while the grasp center point is where the gripper frame is positioned before finger closure.
  • Hand orientation specifies how the robot hand is oriented around the approach vector at the grasp center point.
  • A wrench is a six-dimensional vector summarizing the force and torque generated by a contact force on the object.
  • Force closure means contact forces can compensate for any external wrench applied to the object.

III. DEEP LEARNING METHODS IN 6-DOF GRASPING

Deep-learning research on 6-DoF grasp synthesis has grown and is organized into four test-time methodologies: sampling, direct regression, reinforcement learning, and exemplar methods. Sampling approaches generate candidates from several spaces, evaluate them individually, and may refine them afterward.

  • Four main methodologies generate deep-learning grasp poses at test time: sampling, direct regression, reinforcement learning, and exemplar methods.The reviewed categories concern how grasps are generated.
  • Sampling: Sampling methods evaluate each grasp sample individually using information encoded in the sample.Samples can come from discrete or continuous n-dimensional spaces.
  • Sampling: Sampling pipelines first generate information, then estimate grasp quality with a deep network, and optionally refine candidates through optimization.The review uses “quality” to mean confidence of grasp success, while noting no consensus definition exists.
  • Sampling: Sampling spaces include Euclidean space, priors, configuration space, latent space, and multiple views.Euclidean-space methods commonly use heuristic rules to remove irrelevant candidates.
  • Sampling: Euclidean-space candidates may be pruned when the open hand collides with the point cloud or the closing region contains no point.
  • Sampling: Latent-space sampling trains a variational autoencoder and samples its latent space to generate grasp poses.Mousavian et al. use uniform latent-space sampling.

2) Sample Evaluation

Sample evaluation assigns each candidate a predicted grasp metric, commonly success confidence or a continuous quality score. These estimates can incorporate physical quality and workspace reachability.

  • Sampled grasps are evaluated with a function, commonly a neural network, that estimates a numerical grasp metric.The metric may be analytical grasp quality or the likelihood of successful execution.
  • Binary classification predicts whether a grasp will succeed, typically using a CNN output representing success confidence.
  • Continuous grasp metrics provide information for ranking candidates or guiding learning as a fitness score.
  • Learned metrics can represent fingertip or palm efficacy, force-closure quality, force-closure probability, or realism through discriminator loss.
  • Workspace-aware evaluation can multiply grasp-success probability by reachability probability to estimate whole-action success likelihood.The approach was extended to constrained environments such as boxes with limiting walls.

3) Optimization-based Grasp Refinement

Optimization-based refinement improves sampled or regressed grasps by optimizing learned quality, while direct regression processes the full input globally to predict one or more poses. Direct-regression systems also reduce degrees of freedom or decompose prediction into stages.

  • 3) Optimization-based Grasp Refinement: Gradient-based refinement starts from a sampled grasp and adjusts its pose using derivatives of a learned quality function.The objective is to maximize estimated grasp quality.
  • Direct Regression: Direct regression uses one network to process the entire input and predict one or multiple grasps with properties such as pose and quality.
  • Direct Regression: Direct regression can reduce computation relative to sampling by processing data globally in a single pass.
  • Direct Regression: Directly regressing a pose can be ambiguous when an input admits multiple successful grasps, so one method minimizes loss to the closest ground-truth grasp.
  • Direct Regression: Some direct-regression methods reduce the problem below 6 DoF by analytically determining selected degrees of freedom from regressed variables.Examples recover grasp depth from image information or recover position from voxel centers.
  • Direct Regression: Multi-stage regression decomposes grasp prediction into smaller tasks, often using three stages with separate losses and tasks.Stages may generate proposals, estimate remaining degrees of freedom, or refine grasps.

C. Reinforcement Learning

Deep reinforcement-learning grasping learns a policy to maximize cumulative reward over multistep tasks, with reviewed methods divided into on-policy and off-policy learning. The survey found 10 publications using these two RL types, including approaches based on demonstrations and staged control.

  • Deep RL parameterizes a policy with a neural network and seeks to maximize cumulative reward over a multistep task.
  • 10 reviewed publications employed on-policy or off-policy reinforcement learning.The table also records whether approaches learn from expert demonstrations.
  • On-Policy Learning: On-policy methods train from experiences collected by the most recent policy and were more common in the reviewed RL work.
  • On-Policy Learning: One on-policy approach divides grasping into orienting, approaching, and closing stages, training a separate RL model for each stage.Curriculum learning is employed across the stages.
  • Off-Policy Learning: Off-policy methods reuse data collected throughout training to train a new policy.Reviewed examples include Q-learning from human demonstrations and policies trained with Deep Deterministic Policy Gradient.

D. Exemplar Methods

The review distinguishes exemplar grasp transfer from supporting methods that improve grasp synthesis through shape approximation and task-specific affordances. These methods use prior examples, inferred object geometry, and affordance relevance to select or plan grasps.

  • Exemplar methods: Exemplar methods transfer grasps from previous examples by embedding geometrically similar objects nearby and searching for successful neighboring grasps.Metric learning frames grasp selection as nearest-neighbor retrieval in feature space.
  • Shape approximation: Shape completion estimates a full object model from partial input, while shape approximation also includes simpler shape models and multimodal data fusion.The review treats shape completion as the most common form of shape approximation.
  • Shape approximation: Shape-completion systems use 3D CNNs, RGB-D prediction, variational autoencoders, and depth estimation to infer hidden object geometry.These approaches operate on voxel grids, RGB-D images, or partial point clouds.
  • Shape approximation: Uncertainty-aware completion generates probability estimates or multiple completed shapes, after which grasps can be planned over representative or sampled objects.Lundell et al. use Monte-Carlo dropout, plan with GraspIt! over the mean shape, and choose the most suitable grasp across completed shapes.
  • Affordances: Affordance-based methods account for the subsequent task by selecting object regions or grasps relevant to actions such as handover.For scissors, the grasp location differs depending on whether the robot passes the blade or handle.
  • Affordances: Deep networks either segment affordance regions for analytical grasping or estimate sampled-grasp quality conditioned on an affordance label.Murali et al. visualize stable candidates colored by relevance to the specified affordance.

V. DATASET DESIGN

Dataset design strongly affects reported grasp success and reproducibility, but object-set selection and physical acquisition remain inconsistent. The reviewed literature combines household, virtual, procedural, simulated, real-robot, and human-demonstration data.

  • Object sets: Object-set choice affects reported success and reproducibility, yet reviewed works lack a standard procedure for selecting subsets from common sets.Inconsistent subsets make performance comparisons across studies more difficult.
  • Object sets: YCB is the most frequently used object set, appearing almost twice as often as the next most adopted set.YCB contains everyday household objects, including food, toys, and tools.
  • Object sets: Household object sets suit service robotics but may not adequately test robustness on complex objects.Procedurally generated sets vary object difficulty and complexity, while virtual repositories provide many object models.
  • Object sets: Most real-world object sets lack standardized physical acquisition, whereas 3D-printed datasets improve consistency but lack semantic meaning.YCB is identified as an exception to the acquisition issue.
  • Training data: Although benchmark objects are common, most reviewed networks train on purely simulated datasets rather than real-robot data.Some studies use real-robot data or combine simulation with real-world data.
  • Training data: Public resources include hybrid real-and-simulated GraspNet-1Billion, Shape Completion Grasping voxel pairs, and simulation-based ACRONYM.Other datasets collect expert demonstrations through virtual reality, handheld grippers, or kinesthetic teaching.

D. Data Representation

Deep-learning grasping commonly represents spatial input as point clouds, voxel grids, RGB-D images, or depth images. Representation choice is coupled to network backbones and to how grasp candidates or actions are encoded.

  • Representations: The four major sensor representations are point clouds, voxel grids, RGB-D images, and depth images, which are interchangeable for spatial data when camera intrinsics are known.The review compares their popularity alongside network architectures.
  • Point clouds: Point clouds are the most popular format and support multiview fusion when relative camera poses are known.PointNet and PointNet++ are common backbones for point-cloud grasping.
  • Point clouds: Point-cloud methods use direct regression, reinforcement learning, sampling, and shape completion across different grasp-synthesis pipelines.Direct regression often subsamples points, while sampling transforms or filters points relative to the gripper.
  • Backbones: PointNet-style backbones are used for direct regression, reinforcement learning, sampling, and shape completion, while VoxNet is common for voxel inputs.Voxel grids support shape completion, sampling, and direct regression.
  • Images: Image-based methods commonly use ResNet, with VGG, LeNet, DenseNet, and U-Net also appearing in the reviewed works.Images are established inputs in both computer vision and robotic grasp synthesis.
  • Images: Direct regression processes whole images, sampling methods encode gripper-relative depth or grasp features, and reinforcement learning maps camera views to actions.The input-processing strategy therefore varies with the grasping methodology.

VI. BENCHMARKING

Benchmarking spans real-world demonstrations and systematic evaluations, but training and testing often occur across a simulation-to-reality divide. Hardware and metric diversity complicate comparisons across studies.

  • Evaluation: Most reviewed methods are implemented in the real world, although some evaluate only in simulation and many real-world demonstrations lack systematic metrics.Real-world evaluation is weighted more heavily because robotic grasping ultimately targets real-world application.
  • Evaluation: Many models train solely on simulated data yet transfer directly to real-world experiments, sometimes using domain adaptation, domain randomization, or contrastive learning.These techniques target the simulation-to-reality gap or robustness to image sensor noise.
  • Hardware: Robotic arms dominate 6-DoF studies, while mobile and humanoid platforms are less common and mobile systems did not exploit their additional platform-specific degrees of freedom.Humanoid robots frequently appear in affordance-related studies.
  • Hardware: The Franka Emika Panda is the most common robotic platform, partly because its redundant DoF permits greater freedom in joint angles for a specified gripper pose.Most researchers use industrial robots or platforms designed for human-robot interaction.
  • Hardware: Two-finger parallel-jaw grippers are most common, appearing 51 times, while multi-fingered hands introduce additional high-DoF grasp-generation challenges.Examples include the Barrett, Allegro, Shadow, and Kinova three-finger grippers.
  • Metrics: Success rate is the most commonly used grasping metric, but metric definitions and post-grasp success criteria vary across studies.Other metrics include clearance, computation time, and precision.

D. Object Configurations

The review distinguishes singulated, piled-clutter, and structured-clutter scenes, while noting that grasping research remains concentrated on tabletop settings. It also identifies unresolved methodological comparisons and benchmarking limitations that constrain conclusions about practical 6-DoF grasping.

  • Object configurations: Singulated scenes contain one object, piled clutter packs objects tightly, and structured clutter spreads multiple non-touching objects across a scene.Piled clutter is common in bin-picking applications, whereas structured clutter consists of separated objects in a scene.
  • Object configurations: Singulated-object studies often use high-DoF hands for task-specific grasping, while other studies emphasize representation, learning, affordances, or manipulation.Researchers generally do not distinguish between structured and piled clutter scenes in the reviewed literature.
  • Method comparisons: No consensus exists on choosing direct regression over sampling-based methods because both address similar tasks with similar success rates.The review therefore does not establish a definitive method preference from reported success rates.
  • Method comparisons: Direct regression can run in real time, whereas sampling offers adjustable processing speed but no reviewed sampling approach achieved real-time capability.Sampling speed can be modified through the number of samples or the level of optimization.
  • Method comparisons: Exemplar methods were least common and are limited when target objects differ substantially from those represented in their knowledge databases.Their reliance on similarly shaped database objects restricts generalization to dissimilar objects.
  • Future evaluation: Current 6-DoF research rarely tests enclosed, obstructed, near-limit, or in-the-wild environments, and seldom compares 6-DoF directly with 4-DoF approaches.These settings require high-quality and diverse grasp poses to increase the probability of finding feasible trajectories.
  • Benchmarking: Few papers provide ready-to-use implementations, although publicly available packages can become widely used benchmarks for comparative evaluation.The review highlights ten Pas et al. as an example of an open-source ROS implementation used by many others.
  • Future research: The review recommends studying 6-DoF grasping in more varied environments and releasing algorithms in ready-to-use formats for informative benchmarking.The recommendations extend beyond tabletop scenarios and include formats such as ROS packages.

C. Performance Metrics

The review finds that grasping evaluation lacks consistent metrics across varied objects, robots, end-effectors, and scenarios. It recommends combining success and time-based measures, using standardized object sets, and improving dataset comparability and reproducibility.

  • The grasping community still lacks a consistent set of performance metrics, partly because studies vary in objects, robots, end-effectors, and scenarios.
  • Grasp success rate is widely used, but its definition varies from retaining an object above a table to satisfying duration or action-sequence constraints.
  • Mean Picks Per Hour measures average successful grasps completed per hour, capturing time efficacy beyond success rate.
  • Time-based metrics reflect the whole system, including robot hardware, trajectory efficiency, and compute resources, not only grasp-synthesis computation.
  • Combining success rate with time-based performance can provide a more comprehensive evaluation because faster systems may trade success for speed.
  • Standardized object sets enable head-to-head comparisons, while dataset code releases allow researchers to modify data-generation procedures.

E. Trajectory Planning

The review argues that grasping should be planned as part of broader manipulation, including trajectories, environmental contact, sensing, and hand morphology. It identifies several directions needed to move beyond isolated pick-and-place behavior.

  • Trajectory planning: Collision-free planning can be too conservative in clutter, where nudging neighboring objects or deliberate environmental contact may improve robustness.Environmental constraints can funnel uncertainty in perception and control, and this strategy has been demonstrated to work for robots.
  • Trajectory planning: The review encourages motion planning for tight spaces where targets cannot be reached without nudging objects or leveraging contact with the environment.
  • Sensor modalities: Most reviewed systems rely on vision alone, motivating multimodal sensing with touch, force, sound, robotic skins, and other modalities.Tactile sensing can help predict grasp retention, detect slip, estimate pose uncertainty, and reconstruct object geometry.
  • Grippers, hands and beyond: Simple two- or three-fingered grippers dominate the literature, while anthropomorphic, dexterous, in-hand, and soft-hand grasping remain comparatively underrepresented.The review attributes this bias partly to the lower computational complexity and affordability of simpler grippers.
  • Grasping as part of a process: Future manipulation research should extend beyond single 6-DoF grasp poses toward trajectories, online learning, failure recovery, and grasping integrated into complete tasks.The review connects these directions to tasks such as folding clothing, preparing food, and dressing humans.

IX. APPENDIX

The appendix documents a systematic literature review of deep-learning-based 6-DoF grasping. It specifies the search process, inclusion criteria, resulting corpus, and data-verification procedure.

  • Review methodology: The review searched six scholarly libraries using a methodology adapted from prior work to assess deep-learning approaches for 6-DoF robotic grasping.The libraries were IEEE, Springer, ScienceDirect, SpringerLink, arXiv, and Taylor & Francis.
  • Search strategy: Searches combined terms including grasping, point clouds, deep learning, shape completion, 6-DoF grasping, learning, and affordances.
  • Inclusion criteria: Included papers had to address tabletop grasping, use all 6 DoF for the grasp pose, apply deep learning, postdate January 1, 2012, and be written in English.
  • Review corpus: The manually screened search produced 85 papers meeting the inclusion criteria.The survey provides a table of the reviewed papers and their deep-learning approaches in its online version.
  • Data handling: Data extracted from the included papers was verified by at least two authors.
Loading 2207.02556v3…