PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Driven Grasp Synthesis - A Survey

Jeannette Bohg, Antonio Morales, Tamim Asfour, Danica Kragic

arXiv:1309.2660v2cs.RO

TL;DR

Grasp synthesis must select task-suitable configurations from infinitely many candidates, while analytic models face assumptions and a gap between closure metrics and real-world stability. This survey organizes data-driven methods by prior object knowledge, relating representations and perceptual processes to grasp sampling and ranking, and identifies open problems including segmentation, task dependence, lifelong learning, and benchmarking.

  • Problem

    Selecting a suitable grasp from infinitely many candidates remains challenging, while analytic models rely on assumptions that can diverge from real-world grasp behavior.

  • Method

    The survey classifies data-driven approaches by known, familiar, or unknown objects and examines their object representations, perceptual processing, sampling, and ranking methods.

  • Results

    The survey identifies distinct representation and perception requirements across prior-knowledge groups and contrasts empirical data-driven verification with analytic guarantees.

  • Takeaways & Limitations

    The classification highlights open problems in segmentation, task-dependent grasping, lifelong learning, and comprehensive benchmarking.

  • Takeaways & Limitations

    The survey notes that many approaches assume objects are segmented from the background, often using a planar-surface assumption, and that task constraints remain insufficiently studied.

Abstract

from arXiv · show

We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups based on whether they synthesize grasps for known, familiar or unknown objects. This structure allows us to identify common object representations and perceptual processes that facilitate the employed data-driven grasp synthesis technique. In the case of known objects, we concentrate on the approaches that are based on object recognition and pose estimation. In the case of familiar objects, the techniques use some form of a similarity matching to a set of previously encountered objects. Finally for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps. Our survey provides an overview of the different methodologies and discusses open problems in the area of robot grasping. We also draw a parallel to the classical approaches that rely on analytic formulations.

I. INTRODUCTION

Grasp synthesis seeks task-suitable configurations from infinitely many candidates, with analytic and data-driven methods forming the main methodological divide. Analytic approaches optimize geometric, kinematic, or dynamic criteria but face modeling and robustness limitations.

  • Grasp synthesis finds a configuration satisfying task-relevant criteria among infinitely many possible candidates.
  • Analytic methods formulate grasp synthesis as constrained optimization over dexterity, equilibrium, stability, or dynamic-behavior criteria.They commonly use grasp maps that transform contact forces into object wrenches.
  • Data-driven methods sample and rank candidates using prior grasp experience, heuristics, simulation, real-robot trials, demonstrations, perception, or semantics.Candidate grasps are commonly parameterized by grasping point, approach vector, wrist orientation, and initial finger configuration.
  • Analytic grasp analysis relies on simplified contact, friction, and rigid-body assumptions, creating ambiguities especially in grasp-dynamics analysis.Accurate and tractable contact-compliance models remain particularly relevant for statically indeterminate grasps.
  • Robotic sensing and modeling errors make fingertip placement inaccurate, motivating grasps robust to positioning errors through independent contact regions and caging formulations.Caging configurations can make opening or closing the fingers produce an equilibrium grasp without precise finger positioning.
  • Analytic approaches also commonly assume precise object geometry and physical properties, although interaction-based estimation can relax some assumptions while remaining limited in scope.The cited interaction-based approach predicts occluded-object motion, while related methods remain simulation-only or concern 2D objects.

B. Development of Data-Driven Methods

Data-driven grasp synthesis grew alongside simulation, learning, and 3D sensing, while exposing a gap between analytic metrics and real-world grasp success. The survey organizes methods around object representation, perception, candidate sampling, and empirical ranking.

  • Data-driven grasp synthesis became popular after GraspIt! enabled extensive simulation-based development and evaluation.These approaches differ in how they sample candidates from the infinite grasp space.
  • Simulation offers controlled, efficient experimentation but may not transfer reliably to real-world grasping.Studies found classic metrics poor predictors of real-world success, with metric-based grasps often fragile and underperforming human-transferred grasps.
  • Grasp closure guarantees equilibrium existence, not stability under perturbation, revealing a gap between tractable grasp models and reality.
  • Experience-based learning avoids the learned-model transfer problem but makes example collection time-consuming and leaves object representation and generalization as key questions.
  • Machine-learning methods can predict good grasping points from monocular images using synthetic labeled data, including household manipulation demonstrations.
  • Data-driven methods emphasize object representation and perceptual processing, then retrieve, sample, and rank grasps through comparison with prior experience.They can use recognition, classification, pose estimation, feature extraction, or similarity metrics.
  • Data-driven methods lack analytic guarantees for dexterity, equilibrium, stability, and dynamic behavior, so these properties must be verified empirically.They nevertheless support studying grasp dynamics and developing analytic models that better resemble reality.

D. Classification of Data-Driven Approaches

The survey classifies data-driven grasp synthesis by the query object’s prior familiarity: known, familiar, or unknown. This organization links each category to its required object representation and perceptual processing.

  • Known Objects: Known-object methods retrieve previously generated grasps after recognizing the object and estimating its pose.
  • Familiar Objects: Familiar-object methods transfer grasp experience from similar objects using representations and similarity metrics based on visual or functional characteristics.
  • Unknown Objects: Unknown-object methods generate and rank grasps from sensory structure or features without object models or prior grasp experience.
  • The classification ties prior object knowledge to recognition, pose estimation, similarity comparison, or direct feature-to-grasp mapping.
  • Most reviewed approaches use scene sensor data and belong to real robotic systems capable of executing grasps.
  • Candidate generation and ranking depend on object-grasp representations, sensor modality, learning source, and heuristics.

II. GRASPING KNOWN OBJECTS

Known-object systems commonly build grasp databases offline from 3D object models, then recognize and localize objects online before retrieving feasible grasps. Candidate generation varies from primitive-based approximation to surface-based sampling, while evaluations expose limits of classical metrics.

  • Known-object grasp execution reduces to estimating object pose and filtering database hypotheses by reachability.
  • Offline pipelines generate and rank grasp candidates for each object model, storing the resulting hypotheses for online retrieval.
  • 3D Mesh Models and Contact-Level Grasping: Many methods assume a 3D mesh and use force closure with the ϵ-metric, differing mainly in how they sample candidate configurations.
  • 3D Mesh Models and Contact-Level Grasping: Shape approximations with primitives or superquadrics prune candidate searches, while other methods randomly sample using surface geometry, normals, or bounding boxes.
  • Force-closure and ϵ-metric ranking can select fragile grasps, motivating robustness filters that retain grasps whose perturbed neighbors remain successful.
  • Human-planned task grasps outperformed stable grasps selected by classical metrics, with humans optimizing a skewness metric relating hand and object-axis alignment.

2) Learning from Humans:

Human-demonstration systems associate recognized objects with observed grasp configurations, while trial-and-error systems refine or generate object-specific grasps through execution feedback. These approaches use imitation, density modeling, mapping schemes, or reinforcement learning.

  • Learning from Humans: Human-demonstration pipelines recognize the object and extract or recognize the demonstrated grasp before storing them together.
  • Learning from Humans: Eigengrasps reduce robotic-hand configuration space before searching for stable grasps.
  • Learning from Humans: Human examples can define object-specific empirical grasp densities from which hypotheses are sampled.
  • Learning from Humans: Imitation learning can bootstrap reaching policies, while reinforcement learning and reactive controllers refine grasp behavior using rewards.
  • Learning through Trial and Error: Trial-and-error systems retrieve past experiences, execute selected or newly generated grasps, evaluate performance, and update memory.
  • Learning from Humans: Kinesthetic teaching, data gloves, and visual grasp recognition provide distinct ways to transfer human grasps to robots.

B. Online Object Pose Estimation

Online execution of known-object grasps requires recognition, pose estimation, and reachability filtering. Surveyed systems use probabilistic, monocular, deformable-shape, and combined 2D/3D representations, while familiar-object methods seek grasp-relevant similarity.

  • Online Object Pose Estimation: Known-object systems recognize and estimate pose online before retrieving offline grasps and discarding unreachable hypotheses.
  • Online Object Pose Estimation: Probabilistic object representations support grasp selection by density sampling or policies refined from human demonstrations.
  • Online Object Pose Estimation: Deformable known objects can be detected in cluttered monocular scenes using learned probabilistic 2D shape models, including partial occlusions.
  • Online Object Pose Estimation: Combined 2D and 3D features enable single-image pose estimation, with accuracy demonstrated through successful grasps.
  • Familiar Objects: Familiar-object grasping depends on representations and similarity metrics that capture shared visual or functional affordances.
  • Familiar Objects: Discriminative familiar-object methods differ in object features, candidate parameterizations, and whether they predict graspable parts, contacts, or full configurations.

1) Based on 3D Data:

3D-based methods represent objects with geometric structure, then sample and rank grasps using learned models or heuristics. They can exploit detailed shape information but often assume accurate 3D models or segmentation.

  • 3D representations: Superquadrics represent object parts or entire objects, enabling prehensile-part classification or efficient search over grasp configurations.El-Khoury and Sahbani classify segmented superquadric parts with an ANN, while Pelossof et al. optimize approach vector, wrist orientation, and finger spread using an SVM.
  • Learning and ranking: Training data may combine object geometry, candidate grasp configurations, and scalar quality labels to rank candidates.Pelossof et al. train an SVM on superquadric parameters paired with grasp configurations and quality estimates.
  • Limitations: These methods commonly rely on accurate and detailed 3D object models, an assumption that may fail with noisy or incomplete real-world sensory data.The survey identifies uncertainty about whether superquadrics preserve object shape adequately when observations are incomplete.
  • Learning and ranking: Learned models use geometric features to classify graspable locations or predict grasp quality from point clouds and object-part structure.Reviewed examples include box constellations, Markov Random Fields over point clouds, and height-accumulated features for SVM classification.

2) Based on 2D Data:

2D-based methods learn grasp locations from image appearance, local or global shape, and interaction-derived affordance cues. Because 2D evidence underconstrains full grasp configuration, later methods combine it with 3D information.

  • 2D grasp prediction: Image-based classifiers infer good grasping points from labeled synthetic images or robot trial outcomes.Examples use logistic regression on image features and Beta-distribution posteriors estimated from grasping trials.
  • 2D grasp prediction: Affordance cues encode prehensile object parts from human interactions and support matching grasp hypotheses to new visual stimuli.Stark et al. store learned cues with relative object position and scale.
  • Limitations: Inferring a complete grasp configuration from 2D data alone is highly under-constrained because similar image regions can permit different grasps.The survey motivates integrating complementary local and global 2D and 3D modalities.
  • 2D representations: Global shape contexts and local visual features provide alternative representations for selecting grasp candidates.Bohg and Kragic encode object shape relative to a potential grasp point using shape contexts.
  • Integrating 2D and 3D Data: Combining 2D and 3D methods produces a sparser set of grasp hypotheses with good quality and supports full grasp-configuration search.Bergström et al. use 2D selection to search a sparse edge-based 3D representation.
  • Integrating 2D and 3D Data: Local 2D and 3D descriptors can detect deformable shirt collars in scenes containing one or several shirts.The evaluation measures collar-detection reliability under both scene conditions.

B. Grasp Synthesis by Comparison

Comparison-based grasp synthesis retrieves grasps by matching novel objects or parts to previously stored exemplars. The reviewed systems use geometric, physical, semantic, or learned low-dimensional similarity representations.

  • Comparison-based synthesis: Exemplar-based methods synthesize grasps by finding similar database objects or object parts associated with successful grasps.This differs from approaches that learn discriminative functions over graspability features.
  • Shape matching: Human grasp examples can be retrieved by matching an object’s shape features to the interior shapes of available hand postures.Li and Pollard formulate suitable-grasp retrieval as shape matching between hand and object.
  • Object-type matching: Knowledge bases can represent object types with distributions over shape and physical features, together with representative pre-grasps.A new object is matched to the most similar object type before grasp synthesis.
  • Semantic grasp planning: Semantic affordance maps link object features to task-specific approach vectors and grasp features for actions such as pouring or drilling.Retrieved approach directions seed grasp synthesis, while grasp features guide similarity to the synthesized grasp.
  • Functional-grasp transfer: Functional-grasp transfer aligns and warps same-category objects before transferring fingertip contacts, but the reported experiment covers only six cup instances.The survey describes the results as promising while explicitly limiting their scope.
  • Limitations: Several comparison-based approaches require 3D object meshes, leaving transfer from partial sensory data unresolved.The survey suggests that estimating full shape from partial observations could introduce noisy and uncertain meshes.
  • Prototype matching: A lower-dimensional space clusters similarly shaped object parts and grasp configurations into prototypical grasp-inducing parts.These prototypes can be matched to real sensor data to synthesize grasps on novel objects.

2) Sensor-based Exemplars:

Sensor-based exemplar methods connect real observations with grasp experience acquired through trial and error or demonstrations. They retrieve, rank, and execute grasps using 2D, 3D, and part-based representations, including task-specific generative models.

  • Sensor-based exemplars: Sensor-based systems build knowledge bases by linking real sensor representations of objects to grasps executed on robotic platforms.The survey presents this workflow as a way to generalize prior grasp experience to novel objects without relying exclusively on object meshes.
  • Sensor-based exemplars: Kamon et al. learn a function mapping object- and grasp-candidate-dependent quality parameters to a grasp grade.Their object representation includes 2D silhouette, center of mass, and main axis, while grasps use two parameters for fingertip computation.
  • Trial-and-error learning: Morales et al. rank planar hypotheses from 2D contours with KNN using a real-world trial-and-error experience database.The method incorporates three-fingered hand kinematics and visual feedback when inferring successful configurations.
  • Limitations: Camera-derived height, 3D position, and pose may be collected but excluded from final grasp inference and selection.This limitation is explicitly reported for one sensor-based approach.
  • Part-based matching: Some sensor-based systems match local object templates from demonstrations and update similarity using both positive and negative grasp examples.Herzog et al. match height-map templates on convex-hull facets; failed matches become negative examples.
  • Part-based matching: The template approach is evaluated across many objects and different robots, while another system models action-relevant parts with weighted 3D point distributions.Kroemer et al. demonstrate generalization for a pouring task by locating the sub-part most likely to afford the action.
  • Generative models: Generative approaches model joint distributions over grasp variables, tasks, object categories, and constraints using Bayesian networks.Song et al. learn structure from large sets of simulated, quality-annotated grasps and discretize continuous variables for efficient structure learning.

D. Category-based Grasp Synthesis

Category-based approaches use object recognition, categorization, or similarity to previously encountered objects to reuse or infer grasp hypotheses. For unknown objects, methods instead extract sensory structure and apply shape-based, feature-based, or heuristic strategies without relying on an available object model.

  • Known-object methods recognize an object and reuse associated grasp hypotheses when a matching model is available.
  • Object category and task information can be combined in a Bayesian network to infer a full hand configuration satisfying task constraints.
  • Pure 3D object descriptors support recognition, categorization, and pose estimation, but their suitability for generalizing grasps across categories remains unshown.
  • Unknown-object approaches do not assume an object model or prior grasp experience, instead linking sensory structure directly to candidate grasps.
  • Unknown-object methods address sparse, incomplete, and noisy sensing through full-shape approximation, low-level feature heuristics, or global-shape reasoning.
  • Shape-based methods estimate object geometry using primitives, symmetry, shape carving, or multi-view reconstruction before generating grasps.

B. From Low-Level Features to Grasp Hypotheses

Low-level visual features can be mapped to predefined grasp postures, then filtered and ranked using heuristics or feature-based criteria. These approaches operate on contours, local surfaces, point-cloud geometry, depth patterns, or deformable-object borders.

  • Local multimodal contour descriptors are associated with elementary grasping actions, and heuristics reduce the resulting hypotheses.
  • Local surfaces and their interrelations generate and filter two- and three-fingered grasp hypotheses evaluated in mixed real and simulated environments.
  • Segmented point-cloud shape supports top, side, and high-point grasp generation, followed by ranking with weighted geometric features.
  • Depth images can be sampled for patterns resembling the gripper interior, producing and ranking grasp hypotheses without a complete 3D object model.
  • For towels, border detection from depth discontinuities and fitted corners supplies grasping points for variable, unpredictably arranged deformable objects.

C. From Global Shape to Grasp Hypothesis

Global-shape approaches infer grasps from silhouettes, point clouds, footprints, probabilistic shape models, or uncertain object reconstructions. Some methods combine multiple planners or evaluate grasp robustness across perturbed shape and mass estimates.

  • Silhouettes and segmented point clouds support grasp computation using hand kinematics or geometric heuristics such as centering a grasp relative to the object.
  • A 3D Gaussian object model is used to optimize a grasp that minimizes palm distance while maximizing fingertip distance from the object.
  • Object footprints projected onto a supporting surface provide eigenvectors for generating grasp hypotheses.
  • For zero-slip push grasps, perturbed object shapes and center-of-mass positions are sampled so one candidate can be evaluated under uncertainty.
  • Multiple grasp planners can be integrated to reach consensus on how to grasp a segmented point cloud.
  • Open Problems: The survey identifies object segmentation, lifelong grasp learning, autonomous manipulation planning, and robust execution as open problems.

A. Final Notes

The survey categorizes data-driven grasp synthesis by prior knowledge and object assumptions, identifies field trends and challenges, and highlights the lack of general comparison benchmarks. It also maintains an online, structured reference resource for the surveyed work.

  • The survey categorizes published data-driven grasping work by the type and level of prior knowledge and assumptions about manipulated objects.
  • It identifies recent trends and discusses remaining challenges in data-driven grasp synthesis.
  • General benchmarks and performance metrics suitable for comparing approaches are lacking, and existing datasets are not commonly used for comparison.
  • A continuously updated website organizes the survey’s references according to the proposed classification and tagged aspects.
Loading 1309.2660v2…