PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations

Xinchen Yan, Jasmine Hsu, Mohi Khansari, Yunfei Bai, Arkanath Pathak, Abhinav Gupta, James Davidson, Honglak Lee

arXiv:1708.07303v4cs.ROcs.AIcs.CVcs.LG

TL;DR

Grasp learning must handle geometry, physical properties, and variation in illumination, object location, and viewpoint. The paper reconstructs 3D scene geometry from RGBD input and uses that representation to predict grasp outcomes; it reports improved outcome prediction over a CNN baseline and generalization to novel viewpoints and object instances.

  • Problem

    Grasping requires geometric and physical-property understanding and invariance to illumination, object location, and viewpoint, while data-driven methods use hundreds of thousands of examples.

    Source: §I · p. 1

  • Method

    The network reconstructs a 3D occupancy representation from RGBD input, hallucinates a gripper-perspective surface view, and uses the representation to predict grasp outcomes.

    Source: §I · p. 1

  • Results

    The authors report improved grasp-outcome prediction over the CNN baseline and better generalization to novel viewpoints and object instances.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    The results support using a geometry-aware representation for grasp-outcome prediction and analysis-by-synthesis grasp planning in the reported setting.

    Source: §V · p. 7

Abstract

from arXiv · show

This paper focuses on the problem of learning 6-DOF grasping with a parallel jaw gripper in simulation. We propose the notion of a geometry-aware representation in grasping based on the assumption that knowledge of 3D geometry is at the heart of interaction. Our key idea is constraining and regularizing grasping interaction learning through 3D geometry prediction. Specifically, we formulate the learning of deep geometry-aware grasping model in two steps: First, we learn to build mental geometry-aware representation by reconstructing the scene (i.e., 3D occupancy grid) from RGBD input via generative 3D shape modeling. Second, we learn to predict grasping outcome with its internal geometry-aware representation. The learned outcome prediction model is used to sequentially propose grasping solutions via analysis-by-synthesis optimization. Our contributions are fourfold: (1) To best of our knowledge, we are presenting for the first time a method to learn a 6-DOF grasping net from RGBD input; (2) We build a grasping dataset from demonstrations in virtual reality with rich sensory and interaction annotations. This dataset includes 101 everyday objects spread across 7 categories, additionally, we propose a data augmentation strategy for effective learning; (3) We demonstrate that the learned geometry-aware representation leads to about 10 percent relative performance improvement over the baseline CNN on grasping objects from our dataset. (4) We further demonstrate that the model generalizes to novel viewpoints and object instances.

I. INTRODUCTION

Grasping combines geometric understanding with physical-property estimation and invariance to illumination, location, and viewpoint, while data-driven methods use hundreds of thousands of examples. This work reconstructs scene geometry from RGBD input and reuses it to predict grasp outcomes, alongside a VR demonstration dataset and reported gains in prediction and generalization.

I. INTRODUCTION

  • Grasping is challenging because it requires geometric and physical-property understanding alongside invariance to illumination, object location, and viewpoint.Current data-driven approaches use hundreds of thousands of examples.Source: §I · p. 1
  • The geometry-aware network reconstructs 3D scene shape from RGBD input, hallucinates a gripper-perspective local surface view, and reuses the representation to predict grasp outcomes.It combines a shape-generation network with a grasp-outcome prediction network and a learning-free physical projection layer.Source: §I · p. 1
  • 101 everyday objects and around 150K grasping demonstrations form a VR dataset augmented with synthetic gripper-pose perturbations.The demonstrations include pre-grasp object and gripper poses and grasp outcomes; each object has 10–20 human attempts.Source: §I · p. 2
  • The paper presents a method to learn a 6-DOF grasping network from RGBD input.The authors describe it as the first such method to their knowledge.Source: §I · p. 2
  • The authors report better shape and grasp-outcome learning than models without geometry, plus improved generalization to novel viewpoints and object instances.They also report advantages in guiding grasping exploration.Source: §I · p. 2

II. RELATED WORK

Prior grasping work includes image-based grasp detection, geometric and analytic modeling, and top-down point-cloud methods. This paper distinguishes its approach by learning a 6D grasping network from RGBD input and linking 2D observations to 3D shape without explicit 3D volume supervision.

II. RELATED WORK

  • Image-based grasp detection ranges from planar visual-feature models to deep pipelines and large-scale trial-and-error learning.Prior approaches include two-step object and grasp-part detection and a CNN trained on 700 hours of robotic grasping data.Source: §II · p. 2
  • Other work models grasp geometry through known-shape analytic methods, occupancy-grid shape completion, learned shape context, and part-based object models.The cited shape-completion approach uses ground-truth 3D occupancy grids during training; the paper says its own approach does not require full 3D volume supervision.Source: §II · p. 2
  • DexNet work uses large-scale point-cloud databases for top-down planar grasping, while this paper targets learned 6D grasping interaction.The related work also includes semantic or task-specific grasping.Source: §II · p. 2
  • The approach combines generative 3D shape modeling with predictive 6D grasping from RGBD input and a learning-free projection layer, without explicit 3D volume supervision.The projection layer links 2D observations with 3D object shape.Source: §II · p. 2

III. MULTI-OBJECTIVE FRAMEWORK WITH GEOMETRY-AWARE REPRESENTATION

The framework represents scene geometry as an occupancy grid that preserves object shape, location, and orientation while remaining invariant to camera viewpoint and distance. It learns this representation from RGBD through depth and silhouette supervision, using an in-network projection operator.

III. MULTI-OBJECTIVE FRAMEWORK WITH GEOMETRY-AWARE REPRESENTATION

  • The framework jointly develops 3D shape generation and grasp-outcome prediction, with reconstruction from RGBD serving as the geometry-aware representation.Prior 3D-supervised methods would require more data in this setting because shape, location, and orientation are entangled.Source: §III · p. 2, A. Learning generative geometry-aware… · p. 2

A. Learning generative geometry-aware representation from RGBD input

  • The geometry-aware representation is a 3D occupancy grid encoding object shape, location, and orientation, invariant to camera viewpoint and distance.Earlier normalized grids centered at the origin, while this formulation accounts for location and orientation.Source: A. Learning generative geometry-aware… · p. 2

B. Depth supervision with in-network projection layer

  • Depth supervision trains 3D reconstruction by projecting the shape into a 2D depth map with the camera transformation matrix; adding a silhouette objective makes learning stable and efficient.The projection layer implements exact ray tracing without extra free parameters.Source: B. Depth supervision with in-network… · p. 3
  • The projection maps volume samples into normalized device coordinates, then flattens the 3D spatial output across one dimension.The output volume is related to the input volume through the transformation matrix P.Source: B. Depth supervision with in-network… · p. 3

C. Viewpoint-invariant geometry-aware representation with multi-view supervision

Because single-view RGBD reconstruction has shape ambiguity, the method uses shape consistency across multiple viewpoints to train a viewpoint-invariant representation. At test time, reconstruction uses only one viewpoint.

C. Viewpoint-invariant geometry-aware representation with multi-view supervision

  • Shape consistency learning addresses shape ambiguity by enforcing viewpoint invariance across multi-view observations.The decoder receives averaged identity units from multiple viewpoints, and multiple projections supervise reconstruction.Source: C. Viewpoint-invariant geometry-aware… · p. 4
  • During training, observations from multiple views share a representation and supervise a common 3D reconstruction.The reconstruction maps a series of observations to V, with each viewpoint's depth and camera transformation matrix used for projection.Source: C. Viewpoint-invariant geometry-aware… · p. 4
  • Testing requires RGBD input from only a single viewpoint, despite the multi-view training procedure.Source: C. Viewpoint-invariant geometry-aware… · p. 4

D. Learning predictive grasping interaction with geometry-aware representation.

The model predicts grasp outcome from the current RGBD observation and a proposed 6D gripper pose, using reconstructed geometry as an explicit input. It also projects a local surface view from the gripper's perspective.

D. Learning predictive grasping interaction with geometry-aware representation.

  • The model predicts binary grasp outcome from the current RGBD state and the gripper's 6D pre-grasping position and orientation.The paper frames the outcome as a future state of the interaction.Source: D. Learning predictive grasping… · p. 4
  • Unlike the baseline interaction model, the geometry-aware model constrains prediction with explicit geometry information.The baseline maps the observation and action to outcome; cited approaches used millions of random grasps, an additional eye- or hand-view, or extra processing.Source: D. Learning predictive grasping… · p. 4
  • The gripper is treated as a virtual camera to infer the local surface view from the reconstructed scene using its 6D pose.The model combines current observation, proposed action, and inferred geometry to predict outcome.Source: D. Learning predictive grasping… · p. 4

E. DGGN: Deep geometry-aware grasping network.

DGGN combines a shape generation network with a grasp-outcome prediction network. The first reconstructs geometry from RGBD; the second uses scene and gripper state together with global and local geometry features.

E. DGGN: Deep geometry-aware grasping network.

  • DGGN comprises a shape generation network and an outcome prediction network.Each network uses a 2D convolutional encoder; the shape decoder is 3D deconvolutional, and the outcome predictor is fully connected.Source: E. DGGN: Deep geometry-aware grasping… · p. 4
  • The shape network encodes RGBD and camera-view matrices, decodes a 32 × 32 × 32 voxel grid, and projects it into 128 × 128 silhouettes and depth maps.Its generative pre-training learns viewpoint-invariant units through object segmentation and depth prediction.Source: E. DGGN: Deep geometry-aware grasping… · p. 4
  • The outcome network combines pre-grasp RGBD, gripper action, and global and local geometry features to predict grasp outcome.Its local projection transforms the surface around the gripper fingers into a 48 × 48 silhouette and depth map.Source: E. DGGN: Deep geometry-aware grasping… · p. 4

IV. EXPERIMENTS

The experiments introduce a VR grasping dataset and describe how human demonstrations are expanded into synthetic grasps. Data collection also varies camera viewpoints, with some elevation angles reserved for evaluation.

A. Dataset collection

  • The dataset contains human grasp demonstrations on 101 everyday objects across seven categories, split into training, validation, and testing sets.Five right-handed users contributed 1,597 human grasps; the object split was 70% training, 10% validation, and 20% testing.Source: A. Dataset collection · p. 4
  • Figure 3 illustrates the VR-Grasping-101 dataset.Source: Figure 3 · p. 5
  • Perturbing human demonstrations with PyBullet expanded the dataset to 150K grasping demonstrations.The synthetic grasps include successful and unsuccessful trials.Source: A. Dataset collection · p. 5
  • Camera views varied across eight azimuths and four elevations, with only 15- and 45-degree elevations used for training.The remaining elevation angles were reserved for evaluation.Source: A. Dataset collection · p. 5

B. Implementation details

The visualization reports detailed occupancy-grid generation from single-view RGBD input, including reasonable generalization to novel objects and local geometry estimates from a gripper viewpoint. The implementation uses a two-stage training procedure for the geometry-aware model and a CNN baseline.

B. Implementation details

  • The CNN baseline removes the shape encoder and decoder, and adding view and static-scene inputs did not significantly improve it.Source: B. Implementation details · p. 5
  • The geometry-aware model is trained in two stages: shape generation pretraining followed by fine-tuning the state encoder and outcome predictor.The first stage uses multi-view supervision; grasp outcome prediction is trained as binary classification.Source: B. Implementation details · p. 5
  • Both models use convolutional encoder-decoder architectures with residual connections, trained using 20 GPU workers and 32 parameter servers.Source: B. Implementation details · p. 5

C. Visualization: 3D shape generation

D. Model evaluation: Grasping outcome prediction

The geometry-aware model consistently outperforms the CNN baseline in grasp outcome classification and is less affected by novel elevation angles, particularly for teapots and plates. In planning on novel objects, CEM-guided geometry-aware grasping also achieves higher success rates than the baseline.

D. Model evaluation: Grasping outcome prediction

  • Outcome classification is evaluated on 30K demonstrations from novel object instances across diverse viewpoints, using average accuracy over perturbed grasps.The evaluation repeats across four elevation angles; random guessing achieves 50% accuracy.Source: D. Model evaluation: Grasping outcome… · pp. 5–6
  • The geometry-aware model consistently outperforms the CNN baseline in grasp outcome classification and is less affected by novel elevations, especially for teapots and plates.Teapots have irregular parts and plates are fairly flat; these categories are described as challenging for outcome prediction.Source: D. Model evaluation: Grasping outcome… · p. 6, Table I · p. 6, Table II · p. 6

E. Application: Analysis-by-synthesis grasping planning.

  • Planning starts from a failure grasp, samples ten candidate directions per step, and uses the network score to select a direction for up to 20 steps.The reported success rate averages eight repetitions per testing demonstration and counts failure when no success occurs within 20 steps.Source: E. Application: Analysis-by-synthesis… · pp. 6–7
  • CEM-guided planning with the geometry-aware model achieves consistently better success rates than the baseline on novel objects, with the largest improvement for bottles.Improvement is less significant for bowls, partly due to difficulty predicting concave shapes in novel instances.Source: E. Application: Analysis-by-synthesis… · p. 7, Table III · p. 7

V. CONCLUSIONS AND FUTURE WORK

The geometry-aware network combines shape generation, grasp-outcome prediction, and physical projection; it improves outcome prediction over a CNN baseline and supports better analysis-by-synthesis planning. Future work considers other hand types and model designs aimed at adaptation to domains such as real-robot setups.

V. CONCLUSIONS AND FUTURE WORK

  • The proposed network performs shape generation and grasp-outcome prediction with a learning-free physical projection layer.Source: §V · p. 7
  • Generative shape modeling improved grasp-outcome prediction over the CNN baseline, while geometry-guided analysis-by-synthesis produced better grasp planning.Source: §V · p. 7
  • Future work proposes applying the representation to hands with different kinematics and exploring designs without an auxiliary state encoder for adaptation to domains such as real-robot setups.Source: §V · p. 7

APPENDIX: DETAILS OF SYNTHETIC DATA GENERATION

Synthetic grasps are generated by alternating demonstration-log playback with simulation-based pose perturbation and grasp testing, then recording the outcomes.

APPENDIX: DETAILS OF SYNTHETIC DATA GENERATION

Loading 1708.07303v4…