Source-linked AI summary
Shape Completion Enabled Robotic Grasping
Jacob Varley, Chad DeChant, Adam Richardson, Joaquín Ruales, Peter Allen
TL;DR
The paper addresses difficult grasp planning from occluded, incomplete sensory geometry. It trains a 3D CNN on large paired occupancy-grid data to complete single-view pointclouds, then evaluates completion quality and grasping utility. The method produces rapid runtime completions, improves over partial and mirrored approaches across reported metrics, and improves generalization to unseen objects as training-model count increases.
Problem
Occlusion and incomplete scene geometry make grasp planning from raw sensory data difficult, while existing approaches may not extend to novel objects.
Method
A 3D CNN learns from paired partial and complete occupancy grids to complete single-view pointclouds, producing fast or detailed meshes for planning and grasping.
Results
Across all metrics, the method produces more accurate completions than other general completion approaches, and more training models improve performance on unseen objects.
Takeaways & Limitations
Completed shapes support fast path planning, scene understanding, and grasp planning, including for objects not seen during training.
Abstract
from arXiv · showhide
This work provides an architecture to enable robotic grasp planning via shape completion. Shape completion is accomplished through the use of a 3D convolutional neural network (CNN). The network is trained on our own new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. At runtime, a 2.5D pointcloud captured from a single point of view is fed into the CNN, which fills in the occluded regions of the scene, allowing grasps to be planned and executed on the completed object. Runtime shape completion is very rapid because most of the computational costs of shape completion are borne during offline training. We explore how the quality of completions vary based on several factors. These include whether or not the object being completed existed in the training data and how many object models were used to train the network. We also look at the ability of the network to generalize to novel objects allowing the system to complete previously unseen objects at runtime. Finally, experimentation is done both in simulation and on actual robotic hardware to explore the relationship between completion quality and the utility of the completed mesh model for grasping.
I. INTRODUCTION
The work uses a 3D CNN to complete occluded object geometry from a single-view pointcloud, enabling grasp planning and other robotics tasks. Its two-stage framework shifts most computation offline and supports rapid runtime completion.
- Single-view pointcloud completion addresses the occlusion and incomplete scene geometry that make raw-sensor grasp planning difficult.
- The framework trains on paired partial and ground-truth occupancy grids, then completes mesh models at runtime from one viewpoint.
- Under a tenth of a second on average is required to complete an object at runtime before quickly meshing it.
- Completed outputs support fast path planning and scene understanding or detailed grasp-planning meshes that integrate higher-resolution visible regions.
- The system completes each object separately, supports crowded scenes, and generalizes across domains, unseen views, and entirely novel objects.
II. RELATED WORK
Prior robotics methods often rely on geometric heuristics or recognition of known objects, whereas this approach learns completion from exemplars and extends to novel objects. The work also addresses robotics-specific data, depth, and resolution requirements.
- Symmetry and extrusion heuristics work well for objects represented by geometric primitives but do not require arbitrary-object learning.
- Object-recognition and pose-detection approaches assume a small, known database of objects, while this approach extends to novel objects.
- Computer-vision shape-completion methods include deep belief networks, Gibbs sampling, Random Forests, exemplar approaches, occupancy learning from images, and database retrieval.
- Direct application to robotic manipulation was limited by the lack of a large handheld-object rendering dataset, RGB-only inputs, and resolutions too low for contemporary grasp planning.
- Unlike complete-mesh database-query methods, this framework can complete a partial observation before using the completed model as a grasp-query mesh.
- The dataset draws on YCB and Grasp Database models because they combine widely available physical objects with meshes geared toward robotic manipulation.
III. TRAINING
Training data combines mesh models from robotics datasets with many synthetically rendered viewpoints. These data are converted into paired low-resolution partial and complete occupancy grids for CNN training.
- Meshes from the YCB and Grasp Database were voxelized with binvox to generate 256^3 occupancy grids.
- Each object produced 726 depth images from uniformly sampled roll-pitch-yaw rotations around the mesh.
- Rendered depth images formed occupancy grids for camera-visible mesh portions, while binvox grids represented the complete meshes.
- Both occupancy-grid sets were transformed to align and down-sampled to 40^3, creating paired training examples.
B. Model Architecture and Training
The CNN predicts a complete occupancy grid from a partial grid using convolutional and dense layers, sigmoid-bounded outputs, cross-entropy training, and Jaccard evaluation across increasingly difficult test conditions.
- Model Architecture: ReLU activations are used throughout except the final fully connected layer, whose sigmoid restricts voxel outputs to [0, 1].
- Training: Cross-entropy training encourages each output toward 0 for unoccupied target voxels or 1 for occupied voxels.
- Evaluation: The Jaccard similarity compares generated and ground-truth occupancy grids, ranging from 0 for no intersection to 1 for identical sets.
- Evaluation: Networks are evaluated on training views, novel views of training objects, and objects absent from the training data.
C. Training Results
Training evaluates how completion quality changes with training-set size and generalization, while the runtime pipeline converts a sensed partial view into a completed mesh for grasp planning.
- Training-set size: Three networks were trained with 14, 94, or 486 mesh models to study how training-set size affects reconstruction quality.The networks used identical architectures and were trained until learning plateaued; the 486-model network took over a week.
- Training-set size: Fewer training models produced better completion results on training views.
- Generalization: All networks performed approximately the same on novel views of mesh objects included in training.
- Generalization: More training models improved completion quality on previously unseen objects by broadening the variety of learned inputs.
- Runtime pipeline: At runtime, the system captures, segments, scales, and voxelizes a pointcloud before passing it through the CNN to produce a completed voxel grid.
- Runtime pipeline: The completed grid supports either fast marching-cubes meshing or detailed meshing that integrates the dense observed pointcloud before grasp planning.The detailed path upsamples and merges the CNN output with observed points, fills gaps, applies CUDA optimization, and then meshes the result.
V. EXPERIMENTAL RESULTS
The evaluation uses sampled training views, novel views of training objects, and views of previously unseen models, comparing completion methods with three geometric metrics.
- Test dataset: The test dataset contains 50 training views, 50 holdout views, and 50 views of holdout models.Training and holdout views came from 14 YCB training objects; holdout-model views came from holdout YCB and Grasp Dataset objects.
- Evaluation metrics: Completion accuracy is compared using Jaccard similarity, Hausdorff distance, and geodesic divergence.
A. General Completion Results
The proposed completion method is compared with partial and mirroring baselines using voxelized mesh accuracy and geometric-distance measures across tested views.
- Compared methods: The comparison includes partial completion, mirroring completion, and the proposed CNN method trained on 484 YCB and Grasp Dataset objects.
- Jaccard similarity: The completed meshes are voxelized at 80^3 and compared with the ground-truth mesh using Jaccard similarity.Table I reports Jaccard similarity, for which larger values are better.
- Jaccard similarity: The proposed method achieves higher similarity to ground-truth meshes than partial and mirroring approaches for all tested views.
- Hausdorff distance: CNN completions have significantly lower symmetric Hausdorff distance to the ground truth than partial and mirrored completions.Table II evaluates mean distance in millimeters, where smaller values are better.
- Geodesic divergence: The proposed method outperforms the other completion methods on geodesic divergence.
- Overall comparison: Across all metrics, the proposed method produces more accurate completions than the other general completion approaches.
B. Comparison to Database Driven Methods
The method is compared with RANSAC-based database completion and evaluated on known and previously unseen objects, including geometric fidelity metrics and grasp-planning relevance.
- Comparison scope: RANSAC is evaluated in a highly constrained setting with a small number of known objects and cannot load the 484 objects used by the proposed method.The paper identifies limited scalability to large object databases as a motivation for its approach.
- Grasp-planning relevance: Planned and executed grasps differ for partial and mirrored completions, whereas the proposed method shows fidelity between planned and executed grasps.The comparison is illustrated using planned grasps on completions and execution on the ground-truth object.
- Known and novel objects: Our method performs comparably to a 14-object RANSAC approach on objects in its database while training on far more objects and generalizing to novel objects.Known-object metrics are Jaccard 0.771 versus 0.8566, Hausdorff 3.6 versus 3.1, and geodesic 0.0867 versus 0.1245.
- Known and novel objects: On holdout models, our method significantly outperforms RANSAC on Jaccard, Hausdorff, and geodesic metrics.The reported values are Jaccard 0.6496 versus 0.4063, Hausdorff 5.9 versus 20.4, and geodesic 0.1412 versus 0.4305.
C. Simulation Based Grasp Comparison
Simulation evaluates whether grasps planned on completed objects transfer to ground-truth objects, using more than 24,000 grasp trials and planned-versus-realized error measures.
- Simulation protocol: More than 24,000 grasps were planned in simulation across completions using uniformly sampled approach directions.The planned grasps were executed on ground-truth meshes rather than on the completed objects.
- Simulation results: Joint Err. measures mean planned-versus-realized joint differences in degrees, while Pose Err. measures mean grasp-pose differences in millimeters; smaller is better for both.These metrics quantify transfer fidelity from completed-object planning to ground-truth execution.
- Simulation protocol: The simulation replaced each completion with its ground-truth object before moving and closing the hand according to the planned grasp.This procedure approximated real-world grasp execution while preserving the planned grasp configuration.
- Simulation results: Using our method placed the end effector closer to its intended location in both joint space and palm Cartesian position.Table IV reports differences between planned and realized joint states and end-effector pose.
D. Performance on Real Hardware
Real-hardware experiments evaluate end-to-end grasping and crowded-scene completion, showing improved grasp success and fast selective completion of target and non-target objects.
- Hardware grasping: On 15 objects, our method improved grasp success rate by 20% over general shape-completion methods and produced lower joint error.The hardware system ran once per object for each completion method.
- Hardware setup: The experiments used a Barrett Hand, Staubli TX60 arm, and 15 experiment objects.The hardware and objects are shown in Fig. 7.
- Crowded-scene completion: Completing nearby obstacles allows the planner to flag a grasp as unreachable when a partial planning scene would permit a collision.The crowded-scene example shows a failed grasp caused by a poor planning scene and corrected after CNN completion.
- Completion timing: Average completion times were 0.119 seconds for segmentation, 2.136 seconds for target completion, and 0.142 seconds for non-target completion.These averages were measured over 15 runs.
- Crowded-scene completion: The system selectively generates detailed completions for manipulated objects while quickly filling occluded regions for non-target objects.Non-target objects can be meshed directly from CNN output without post-processing, whereas grasp targets receive more detailed processing.
VI. CONCLUSION AND FUTURE WORK
The paper concludes that CNN-based completion supports fast mesh generation and grasp planning with improved grasp outcomes, while identifying several directions for future work.
- Conclusion: The framework completes and meshes single-view objects before planning grasps, with fast completions and several-second post-processed meshes suitable for grasp planning.The dataset and code are open source for use by other researchers.
- Conclusion: Grasps planned from the proposed completions are more often successful and execute closer to the intended hand configuration than grasps from other methods.The conclusion links completion quality to both grasp success and planned-versus-executed hand configuration.
- Future work: Future work includes GAN-based training, migration to a larger dataset such as ShapeNet, and retrieving grasps for similar completed objects.The retrieval direction references the Columbia Grasp Database and DexNet.