Source-linked AI summary
Learning Continuous 3D Reconstructions for Geometrically Aware Grasping
Mark Van der Merwe, Qingkai Lu, Balakumar Sundaralingam, Martin Matak, Tucker Hermans
TL;DR
Grasping previously unseen objects from partial views remains challenging, and existing approaches often lack explicit geometric reasoning about the scene. This paper combines learned 3D reconstruction with grasp-success prediction and collision constraints; in 96 robot trials, Reconstruction-Grasping succeeded less often than Partial-View-Grasping.
Problem
Existing grasping approaches often lack explicit modeling of full scene geometry, despite partial views and unknown object properties complicating grasp synthesis.
Source: §I · p. 1Method
The approach learns PointSDF reconstructions and a geometrically aware grasp-success predictor, then uses the reconstruction to constrain grasp optimization against collisions.
Source: §I · p. 2, §IV · p. 3Results
Reconstruction-Grasping succeeded on 48% of its grasp attempts (22 grasps), versus 60% (29 grasps) for Partial-View-Grasping.
Source: C. Grasp Synthesis · p. 6Takeaways & Limitations
The approach exhibits qualitative geometric reasoning, while the paper leaves open when object geometry matters for grasp outcomes.
Source: §VI · p. 6The authors report that optimization difficulty hurts the approach's quantitative gains.
Source: §VI · p. 6
Abstract
from arXiv · showhide
Deep learning has enabled remarkable improvements in grasp synthesis for previously unseen objects from partial object views. However, existing approaches lack the ability to explicitly reason about the full 3D geometry of the object when selecting a grasp, relying on indirect geometric reasoning derived when learning grasp success networks. This abandons explicit geometric reasoning, such as avoiding undesired robot object collisions. We propose to utilize a novel, learned 3D reconstruction to enable geometric awareness in a grasping system. We leverage the structure of the reconstruction network to learn a grasp success classifier which serves as the objective function for a continuous grasp optimization. We additionally explicitly constrain the optimization to avoid undesired contact, directly using the reconstruction. We examine the role of geometry in grasping both in the training of grasp metrics and through 96 robot grasping trials. Our results can be found on https://sites.google.com/view/reconstruction-grasp/.
I. INTRODUCTION
Reliable grasping of previously unseen objects from partial views remains challenging. The paper proposes combining learned 3D reconstruction with grasp-success prediction and collision-constrained grasp synthesis.
I. INTRODUCTION
- Grasping unseen objects under noisy, partial sensing remains an open robotics challenge; this paper focuses on isolated tabletop objects and dexterous hands.Unknown object properties compound the difficulty of grasp synthesis.Source: §I · p. 1
- End-to-end grasp learning often lacks explicit scene geometry, so optimization may produce robot-object collisions despite learned grasp-success scores [5] [8].The paper seeks to reduce such collisions by explicitly modeling the 3D environment in grasp planning.Source: §I · p. 1
- PointSDF learns a signed distance function for partially viewed objects, providing a reconstruction that the grasping system uses for geometric reasoning.The paper presents this reconstruction as both implicitly and explicitly useful for geometrically aware grasp synthesis.Source: §I · p. 2
- The approach combines a grasp-success predictor with constrained optimization in the full robot configuration space, enforcing kinematic feasibility and collision avoidance.The paper also introduces a grasp-success architecture with geometrically aware point-cloud encodings.Source: §I · p. 2
II. 3D RECONSTRUCTION VIA LEARNED SIGNED DISTANCE FUNCTION
PointSDF reconstructs an object from a single-view point cloud by predicting its signed distance function. This continuous representation also provides geometric gradients and can be evaluated without first producing a mesh.
II. 3D RECONSTRUCTION VIA LEARNED SIGNED DISTANCE FUNCTION
- PointSDF predicts an object's signed distance function from a single-view point cloud using a single-pass encoder-decoder architecture.The object surface is represented by the function's zero level set; values are negative inside and positive outside.Source: §II · p. 2
- The network takes a point cloud and query point as input and returns the signed distance to the reconstructed surface.Its point-cloud embedding and query point feed fully connected layers that produce the SDF estimate.Source: §II · p. 2, Figure 2 · p. 3
- SDF predictions can be evaluated at arbitrary resolution without mesh conversion, and gradients with respect to query points provide geometric directions.A mesh can still be extracted from the zero isosurface when desired.Source: §II · p. 2
III. LEARNING GRASP SUCCESS PREDICTION VIA 3D RECONSTRUCTION
The paper models grasp planning as probabilistic inference over configurations given a point-cloud observation. Its grasp-success predictor uses the PointSDF point-cloud encoder to estimate success probability.
III. LEARNING GRASP SUCCESS PREDICTION VIA 3D RECONSTRUCTION
- Grasp planning maximizes the posterior probability that an arm-hand configuration will yield a successful grasp given the observed point cloud.The formulation represents the hand joints and the palm pose derived from arm joints.Source: §III · p. 3
- The posterior combines a learned grasp-success probability with a grasp prior modeled as a two-component Gaussian mixture fitted to training configurations.The two distributions are parameterized separately.Source: §III · p. 3
- The grasp-success network reuses PointSDF's point-cloud encoder and combines its embedding with the grasp configuration to predict success probability.The network produces its estimate through a sigmoid output.Source: §III · p. 3
IV. RECONSTRUCTION-AWARE GRASP SYNTHESIS VIA CONSTRAINED OPTIMIZATION
Grasp synthesis is formulated as a constrained optimization over the full arm-hand joint configuration. Learned SDF values enforce collision avoidance for the partially observed target object, alongside constraints for known environment geometry.
IV. RECONSTRUCTION-AWARE GRASP SYNTHESIS VIA CONSTRAINED OPTIMIZATION
- The optimizer searches for a preshape arm-hand configuration that maximizes predicted grasp success while avoiding collisions.The formulation optimizes over the full robot joint configuration.Source: §IV · p. 3
- Collision constraints combine an analytical SDF for known environment geometry with PointSDF values for the unknown target object.The target-object constraint uses the learned reconstruction directly, without requiring a mesh.Source: §IV · p. 3
- For target-object collision checks, the method queries each robot-link mesh vertex and uses the minimum SDF value per link; SDF gradients can move links out of collision.The robot-link query points are obtained from forward kinematics.Source: §IV · pp. 3–4
- Optimizing over the full arm-hand configuration enforces joint feasibility and directly constrains unintended robot-object contact.The optimization also includes the robot's joint limits.Source: §IV · pp. 3–4
V. EXPERIMENTS
The experiments assess PointSDF reconstruction, grasp-success prediction, and real-robot grasping. They focus on single-depth-view reconstruction, the effect of implicit geometric reasoning on prediction, and performance in robot trials.
V. EXPERIMENTS
- The experiments ask how well PointSDF reconstructs objects from a single depth view.This is the first of the three stated evaluation questions.Source: §V · p. 4
- The experiments assess grasp-success prediction and how implicit geometric reasoning affects its performance.The second question concerns the grasp metric and the effect of implicit geometry.Source: §V · p. 4
- The experiments test the grasp formulation through deployment on a real robot.The abstract reports 96 robot grasping trials, but this section passage does not specify that number.Source: §V · p. 4
A. PointSDF Reconstruction
PointSDF is evaluated against a voxel-based reconstruction baseline on reconstruction metrics and qualitative shape results. The paper reports better IoU and Chamfer-L1, near-matching normal consistency, and smoother reconstructions with finer details.
A. PointSDF Reconstruction
- PointSDF training uses synthetic views of 590 Grasp Database meshes and 76 YCB Database meshes, rendered at 200 random orientations each with depth noise.The rendered point clouds are paired with signed-distance labels sampled from the correspondingly rotated ground-truth meshes.Source: A. PointSDF Reconstruction · p. 4
- Point clouds use a centered object frame with camera-frame orientation and are scaled into a 1m×1m×1m bounding box.Random rotations during data collection make PointSDF camera-pose invariant; SDF estimates can be scaled back at inference.Source: A. PointSDF Reconstruction · p. 4
- The evaluation compares PointSDF with a reimplemented voxel-based 3D CNN encoder-decoder, VoxelCNN, using IoU, Chamfer-L1 distance, and normal consistency.The metrics are computed between each reconstructed test object and its ground-truth mesh.Source: A. PointSDF Reconstruction · p. 4
- PointSDF outperforms VoxelCNN on IoU and Chamfer-L1 and nearly matches its normal consistency.The qualitative comparisons also report smoother PointSDF reconstructions that retain finer details; slices show a clear zero level set and gradient.Source: Table I · p. 4, A. PointSDF Reconstruction · p. 4, Figure 3 · p. 4
B. Grasp Success Network
On simulated grasp data, point-cloud embedding networks outperform voxel-based networks, while fixing the reconstruction-trained PointSDF embedding slightly reduces classifier performance. The evaluation compares scratch-trained and fixed embeddings using a four-fingered Allegro hand and Kuka arm setup.
B. Grasp Success Network
- Training and experiments use a four-fingered, 16-DOF Allegro hand mounted on a 7-DOF Kuka LBR4 arm.The grasp configuration includes the palm pose and eight joint values for the two finger joints nearest the palm.Source: B. Grasp Success Network · p. 4
- The grasp network is trained on 7,290 simulated examples and tested on a held-out set of 1,821 grasps.The data were collected in Gazebo with the DART physics engine, using a geometry-based planner and sampled finger-joint angles.Source: B. Grasp Success Network · pp. 4–5
- PointSDF-Scratch trains its embedding on the grasp dataset, whereas PointSDF-Fixed first trains on reconstruction and then freezes the embedding while fitting the grasp metric.The voxel-based comparison uses corresponding VoxelCNN-Scratch and VoxelCNN-Fixed variants.Source: B. Grasp Success Network · p. 5
- Both point-cloud embedding networks outperform voxel-based networks on grasp-success F1, while fixing the PointSDF embedding slightly lowers classifier performance.For VoxelCNN the effect is reversed; the authors suggest its CNN structure appears to encourage overfitting.Source: B. Grasp Success Network · p. 5, Figure 4 · p. 5
C. Grasp Synthesis
The section evaluates reconstruction-aware grasp synthesis on a physical robot, comparing it with a partial-view baseline. Across the reported trials, the reconstruction approach achieved a lower success rate, while qualitative examples show it avoiding some collisions with occluded geometry.
C. Grasp Synthesis
- Reconstruction-Grasping uses PointSDF for robot-object collision constraints and PointSDF-Fixed for its grasp-success objective, while Partial-View-Grasping uses partial geometry.The partial-view comparison replaces the learned collision constraint with an analytical one.Source: C. Grasp Synthesis · p. 5
- Across 96 robot grasps, the evaluation used a Kuka LBR4 arm with an Allegro hand and eight objects, with only the chips object included in training.The camera was tested in high and low positions; the lower position yielded heavier point-cloud occlusion.Source: C. Grasp Synthesis · pp. 5–6
- 48% of Reconstruction-Grasping attempts succeeded, versus 60% for Partial-View-Grasping; average planning times were 37.81s and 36.55s, respectively.Fig. 7 reports success rates for high and low camera setups, with the low setup producing more occlusion.Source: C. Grasp Synthesis · p. 6, Figure 7 · p. 6
- Reconstruction-Grasping struggled with pitcher, juice-box, and max-gel, and its optimization failed to find a heuristic-sufficient solution 11 times among 26 failures, versus 4 for the baseline.The authors attribute part of the limited quantitative improvement to the difficulty of satisfying full-geometry constraints while meeting grasp-success cutoffs.Source: C. Grasp Synthesis · p. 6
- Qualitative examples show Partial-View-Grasping planning grasps in collision with occluded space, which Reconstruction-Grasping can avoid using its full-geometry representation.The paper notes that only one Partial-View-Grasping failure was clearly due to planning in contact with the object.Source: C. Grasp Synthesis · p. 6, Figure 8 · p. 6
VI. CONCLUSION AND FUTURE WORK
The paper combines a learned signed-distance reconstruction with grasp-success prediction and collision constraints for geometry-aware grasp synthesis. It reports qualitative geometric reasoning, while optimization difficulty limits quantitative gains; future work targets optimization efficacy, richer sensing, and in-hand manipulation.
VI. CONCLUSION AND FUTURE WORK
- The approach combines a shared-embedding signed distance function with grasp-success prediction and collision constraints, but optimization difficulty limits its quantitative gains.Future directions include improving constrained optimization, incorporating multiple viewpoints and tactile sensing, and using grasp-success models in in-hand manipulation optimization.Source: §VI · p. 6