Source-linked AI summary
S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes
Yuzhe Qin, Rui Chen, Hao Zhu, Meng Song, Jing Xu, Hao Su
TL;DR
The paper addresses 6-DoF grasping of unknown objects in clutter from a single partial depth view. S4G learns single-shot amodal grasp proposals from synthetic scenes using a gripper contact model, and experiments report improved speed and success rate over state-of-the-art methods.
Problem
Generic objects in clutter remain difficult to grasp from a single partial depth view, especially without category assumptions.
Method
S4G directly regresses per-point 6-DoF grasp poses and quality from scene point clouds, trained on synthetic multi-object scenes with contact-model-based annotations.
Results
Improved speed and success rate over state-of-the-art methods were reported in real-world scenarios, following synthetic and real-environment experiments.
Takeaways & Limitations
Single-shot grasp proposals and the gripper contact model are reported as effective design choices for this cluttered single-view 6-DoF setting.
Takeaways & Limitations
The study assumes a fixed single viewpoint, a commodity depth camera, casually scattered household objects, and a parallel gripper.
Abstract
from arXiv · showhide
Grasping is among the most fundamental and long-lasting problems in robotics study. This paper studies the problem of 6-DoF(degree of freedom) grasping by a parallel gripper in a cluttered scene captured using a commodity depth sensor from a single viewpoint. We address the problem in a learning-based framework. At the high level, we rely on a single-shot grasp proposal network, trained with synthetic data and tested in real-world scenarios. Our single-shot neural network architecture can predict amodal grasp proposal efficiently and effectively. Our training data synthesis pipeline can generate scenes of complex object configuration and leverage an innovative gripper contact model to create dense and high-quality grasp annotations. Experiments in synthetic and real environments have demonstrated that the proposed approach can outperform state-of-the-arts by a large margin.
1 Introduction
S4G targets generic 6-DoF grasping for unknown objects in clutter from a single partial depth view. It uses single-shot regression and synthetic scene-level training to predict amodal grasps efficiently.
- Generic grasping remains open for unknown objects in complex scenes, beyond classical methods designed for known geometry.
- 6-DoF grasping addresses interaction limits of vertically constrained 3/4-DoF approaches, including buried objects and horizontally placed plates.
- The target setting uses casually scattered household objects, a commodity depth camera, one fixed viewpoint, partial point clouds, and a parallel gripper.
- Single-view clutter and unknown categories require noise-resistant, amodal grasp detection that infers viable grasps from incomplete observations.
- S4G directly regresses 6-DoF grasps from the entire scene point cloud in one pass using per-point scoring and pose regression.
- Its training pipeline synthesizes multi-object scenes and uses a gripper contact model considering contact-area shape and size to produce grasp proposals.
2 Related work
Related work spans deep-learning grasp proposals, geometric 6-DoF sampling, synthetic grasp-data generation, and point-cloud architectures. S4G builds on these directions for dense-clutter, generic-object grasping.
- Deep learning has become effective for robotic grasping, while 6-DoF proposal methods address generic objects without requiring exact object models.
- Geometric sampling methods guide 6-DoF hypotheses with local surface frames but may miss thin structures when partial noisy observations make normals difficult to compute.
- Manual annotation of 6-DoF grasp poses is impractical, motivating analytic synthesis for large-scale labeled training data.
- The generated dataset is described as the first large-scale synthetic 6-DoF grasp dataset for dense clutter, combining object-level grasps with scene-level rejection of infeasible grasps.
- S4G uses PointNet++ as its single-shot grasp-detection backbone because PointNets are robust to varying point-cloud sampling densities.
3 Problem Setting
The problem is represented using a single-view point cloud and a parallel-gripper description. A grasp configuration combines a gripper pose in SE(3) with a scalar quality score.
- The single-view point cloud is denoted P, and the parallel gripper is denoted G.
- A grasp configuration is c = (h, s_h), where h ∈SE(3) is the gripper pose and s_h ∈R measures grasp quality.
4 Training Data Generation
S4G synthesizes physically plausible, cluttered training scenes and dense grasp annotations by combining a gripper contact model with simulated sensing, collision analysis, and pose-robustness evaluation.
- Scene synthesis: The pipeline uses YCB objects to generate cluttered scenes with complete and partially observed point clouds for single-view grasp detection.MuJoCo and V-HACD simulate objects settling under gravity, friction, and contact forces before reconstructing the 3D scene.
- Gripper contact model: The gripper contact model identifies feasible grasps from antipodal contact pairs using surface normals, contact geometry, and closing-region occupancy.Normals are smoothed for rugged surfaces, while the closing-region point count measures object volume inside the gripper.
- Gripper contact model: The model uses physically meaningful gripper parameters, including 3mm allowable pad deformation and a 23mm normal-smoothing radius.These settings are tied to the soft rubber pad and gripper width rather than extensive parameter tuning.
- Gripper contact model: The contact model is reported to handle rugged and flat surfaces better than Darboux-frame methods, which can fail on thin walls of plates and mugs.The comparison specifically concerns generating successful grasp poses for these geometries.
- Grasp prediction: The network uses PointNet++ features and predicts one 6-DoF grasp pose and quality score for every original scene point.Hierarchical features are propagated through inverse-distance interpolation and skip links before per-point prediction.
- Sensor simulation: Viewed point clouds are produced by applying depth-sensor noise, with observed distance modeled as ˜D_o,p = (1 + N(0, σ^2))D_o,p and σ = 0.003.This creates noisy observations for training while preserving the simulated scene structure.
- Robustness and labeling: Synthetic grasp labels incorporate collision masks and perturbed-pose evaluation to score collision-free grasps robust to pose uncertainty.The final viewed point cloud, ground-truth grasps, and scores form the S4G training data.
5 Single-Shot Grasp Generation
S4G uses a PointNet++-based single-shot network to assign each scene point a 6-DoF grasp and quality score, then selects executable grasps through suppression and quality-weighted sampling.
- PointNet++ based grasp proposal: The network takes a single-view point cloud and assigns each point a viable grasp and its quality score.The architecture predicts these attributes per point using inverse indexing.
- PointNet++ based grasp proposal: Hierarchical features combine local and global geometry to infer grasp relationships from partial, noisy observations.The design extracts local features while incorporating scene-level geometric context.
- 6-DoF pose regression: S4G directly regresses 6-DoF grasp poses using a continuous 6D rotation representation instead of discontinuous quaternions or Euler angles.The representation encodes two 3D vectors that are mapped to a rotation matrix.
- Training objectives: The loss combines rotation, translation, and grasp-score objectives, supervising pose predictions only at points assigned viable grasps.The experiment sets λrot, λt, and λs to 5.0, 20.0, and 1.0.
- Grasp selection: Non-maximum suppression keeps locally highest-scoring predictions, followed by weighted random sampling to choose one grasp for execution.This converts dense per-point predictions into an executable grasp set.
6 Experiments
Experiments evaluate S4G in simulation and on a real robot under clutter, partial sensing, and unseen objects, with results emphasizing the value of unrestricted 6-DoF proposals and direct regression.
- Simulation experiments: 63.38% of objects are grasped using nearly vertical grasps between 0° and 15°, while SE(3) advantages increase with scene complexity.The simulation study covers 6k scenes and approximately 2.6M generated grasps across three density levels.
- Simulation experiments: Simulation evaluates recall rate across grasp approach angles and scene densities, using simple, semi-dense, and dense clutter conditions.Recall is the percentage of objects grasped with approaches between vertical and a specified angle.
- Baseline comparison: The baseline comparison uses GPD with 3 and 12 channels and PointNetGPD, ranking their sampled candidates alongside S4G predictions.Baseline training generates collision-free grasp candidates across more than 6.5k scenes.
- Robotic experiments: The robotic evaluation uses one KinectV2 sensor and 30 objects absent from training, arranged into cluttered scenes with repeated grasp attempts.Each method is tested over four repetitions, with up to 15 attempts per scene.
- Robotic results: S4G outperforms baselines in success rate, completion rate, and time efficiency in dense-clutter robotic experiments.Baseline methods spend around 20 seconds detecting collisions and extracting local geometry for each sampled grasp.
- Experimental conditions: The experimental setting is harder than the baseline papers because S4G uses one depth sensor, while baseline GPD used two sensors for nearly complete point clouds.Heuristic sampling also performs poorly on non-convex or thin objects under noisy point-cloud normals.
7 Conclusion
The paper studies single-view 6-DoF grasping in clutter with a commodity depth sensor and reports that synthetic training transfers to real-world operation. Its single-shot proposal and gripper contact model are presented as effective design choices.
- Conclusion: The approach targets 6-DoF parallel-gripper grasping in cluttered scenes captured from a single viewpoint with a commodity depth sensor.The problem setting is explicitly framed around real-world operation.
- Conclusion: Training on synthetic scenes supports real-world grasping with improved speed and success rate compared with state-of-the-art methods.The conclusion reports this outcome without specifying a numerical margin.
A.1 Network Details
The network uses three PointNet++ set-abstraction layers built from multilayer perceptrons and farthest point sampling to improve point-cloud coverage.
- Network architecture: S4G uses three point-set abstraction layers with three-layer MLPs containing (128, 128, 256), (256, 256, 512), and (512, 512, 1024) units.ReLU is used as the activation function in these layers.
- Network architecture: Farthest Point Sampling selects points for more uniform coverage of the input point set than random sampling.Each selected point is chosen as the most distant point from the current set.
A.2 Robotics Experiments Dataset
The robotics experiments use a dataset of 30 daily objects, distinct from the YCB dataset used to generate training data.
- The experimental dataset consists of daily objects.
- The robotics-experiment dataset differs from the YCB dataset used to generate training data.
- 30 objects are used in the robotics experiments.
A.3 Robotics Experiments Grasp Proposal
The robotics experiments present viewed point clouds alongside proposed high-quality grasp sets. Figures 8 and 9 document these experimental outputs, while Figures 2 and 3 provide additional examples.
- Figures 8 and 9 show viewed point clouds from the depth sensor and proposed high-quality grasp sets in robotic experiments.
- The displayed experimental examples pair depth-sensor point-cloud views with proposed grasp sets.
- Figures 2 and 3 provide additional viewed point clouds and proposed high-quality grasp sets from robotic experiments.