Source-linked AI summary
6-DOF Grasping for Target-driven Object Manipulation in Clutter
Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Chris Paxton, Dieter Fox
TL;DR
The paper addresses 6-DOF grasp generation for unknown target objects in clutter, where occlusions and collision avoidance challenge planar or clutter-agnostic approaches. It uses a cascaded, learning-based framework with instance segmentation and learned collision checking, achieving 80.3% grasp accuracy and outperforming a clutter-agnostic baseline by 17.6%.
Problem
Existing data-driven methods mainly target planar grasping, while grasping unknown objects in clutter requires selecting collision-free, kinematically feasible 6-DOF grasps from partial observations.
Method
The method estimates successful grasp poses from a single-view depth image and target-object mask using cascaded object-level grasp generation followed by clutter collision checking.
Results
80.3% grasp accuracy was achieved on 23 real-world test objects in clutter, outperforming a clutter-agnostic 6-DOF GraspNet baseline by 17.6%.
Takeaways & Limitations
CollisionNet classifies collisions with 89.7% accuracy when voxel-based checking fails and avoids biases caused by voxel expansion around target objects.
Abstract
from arXiv · showhide
Grasping in cluttered environments is a fundamental but challenging robotic skill. It requires both reasoning about unseen object parts and potential collisions with the manipulator. Most existing data-driven approaches avoid this problem by limiting themselves to top-down planar grasps which is insufficient for many real-world scenarios and greatly limits possible grasps. We present a method that plans 6-DOF grasps for any desired object in a cluttered scene from partial point cloud observations. Our method achieves a grasp success of 80.3%, outperforming baseline approaches by 17.6% and clearing 9 cluttered table scenes (which contain 23 unknown objects and 51 picks in total) on a real robotic platform. By using our learned collision checking module, we can even reason about effective grasp sequences to retrieve objects that are not immediately accessible. Supplementary video can be found at https://youtu.be/w0B5S-gCsJk.
I. INTRODUCTION
The paper addresses 6-DOF grasping for unknown target objects in structured clutter, where occlusion, collisions, and limited observations make conventional approaches inadequate. It introduces a learning-based approach using instance segmentation, partial point clouds, and a learned collision checker, achieving strong real-world results and supporting retrieval of initially inaccessible objects.
- Motivation: Grasping in clutter requires collision-free, kinematically feasible grasps despite occluded geometry and limited workspace.The target must be grasped without unwanted contact with surrounding objects or the environment, while the manipulator must still reach the gripper pose.
- Research gap: 6-DOF grasp generation in structured clutter remains underexplored because many data-driven methods focus on planar grasps or bin-picking.The paper instead targets novel objects in structured clutter using a single-view point cloud and instance segmentation.
- Approach: The method uses a learned collision checker conditioned on gripper information and the raw scene point cloud.This supports collision reasoning under varying occlusion without relying solely on conventional occupancy-grid checking.
- Results: 80.3% grasp accuracy was achieved on 23 real-world test objects in clutter after training only with synthetic data.The result was obtained on a real robotic platform despite simulation-only training.
- Results: 17.6% higher performance than clutter-agnostic 6-DOF GraspNet was reported with state-of-the-art instance segmentation.The comparison concerns grasp accuracy in clutter.
- Application: The approach can move blocking objects away to grasp a target that is initially occluded and otherwise impossible to grasp.This demonstrates use beyond a single immediate pick.
II. RELATED WORK
Related work differs from this paper along four axes: isolated-object versus cluttered grasping, bin-picking versus structured clutter, planar versus spatial grasps, and model-based versus model-free planning. The paper also emphasizes target-driven selection through instance segmentation.
- Grasping in clutter vs. isolated objects: Learning-based grasping often addresses isolated objects, whereas this paper targets clusters where workspace is limited and occlusions and contact interactions are more likely.The related-work distinction frames cluttered multi-object grasping as a harder setting than isolated planar surfaces.
- Bin-picking vs. structured clutter: Structured clutter consists of packed configurations of mostly larger, heavier objects, unlike the small, light objects commonly used in bin-picking.Successful grasps are more sparsely distributed and unintended contact can be more catastrophic in structured clutter.
- Planar vs. spatial grasping in clutter: Planar grasp representations limit grasp diversity, while spatial grasping is unavoidable in structured clutter when task or arm constraints restrict access.The paper contrasts oriented rectangles or image pixels with full grasp poses.
- Model-based vs. model-free: Model-based approaches rely on full observability and prior object knowledge, whereas this method plans from raw depth images without object models or poses.Its comparison is with data-driven clutter methods that also avoid explicit object models.
- Target-agnostic vs. target-driven: The method is target-driven because instance segmentation matches generated grasps with specific objects in clutter.This distinguishes it from approaches that do not focus on selecting a desired object.
III. 6-DOF GRASP SYNTHESIS FOR OBJECTS IN CLUTTER
The framework estimates successful grasps for a segmented target object and separately evaluates their collisions with the surrounding scene. This cascaded factorization supports 6-DOF grasp generation from partial observations and enables reasoning about which blocking objects to remove.
- Problem formulation: The input is a depth-derived point cloud observation and a binary mask identifying the target object.The method represents a grasp as a parallel-yaw gripper pose in SE(3) that robustly grasps the object when the fingers close.
- Challenges: Successful grasps are complex, multimodal, discontinuous, and constrained by object geometry, physics, collisions, and the limited workspace in clutter.A visible object part may still be unreachable or ungraspable because of gripper collision constraints.
- Cascaded framework: The cascaded model factors grasp estimation into isolated-object grasp generation and CollisionNet-based collision discrimination.The isolated-object model uses the segmented target point cloud, while CollisionNet evaluates collisions between a gripper pose and the cropped scene point cloud.
- Planning beyond immediate picks: Separating target geometry from collision-free, reachable gripper poses enables reasoning about which object to remove to improve target grasp success.The paper reports this as a capability beyond simple pick operations.
- Inference pipeline: Instance segmentation selects the target, after which candidate grasps are generated while ignoring clutter and combined with CollisionNet collision results.This separates object-level grasp reasoning from environmental collision checking.
- Design evaluation: The design is evaluated against variants that do not distinguish collision failures from target-geometry failures or that use non-learned components.These comparisons examine the role of the cascaded formulation and its learned components.
B. 6-DOF Grasp Synthesis for Isolated Objects
The method generates 6-DOF grasps from segmented object point clouds using a conditional variational autoencoder, then evaluates and refines candidates to improve grasp success.
- Variational Grasp Sampling: The grasp sampler is a conditional Variational Autoencoder that predicts grasp g from object point cloud Xo and latent variable z.During inference, latent values are sampled from N(0, I), while the encoder is discarded.
- Variational Grasp Sampling: The VAE is trained with reconstruction loss on positive grasps plus a KL-divergence penalty that regularizes the latent distribution toward N(0, I).An encoder maps each point-cloud/grasp pair into latent space, and the decoder reconstructs the grasp.
- Variational Grasp Sampling: The VAE input is the target object’s point cloud segmented from the scene using an instance mask.Segmentation imperfections are modeled during simulation through boundary noise and random merging of partially occluded objects.
- Grasp Evaluation: The evaluator predicts P(S|Xo, g) to identify failed grasps that the positive-only sampler may generate.Training includes true negatives and hard negatives created by perturbing positive grasps into collisions or positions too far to grasp.
- Grasp Refinement: The system can refine sampled grasps by searching over perturbations Δg to increase predicted success probability.Metropolis-Hastings sampling provides performance similar to gradient descent while being computationally twice as fast.
C. Collision Detection for Grasps in Clutter: CollisionNet
CollisionNet adds clutter-aware collision prediction by scoring candidate grasps from full-scene and gripper point clouds, using simulation-generated labels and balanced training samples.
- CollisionNet: CollisionNet predicts a clutter-centric collision score P(C|X, g) from the full scene point cloud X and grasp pose g.This complements object-centric grasp evaluation by explicitly modeling collisions with surrounding clutter.
- Training: Simulation generates collision labels with checker Ψ under full state information, separating colliding and non-colliding reference grasps.The reference set includes positive and negative grasps, hard negatives, and free-space grasps.
- Training: Balanced sampling across grasp subsets improves training stability and test-time generalization over uniform sampling from G+ ∪ G−.Scene/object and gripper point clouds are combined with an indicator feature, and PointNet++ classifies grasps using their relative information.
D. Implementation Details
Training data is generated online from randomized multi-object scenes using stable, non-colliding object poses and grasp examples from 126 objects across several categories.
- Training Data: Online scene generation randomly arranges multiple objects at stable poses while rejection sampling avoids collisions with existing clutter.Positive and negative grasps from 126 objects, including boxes, cylinders, bowls, and bottles, are combined to generate scene grasps.
A. Ablation analysis and Discussion
The ablations evaluate grasp sampling, collision checking, and evaluator architecture using success rate, coverage, and AUC. Results favor learned, object-aware, cascaded components over geometric, voxel-based, single-stage, and instance-agnostic alternatives.
- Evaluation setup: Success rate measures collision-free grasping, while coverage measures ground-truth grasps within 2 cm of generated grasps.AUC of the success-coverage plot compares ablated methods in simulation.
- Learned vs. Surface Normal Based Grasp Sampler: The learned VAE sampler generates more grasp coverage than surface-normal sampling, especially for complex shapes with rims or handles.The surface-normal baseline performs well on simpler shapes such as boxes but fails on more complex geometry.
- CollisionNet vs. Voxel-Based Collision Checking: CollisionNet outperforms voxel-based collision checking in precision and coverage, achieving 89.7% collision-classification accuracy where the voxel method fails.Voxel errors arise from occluded or missing depth points and from expanded voxels around target-object points.
- Single-stage vs. Cascaded Evaluator: The cascaded model outperforms a single-stage evaluator because it separates grasp robustness reasoning from collision prediction.The single-stage formulation directly estimates joint success and non-collision, while the cascaded design uses separate components.
- Role of Object Instance Segmentation: The cascaded sampler reaches AUC 0.22 versus 0.02 for the instance-agnostic baseline in both success and coverage.The instance-agnostic latent space can represent grasps for multiple objects, confusing the sampler and degrading grasp quality.
B. Real Robot Experiments
Real-robot experiments evaluated the approach across nine cluttered scenes using a 7-DOF Franka Panda with a parallel-jaw gripper. The framework achieved 80.3% success and outperformed the baseline by 17.6%, while CollisionNet addressed collision-related failures.
- Experimental Setup: Nine cluttered scenes were tested on a 7-DOF Franka Panda robot with a parallel-jaw gripper.Objects were tested in a fixed, randomly pre-computed order, and success required grasping within two consecutive attempts.
- Results: 80.3% success rate was achieved, outperforming the baseline 6 DOF-GraspNet approach by 17.6%.The experiments evaluated the method across the real-robot test scenes summarized in Table I.
- Failure Analysis: CollisionNet filtered grasps that collided with objects despite receiving high scores from the object-centric evaluator.A remaining failure mode occurred because voxel-based representation could not capture all collisions.
C. Application: Removing Blocking Objects
The approach can retrieve a target that is initially inaccessible by reasoning about potential grasps and removing the objects that block them. It ranks blockers according to how strongly their removal improves collision scores.
- Grasp Generation: Potential grasps are generated for a target even when CollisionNet indicates that the target is not physically reachable.Low CollisionNet scores identify the sampled grasps as kinematically infeasible in the current clutter configuration.
- Blocking-Object Identification: The method identifies objects interfering with the target’s potential grasps.This connects inaccessible-target retrieval to collision analysis rather than requiring the target to be immediately graspable.
- Blocking-Object Removal: The blocking object is selected by the largest increase in collision scores when the corresponding object is removed.This ranking determines which clutter object should be moved first.
V. CONCLUSION
The paper presents a learning-based framework for 6-DOF grasp synthesis of novel objects in structured clutter, achieving 80.3% accuracy on a real robotic platform despite simulation-only training. It identifies motion planning as a key boundary and points toward trajectory and task-planning extensions.
- Conclusion: The framework performs 6-DOF grasp synthesis for novel objects in structured clutter.The conclusion emphasizes occlusions and collision avoidance as central challenges in this setting.
- Conclusion: 80.3% grasp accuracy was achieved on novel cluttered objects using a real robotic platform despite training only in simulation.This is the principal reported real-world outcome in the conclusion.
- Limitations and Future Work: The method considers gripper pre-shape collisions by design, so motion planning can still fail on generated grasps.Future work includes trajectory generation in grasp planning and applications to task planning and challenging environments.