PaperlayerView exact revision on arXiv

Source-linked AI summary

SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

Yang Xu, Gurpreet Singh Mukker, Raymond Wang, Jasper Gerigk, Maria Attarian, Igor Gilitschenski

arXiv:2607.20207v1cs.RO

TL;DR

Grasping in cluttered scenes must combine language-specified task requirements with precise 3D contact geometry across different robot embodiments. SeededGrasp addresses this by conditioning lightweight grasp generation on a VLM-predicted seed point, outperforming baselines in simulation and achieving strong real-world performance.

  • Problem

    Language-guided grasping lacks a unified approach that connects high-level intent to low-level contact geometry across cluttered scenes and diverse embodiments.

  • Method

    SeededGrasp uses an off-the-shelf VLM to predict a task-specific seed point, then conditions a lightweight flow-matching model to generate grasps for multiple grippers.

  • Results

    SeededGrasp outperforms all baselines in simulation and achieves strong grasping performance in real-world tests.

  • Takeaways & Limitations

    Seed point conditioning provides an effective bridge between language grounding and grasp generation in cluttered, multi-embodiment scenes.

  • Takeaways & Limitations

    The model predicts grasp poses without awareness of robot-arm kinematics, so it may propose infeasible actions during real-world deployment.

Abstract

from arXiv · show

Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/

1 Introduction

SeededGrasp addresses language-grounded grasping in cluttered scenes by using a VLM to select a task-specific seed point that conditions lightweight, multi-embodiment grasp generation. The work also introduces a large cluttered-scene grasping dataset spanning three grippers.

  • Dexterous grasping requires precise, stable, task-aware contact while respecting surrounding objects in cluttered, multi-object scenes.
  • Supporting multiple embodiments can improve grasping performance and data efficiency, but simple grippers and highly dexterous hands have complementary limitations.Two-finger grippers are limited to pinch grasps, whereas high-dexterity grippers are difficult to control.
  • Existing language-grounded approaches either simplify scenes, require large annotated datasets and expensive VLM-inclusive end-to-end training, or rely on complex pose optimization.
  • SeededGrasp uses a VLM to select a task-specific point from the scene point cloud, isolating the target and relevant part while preserving grasping flexibility.The seed point indicates where the grasp should be centered and conditions subsequent grasp generation.
  • 2.56M grasps span 610 cluttered scenes and three grippers in the new multi-embodiment grasping dataset.

2 Related Work

Related work spans analytic and simulation-based grasp synthesis, learning from datasets, cluttered-scene grasping, multi-embodiment generalization, and language-grounded grasping. Existing approaches advance these areas but leave gaps in semantic conditioning, cluttered multi-embodiment data, and precise grasp generation.

  • Analytic and simulation-based grasp synthesis: Analytic and optimization-based methods formulate grasp generation around stability criteria and improve dexterous grasp synthesis speed and quality.However, they remain relatively unwieldy at inference and generally do not handle cluttered scenes or language intent, while supporting valid-grasp dataset generation.
  • Datasets: Existing datasets support parallel-jaw, dexterous, cluttered-scene, or multi-embodiment grasping, but none provide multi-embodiment data in cluttered scenes.The authors build their dataset upon MultiGripperGrasp (MGG), which filters generated grasps.
  • Grasping in cluttered scenes: Learning-based cluttered-scene methods predict grasp distributions from scene observations while addressing occlusion and interactions with surrounding objects.Approaches commonly use point clouds or depth images, with dexterous systems employing generative models and contact-aware representations.
  • Multi-embodiment grasping: Multi-embodiment methods condition grasp prediction on robot geometry or kinematic structure instead of training separate policies for each end-effector.Other strategies use hand-agnostic contact maps or keypoints before solving for specific gripper configurations.
  • Language-grounded grasping: Language-grounded methods use VLMs to infer task-relevant regions, affordances, or object parts, then propose or finetune grasps through heuristics.These plug-and-play approaches often produce imprecise results, while other works combine VLM-based semantic grounding with larger grasping systems.

3 Dataset Generation

SeededGrasp builds a multi-embodiment cluttered-scene grasp dataset by adapting filtered MGG mid-air grasps into valid in-scene grasps. The resulting dataset contains 2.56 million grasp poses across three grippers, with distribution skewed toward Franka Panda.

  • Dataset construction: The dataset adapts MultiGripperGrasp’s synthetic mid-air poses into valid grasps for cluttered scenes and multiple grippers.This avoids generating raw grasp poses from scratch while providing training data for the seed point conditioned model.
  • Dataset construction: Three grippers were selected after re-filtering MGG: Franka Panda, Robotiq 3-Finger, and Allegro.The selection prioritized high-quality data volume and a mix of parallel-jaw and dexterous grasping kinematics.
  • Scene generation: Cluttered scenes use 284 training objects and 50 unseen evaluation objects from Google Scanned Objects and YCB, with one to ten randomly dropped objects per scene.The objects include simple shapes and irregular geometries, producing both sparse and dense clutter regions.
  • Grasp validation: Heuristic collision checks determine grasp viability after transforming each object’s MGG grasps into the scene’s global frame.These checks replace computationally prohibitive dynamic in-scene lift tests for millions of grasps.
  • Dataset statistics: 2.56 million grasp poses comprise the training and evaluation dataset across the three robot grippers.The distribution is skewed toward Franka Panda because its simpler kinematics allow more poses to pass filtering.

4 Model

SeededGrasp separates grasping into VLM-based seed-point prediction and conditioned grasp-pose generation. The model encodes robot and scene geometry, represents variable-DOF poses, and uses flow matching for multi-modal grasp generation.

  • Two-Stage Pipeline: The pipeline first predicts a seed point and then predicts a grasp pose conditioned on that seed point.An off-the-shelf VLM uses a BEV scene image and user instructions instead of a separately trained seed-point model.
  • Point Cloud Encoding: Point-cloud representations combine Fourier-encoded coordinates with local normals, curvature, seed-point distance, and graph-convolution features.Graph Convolutional Networks use sixteen nearest neighbors, while max and mean pooling provide global representations.
  • Grasp Pose Parameterization: Each grasp pose is parameterized by translation, rotation, and gripper joint configuration, with padding for variable embodiments up to sixteen Allegro joints.Rotation is represented with Euler angles, and the pose is defined in the world frame.
  • Grasp Pose Generation: Flow matching models the inherently complex and multi-modal distribution of stable grasp poses conditioned on robot and scene features.At inference, a fixed canonical grasp pose adjacent to the seed point is denoised using the chosen seed point, point clouds, and target robot query.
  • VLM Seed Point Prediction: The VLM zero-shot predicts a directive-aligned grasp location, which is projected into the 3D scene point cloud for grasp generation.The method requires neither fine-tuning nor few-shot examples; Gemini models perform best in the reported ablation.

5 Experimental Results

SeededGrasp is evaluated across seed-conditioned grasp prediction, multi-embodiment performance, language-conditioned grasping, and ablations. It outperforms relevant baselines, benefits from dual gripper representations and larger datasets, and shows sensitivity to VLM choice.

  • Seed Point Selection and Conditional Grasp Prediction: 13%: SeededGrasp outperforms DGN2.0 in single-embodiment grasping when both methods use identical seed points.The comparison trains both methods only on Allegro data because DGN2.0 supports a single gripper.
  • Multi-Embodiment Performance: Roughly 35%: SeededGrasp increases multi-embodiment success over Geomatch in complicated objects and cluttered scenes.Geomatch was designed for mid-air single-object grasping, whereas SeededGrasp handles the more complex tabletop setting.
  • Language-Conditioned Grasp Quality: SeededGrasp outperforms ShapeGrasp and GraspMAS in recognizing the target object, identifying its correct part, and producing a valid grasp.The 226-pair test set contained 164 Easy and 62 Difficult scene-prompt queries, with adherence manually scored.
  • Gripper Encoding: Using both robot point-cloud encoding and a learnable gripper-specific query vector is important, especially for Robotiq 3-Finger and Allegro grippers.Removing either representation decreases success rate and weakens modeling of embodiment-specific geometric intricacies.
  • VLM: Gemini 3.1 Flash outperforms competing VLM architectures for seed point prediction, while increasing the reasoning level does not improve performance.The framework is agnostic to the specific VLM, but backbone selection remains important given current VLM capabilities.
  • Dataset Scale: 71.5%: Performance improves with dataset size but saturates at around this level, while reducing training scenes disproportionately harms Robotiq 3-Finger and Allegro performance.The model was retrained using 25%, 50%, and 75% of the training scenes.

6 Limitations

SeededGrasp may predict grasp poses that robotic arms cannot execute because it lacks awareness of arm kinematics and obstacles. Future work should jointly optimize grasp selection and arm motion planning while incorporating multi-view VLM conditioning.

  • Kinematic feasibility: The model may propose infeasible grasp poses because robotic arms cannot reach every predicted pose, particularly when avoiding obstacles.Current predictions are unaware of the arm’s kinematics.
  • Kinematic feasibility: Future work should address grasp selection and arm motion planning jointly.
  • VLM conditioning: The model’s flexibility could improve by moving beyond single-image VLM conditioning to incorporate multi-view perspectives.

7 Conclusion

SeededGrasp enables language-grounded, multi-embodiment grasping in cluttered scenes by conditioning a flow-matching grasp generator on a VLM-predicted seed point. Its strong simulation and real-world performance demonstrates seed point conditioning as an effective bridge between language grounding and grasp generation.

  • 7 Conclusion: SeededGrasp combines a VLM-predicted seed point encoding semantic intent with a flow matching model for grasp generation.The framework is trained on a synthetic dataset for language-grounded grasping in cluttered scenes.
  • 7 Conclusion: SeededGrasp supports language-grounded, multi-embodiment grasping in cluttered scenes.
  • 7 Conclusion: Seed point conditioning effectively bridges language grounding and grasp generation.

A Appendix · A.1 Evaluation Setup

The appendix evaluates all three axes in Isaac Sim using a pass/fail grasp-consistency metric. Success requires avoiding significant initialization penetration and lifting the object 30 cm above the table without dropping it.

  • A.1 Evaluation Setup: All three evaluation axes were tested in Isaac Sim.The appendix uses Isaac Sim as the evaluation environment.
  • A.1 Evaluation Setup: Grasp consistency was measured with a pass/fail metric.This metric determines whether each grasp succeeds or fails.
  • A.1 Evaluation Setup: A grasp fails when significant penetration with objects occurs during initialization.Initialization-time object penetration is one stated failure condition.
  • A.1 Evaluation Setup: A grasp also fails if the gripper cannot lift the object 30 cm above the table.The lift criterion is part of the simulator’s pass/fail evaluation.
  • A.1 Evaluation Setup: Dropping the object during the 30 cm lift constitutes grasp failure.The gripper must complete the lift without dropping the object.
  • A.1 Evaluation Setup: These criteria determine success rates for all baseline and ablation studies.The same failure conditions are used to calculate reported success rates.

A.2 Baseline Setup Details

The baseline comparisons used embodiment-matched training and exposed important input constraints: DexGraspNet2.0 required explicit object masks for seed selection, while Geomatch required masks and single-object evaluation. ShapeGrasp and GraspMAS used BEV images with parallel-hand grippers, and ShapeGrasp was evaluated in its 2D mode.

  • DexGraspNet2.0: DexGraspNet2.0 used Allegro Hand data for fair comparison with the Leap Hand baseline and required explicit object masks to select the initial seed point.Most of its pipeline operates on the full scene point cloud, but seed selection requires an object mask.
  • Geomatch: Geomatch could train across multiple embodiments but autonomously selected conditioning through a top-k parameter rather than user-generated seed points.It does not support user-provided seed conditioning.
  • Geomatch: Geomatch required object masks and was evaluated only on single-object scenes, including simulation initialization.The method does not support multi-object scenes out of the box.
  • ShapeGrasp & GraspMAS: ShapeGrasp and GraspMAS used parallel-hand grippers and relied entirely on Bird’s Eye View images of the scene.ShapeGrasp supports only single-object images and was provided with object segmentation masks.
  • ShapeGrasp & GraspMAS: ShapeGrasp was evaluated in its 2D mode using gemini-3-flash-preview because its native 3D mode failed too often during testing.The same model was used in the proposed method and ShapeGrasp evaluation.

A.3 Language Conditioned Grasp Test Dataset … A.7 Real World Individual Item Success Rates

The appendix defines language-conditioned grasp tests, analyzes baseline failures and ablations, specifies the VLM prompt, and reports real-world failure causes. Results show embodiment-dependent dataset-size effects and consistent VLM rankings across grippers.

  • A.3 Language Conditioned Grasp Test Dataset: The test dataset uses easy and difficult prompts to evaluate identifying the correct object and grasping its task-relevant part.Easy prompts specify the part directly, whereas difficult prompts describe a task requiring the method to infer the best graspable part.
  • A.3.1 Failure Cases for Baselines: Geomatch failures primarily arise from object penetrations caused by missing underside geometry and complicated non-convex objects.These issues result from fusing multiple camera views and interactions with the IK optimization algorithm.
  • A.3.1 Failure Cases for Baselines: ShapeGrasp performs well on decomposable non-convex objects, but its decomposition can be too aggressive or relaxed for mostly convex objects.The resulting grasp prediction may become too imprecise, as illustrated by the green speaker and brown shoe cases.
  • A.3.1 Failure Cases for Baselines: GraspMAS failures involve recognizing the wrong object or predicting an imprecise grasp location that targets the wrong part.Both GraspMAS and ShapeGrasp rely on a Bird’s Eye View image, making 3D grasp selection difficult under complex geometry and occlusion.
  • A.4 Dataset Size Ablation: The Franka gripper shows little effect from decreasing dataset size, while the Robotiq 3-Finger and Allegro Hand benefit from the entire dataset.Marginal performance improvement decreases as more of the dataset is used.
  • A.5 VLM Ablation: The relative ranking of the three VLMs is the same for all grippers, with Gemini 3.1 Flash outperforming the others by a large margin.Table 6 reports success rates for seed point prediction across all three grippers.
  • A.6 VLM Prompt: The VLM prompt alternates scene-description text and image-url parts, requesting normalized pixel coordinates as JSON for a stable grasp location.Requirements include collision avoidance, placement on the target surface, and gripper encapsulation of the object.
  • A.7 Real World Individual Item Success Rates: The model predicted feasible grasps for all real-world objects, but failures occurred when the gripper was too far away or the grasp was highly angled.A distant gripper could close without actually grasping the object.
Loading 2607.20207v1…