Source-linked AI summary
TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping
Manav Kulshrestha, Apoorva Vashisth, Damon Conover, Aniket Bera
TL;DR
Task-oriented grasping must match the grasp to the task, but existing approaches are limited by finite task-specific datasets, task-agnostic generators, or planar predictions. TGrasp6D uses a VLM to reason about task-relevant affordances from one RGB-D observation, then refines grasp poses for collision avoidance and task consistency. It achieves a 94.5% TaskGrasp success rate and demonstrates zero-shot transfer in real-robot experiments.
Problem
Task-oriented grasping methods are limited by finite annotated task-specific datasets, task-agnostic generators, or planar predictions, restricting coverage of possible grasp-task pairs.
Source: §I · p. 1Method
TGrasp6D uses a vision-language model to identify task-relevant affordances from an RGB-D observation and instruction, then lifts a grasp prior into 3D and refines poses for collision avoidance and task consistency.
Source: §I · p. 1, C. 6DoF Grasp-Pose Refinement · p. 5Results
94.5% success rate on TaskGrasp, outperforming the strongest baseline by over 20 percentage points, with real-robot experiments demonstrating zero-shot transfer across diverse object-task pairs.
Source: §VII · p. 8Takeaways & Limitations
Vision-language priors can guide geometrically feasible, task-aware grasp synthesis without task-specific training data or task-agnostic grasp generators.
Source: §VII · p. 8A single RGB-D observation makes grasp synthesis sensitive to depth noise, self-occlusion, and incomplete point clouds, and the single-region formulation concentrates generation around one mode of the valid grasp space.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Task-Oriented Grasping (TOG) has emerged as an important skill for purposeful robotic manipulation. Given the task of using a knife to "slice bread," a robot must choose to grasp from the handle, whereas for "hand[ing] the knife over," it is preferable for the handle to remain unobstructed for safe handoff. Current approaches for TOG either focus on 2D planar grasping, remain limited by their reliance on task-agnostic grasp generators, or are constrained by reliance on finite, manually-annotated, task-specific grasp datasets that cannot cover all possible grasp-task pairs. To that end, we present TGrasp6D, a vision-language based approach for generating task-oriented 6DoF grasps in a zero-shot manner. Given a task and a single RGB-D image of an associated object, TGrasp6D reasons about the appropriate semantic affordance for the given task and object geometry to produce an initial task- informed prior. This generated prior is further refined via 6DoF pose optimization for collision avoidance while preserving task consistency. Our approach achieves state-of-the-art success rate of 94% on the TaskGrasp dataset, with the best baseline at 74%. We also deploy our approach on a Franka Research 3 robot to demonstrate zero-shot generalization to real world objects
I. INTRODUCTION
Task-oriented grasping must match hand configuration to intent, but existing approaches are limited by sparse task-specific annotations, fixed task-agnostic generators, or planar predictions. TGrasp6D uses vision-language reasoning and 3D pose refinement to synthesize task-oriented 6DoF grasps without task-specific training, achieving a 94% TaskGrasp success rate and demonstrating zero-shot transfer on a robot.
I. INTRODUCTION
- Task-oriented grasping couples grasp selection to intent, unlike stable-grasp prediction that does not consider whether a configuration supports the intended task.The introduction points to tool handoff and industrial re-grasping as settings where the appropriate grasp depends on the task.Source: §I · p. 1
- Existing approaches rely on bounded task-specific datasets, task-agnostic candidate generators, or planar predictions that exclude out-of-plane grasps.Task-agnostic generators cannot recover a task-appropriate grasp they did not propose, while dataset coverage remains sparse across possible task-object-grasp combinations.Source: §I · p. 1
- For the same object, handover keeps the handle clear, whereas pounding or cooking requires holding the hammer or pan by its handle.This example illustrates how the task changes the desired grasp.Source: Figure 1 · p. 1
- TGrasp6D combines a single RGB-D observation and natural-language instruction with affordance reasoning, 3D grasp-prior generation, and collision-aware pose refinement.The refinement balances collision avoidance with consistency with the task-relevant region.Source: §I · pp. 1–2
- 94% success rate on TaskGrasp, outperforming the strongest baseline by ∼20%.The paper also reports zero-shot transfer to real-world manipulation across novel objects and task instructions on a Franka Research 3 robot.Source: §I · p. 2
II. RELATED WORK
Prior grasping work includes task-agnostic 6DoF synthesis and task-oriented methods that use annotations, fixed candidate generators, or planar predictions. These approaches differ in representation and supervision, but the latter methods remain bounded by dataset coverage, upstream candidate coverage, or image-plane constraints.
A. Task-Agnostic 6DoF Grasp Synthesis
- Task-agnostic 6DoF methods optimize grasp stability and feasibility, while TGrasp6D conditions feasible-pose search on a task-relevant region identified before optimization.The surveyed methods use varied representations and supervision, including point-cloud contacts, dense grasp perception, language goals, shape reconstruction, and SE(3) diffusion.Source: A. Task-Agnostic 6DoF Grasp Synthesis · p. 2
B. Task-Oriented Grasping
- Task-oriented methods use annotated corpora, task-agnostic generators, or image-plane predictions, limiting coverage respectively by dataset scope, proposed candidates, or planar geometry.Task-agnostic candidates cannot supply a task-appropriate grasp that was never proposed, and planar predictions exclude feasible out-of-plane grasps.Source: B. Task-Oriented Grasping · p. 2
C. Vision-Language Models for Affordance Reasoning in Manipulation
Prior VLM manipulation work grounds affordances in 3D or predicts image keypoints, while related grasping work either delegates pose synthesis to a task-free generator or requires fine-tuning. TGrasp6D formulates grasp synthesis from an object image, depth map, and task, then localizes an affordance and lifts it into a refined 6DoF grasp.
C. Vision-Language Models for Affordance Reasoning in Manipulation
- Prior VLM affordance methods include 3D value maps for trajectory planning and image-keypoint outputs that require another module to produce executable actions.For grasping, the cited closest prior approaches use a task-free generator or fine-tune on a large synthetic corpus; TGrasp6D instead localizes an affordance and refines a feasible 6DoF grasp.Source: C. Vision-Language Models for… · pp. 2–3
III. PROBLEM FORMULATION
- The formulation seeks a grasp from one object image, depth map, and task instruction that is both stable for the object and appropriate to the task.It represents grasp pose in SE(3), with translation and rotation mapped into the relevant output space.Source: §III · p. 3
IV. METHODOLOGY
- TGrasp6D uses VLM reasoning and localization to form a 3D grasp prior, samples candidate grasps, and refines poses for collision avoidance and task-region consistency.The method describes these as three main components, with the overview shown in Fig. 2.Source: §IV · p. 3, Figure 2 · p. 3
A. Task-Oriented Reasoning
TGrasp6D uses language-model priors over objects and tasks to reason about task-relevant affordances. It localizes the selected object part in an image, using masking and cropping to reduce distractions.
A. Task-Oriented Reasoning
- The method leverages language-model priors over object and task concepts represented in internet-scale training data.The paper treats the joint space of objects and tasks as represented in the VLM’s learned concept space.Source: A. Task-Oriented Reasoning · p. 3
- The task instruction selects an object-part affordance, such as a sword’s hilt for stabbing or blade for handover.The selected affordance is localized as a 2D image point.Source: A. Task-Oriented Reasoning · p. 4
- An object mask and close crop reduce background distractions before the VLM localizes the task-relevant affordance.The image transformation retains residual information needed to recover the grasp in the original image space.Source: A. Task-Oriented Reasoning · p. 4
B. Grasp Prior Generation
TGrasp6D identifies and localizes a task-relevant object affordance, then lifts image-space position and orientation into 3D to form a vision-language grasp prior. The prior’s mean and covariance bias grasp sampling toward the localized affordance and orientation.
B. Grasp Prior Generation
- A localized affordance alone does not ensure a stable or task-appropriate grasp; for a mug handle, the gripper closing axis must be perpendicular to the handle.Source: B. Grasp Prior Generation · p. 4
- The method represents the image-space grasp orientation as a 2D closing-axis line through the localized affordance and lifts it into 3D using camera intrinsics and extrinsics.The image-space affordance localization is also lifted into 3D; the lifting function uses the depth map and residual image information.Source: B. Grasp Prior Generation · p. 4
- In 3D, the lifted position is a line and the lifted orientation is a plane; the prior mean uses the closest point-cloud point and aligns the gripper to that plane and the camera view.Source: B. Grasp Prior Generation · p. 4
C. 6DoF Grasp-Pose Refinement
TGrasp6D uses its vision-language grasp prior to initialize candidate poses and refines them through independent pose optimization. The objective trades off collision avoidance, task consistency, and deviation from each candidate prior.
C. 6DoF Grasp-Pose Refinement
- The prior mean remains in the image plane, potentially excluding feasible 6DoF grasps, while depth, localization, and orientation errors can compound into failures.TGrasp6D therefore uses the prior to initialize candidate grasps for refinement.Source: C. 6DoF Grasp-Pose Refinement · p. 4
- Each candidate grasp is independently refined through pose optimization, using an objective that scores collision and task consistency.The optimization parameter perturbs the candidate pose in SE(3).Source: C. 6DoF Grasp-Pose Refinement · p. 4
- The objective penalizes gripper collisions with the object point cloud, rewards the task-relevant affordance within the gripper sweep, and regularizes pose changes from the candidate.Source: C. 6DoF Grasp-Pose Refinement · p. 5
- For online refinement, the gripper geometry’s signed distance field is learned offline with DeepSDF, while the sweep volume is estimated using a bounding box.Source: C. 6DoF Grasp-Pose Refinement · p. 5
A. Setup
Evaluation uses selected TaskGrasp object–task pairs, a specified grasp-success criterion, and a shared hyperparameter configuration. Experiments also use a workstation and a Franka Research 3 robot.
A. Setup
- Evaluation retains 310 TaskGrasp instances with at least 20 ground-truth grasp annotations per object–task pair; success requires sweep-volume IoU of at least 0.1 with a ground-truth grasp.The study reports success rate and generation failure rate, plus orientation and position errors for generated grasps classified as failures.Source: A. Setup · p. 5
- Hyperparameters are selected by grid search on a small validation set, with the final configuration used across reported experiments unless otherwise stated.Source: A. Setup · p. 5
- Real-world experiments use a Franka Research 3 manipulator with a parallel-jaw gripper; grasp synthesis and pose refinement run on the workstation.The workstation has an NVIDIA RTX 5080 GPU and a 20-core Intel Core i7 CPU.Source: A. Setup · p. 5
B. Baseline Comparison
TGrasp6D outperforms baseline methods on TaskGrasp, while refinement and VLM ablations show how geometric correction and backbone choice shape outcomes.
B. Baseline Comparison
- In Table I's quantitative baseline comparison, TGrasp6D achieves the highest success rate of 94.5%, exceeding the strongest baseline, FoundationGrasp, at 73.7%. With GPT-5.5 as the underlying VLM, its generation failure rate is 0%.fOE and fPE are reported only for grasps marked as failures. Approaches with lower scores for these metrics are primarily training based and may overfit to grasp distributions seen during training.Source: B. Baseline Comparison · p. 5, Table I · p. 5
- Higher fOE and fPE may reflect a wider spread of generations that the human-annotated dataset does not adequately represent.These metrics are reported only for grasps marked as failures.Source: B. Baseline Comparison · pp. 5–6
- TGrasp6D is more consistent on task-dependent grasps such as handover than methods that handle unambiguous spoon-scooping cases well.Fig. 3 shows five task-oriented grasps per method and object; handover requires avoiding the region intended for subsequent use.Source: B. Baseline Comparison · p. 6, Figure 3 · p. 6
C. Ablation Studies
- Removing pose refinement reduces success from 94.5% to 89.4%; refinement lowers position failures from 3.5% to 1.6% and orientation failures from 3.9% to 0.6%.Affordance failures remain at 3.2%, since pose refinement cannot fully recover an incorrectly inferred task-relevant affordance.Source: C. Ablation Studies · p. 6
- Pose refinement produces more diverse feasible grasps around the task-relevant region while preserving the initial prior's semantic intent.The visual comparison also shows feasible, stable, task-consistent alternatives.Source: C. Ablation Studies · p. 7
- GPT 5.5 achieves 89.35% success, Gemini 3.7 achieves 84.19%, and Qwen3.5-35B-4bit achieves 75.16%, a 14.52-point spread.Gemini's dominant failure is localization (11.0%); Qwen's is generation failure (19.7%); GPT 5.5 has no generation failures.Source: C. Ablation Studies · p. 7
- GPT 5.5 uses the fewest tokens for affordance identification, localization, and grasp-axis prediction, while VLM choice affects success, efficiency, and propagated error types.Its average token counts are 0.20 × 10^3, 0.34 × 10^3, and 0.61 × 10^3, respectively.Source: C. Ablation Studies · p. 7
D. Real-Robot Deployment
On a Franka Research 3, TGrasp6D succeeds in 93 of 100 trials across real-world objects and tasks. Failures arise from observed-geometry imperfections and errors in the vision-language prior.
D. Real-Robot Deployment
- 93/100 trials succeed with TGrasp6D, compared with 41/100 for GraspGPT and 52/100 for FoundationGrasp.The evaluation covers five real-world objects and ten object–task combinations, with ten trials per combination.Source: D. Real-Robot Deployment · p. 7
- Depth noise and self-occlusion dominate TGrasp6D's real-world failures by disrupting feasible grasp construction or collision-aware refinement.Remaining failures arise from affordance-localization or grasp-orientation errors in the vision-language prior.Source: D. Real-Robot Deployment · p. 7
- GraspGPT and FoundationGrasp fail when task-agnostic generators supply poor candidates or task-conditioned scoring selects an inappropriate affordance, such as the sword blade for “stab.”TGrasp6D directly grounds candidates in task-relevant regions, avoiding those candidate-coverage and semantic-misselection problems.Source: D. Real-Robot Deployment · p. 7
VI. LIMITATIONS AND FUTURE WORK
TGrasp6D demonstrates strong benchmark and real-robot results, while its current formulation has perception, computational, and grasp-space limitations.
VI. LIMITATIONS AND FUTURE WORK
- A single RGB-D observation makes synthesis sensitive to depth noise, self-occlusion, and incomplete point clouds, potentially leaving collision geometry unobserved.The refinement stage can accept poses that collide with that unobserved geometry.Source: §VI · p. 8
- Large VLM dependence brings inference latency and hosted-model API costs, while local alternatives can be computationally expensive.Source: §VI · p. 8
- Localizing one task-relevant region concentrates grasp generation around a single mode of the valid grasp space.Source: §VI · p. 8
- The refinement objective omits explicit contact-mechanics and closed-loop execution models; proposed future directions include multi-view perception, multiple affordance hypotheses, and feedback-driven grasping.Source: §VI · p. 8
VII. CONCLUSION
- 94.5% success rate on TaskGrasp, outperforming the strongest baseline by over 20 percentage points; real-robot experiments show zero-shot transfer across diverse object–task pairs.The conclusion reports these results for the proposed method.Source: §VII · p. 8