PaperlayerView exact revision on arXiv

Source-linked AI summary

GraspTwin: Zero-Shot Task-Oriented Grasp Optimization via a Digital Twin

Daniel J. Evans, Yinlong Dai, Simon Stepputtis, Dylan P. Losey

arXiv:2609.30543v1cs.RO

TL;DR

Task-oriented grasping needs poses that support a task, while robust grasping methods may lack task semantics and foundation-model proposals may lack physical grounding. GraspTwin uses VLM proposals as seeds for optimization in a digital twin before executing the resulting grasp on the real robot. Without optimization, overall task performance falls from 51.6% to 22.9%.

  • Problem

    Robust grasping methods may be task-agnostic, while language-based grasp proposals can be difficult to ground in embodied motion.

    Source: §I · p. 1

  • Method

    GraspTwin uses VLM-proposed task-oriented grasps as seeds for local, gradient-free optimization in a digital twin before real-robot execution.

    Source: §I · p. 2, VLM, RVLM · p. 5, §VI · p. 8

  • Results

    Without optimization, overall task performance drops from 51.6% to 22.9%, with reduced robustness and physical feasibility.

    Source: C. Evaluating GraspTwin Components… · pp. 7–8

  • Takeaways & Limitations

    The digital twin lets the system refine semantic grasp proposals into physically feasible grasps for execution on the robot.

    Source: §VI · p. 8

    The pipeline takes roughly 5 minutes, with most time spent evaluating batched grasp proposals in the physics simulator.

    Source: §VI · p. 8

Abstract

from arXiv · show

As robots transition from structured factory settings into homes, they are required to interact with an ever-increasing variety of objects. Many tasks require grasping, and often it is not sufficient to just pick up the target object. Consider a task like "pouring coffee" --- to facilitate the subsequent pouring, the robot should grasp the mug by its handle. Existing learning-based approaches for grasping either find robust and collision-free grasps that are largely agnostic to the task (e.g., picking up the mug by its rim), or leverage foundation models to propose task-appropriate grasp locations that lack fine-grained physical grounding (e.g., reaching for and missing the handle). In this work, we bridge these approaches with a real-to-sim-to-real framework. Based on a single RGB-D observation, we construct a digital twin of the environment, query a large foundation model to propose grasps that align with the object's affordances and task description, and then optimize the proposals to ensure robustness and plausibility before executing the result on the real robot. Our key insight is that the grasp proposals of the foundation model should be regarded as semantic priors that serve as seeds for local, gradient-free optimization. We leverage Bayesian optimization with Thompson sampling to draw batches of nearby poses, which are subsequently evaluated in parallel under domain-randomized physics rollouts. The resulting grasp is both task-oriented and physically feasible for execution by the robot arm. Our full zero-shot real-world transfer only takes a few minutes and improves task-oriented grasping success by up to 33% as compared to other state-of-the-art pipelines. Our code is available here: https://github.com/VT-Collab/GraspTwin/

I. INTRODUCTION

Grasping must satisfy physical constraints, but robust grasps may not support the downstream task, while semantic proposals may be difficult to ground in embodied motion. GraspTwin uses VLM proposals as seeds for optimization in a digital twin before real-robot execution.

I. INTRODUCTION

  • Robust, collision-free grasping alone may miss task requirements, while language-based proposals can be hard to ground in embodied motion.A mug grasped by its body may place the fingers inside the mug during pouring.Source: §I · p. 1
  • GraspTwin combines task-aligned VLM proposals with local optimization in a digital twin to produce physically plausible grasps for real-robot execution.The pipeline starts from a fixed RGB-D observation and uses the proposals as optimization seeds.Source: Figure 1 · p. 1, §I · p. 2
  • GraspTwin builds a digital twin from real-world observations, uses rendered object views to obtain VLM proposals, and refines them under physical constraints and domain randomization.The proposals reflect object affordances and the downstream task.Source: Figure 1 · p. 1, §I · p. 2
  • Across 30 standardized tasks, including clutter and lighting or camera variations, the approach increases absolute and semantically-aware grasping success in simulation and real-world experiments.The comparisons include ablations and state-of-the-art baselines.Source: §I · p. 2

II. RELATED WORKS

Task-agnostic grasping emphasizes stable, feasible grasps, whereas task-oriented methods seek grasps that facilitate completion of the current task. The paper positions GraspTwin as closing the gap between semantic proposal and physical execution.

A. Task-Agnostic Grasp Generation

  • Task-agnostic methods find feasible and stable grasps but do not account for whether those grasps align with the current task.Approaches include analytic metrics, point-cloud methods, and learned grasp models.Source: A. Task-Agnostic Grasp Generation · p. 2

B. Task-Oriented and Semantic Grasping

  • Task-oriented grasping requires a grasp that facilitates task completion, but fine-tuning foundation models needs large labeled corpora and may reduce generalization.Other approaches use off-the-shelf language and vision models to reason about grasp locations, then pass regions to task-agnostic samplers that may alter the grasp point.Source: B. Task-Oriented and Semantic Grasping · p. 2
  • GraspTwin closes the proposal-execution loop by iteratively testing and refining task-oriented grasps in a physics simulator.This addresses the disconnection that can leave proposals kinematically infeasible, collision-prone, or inconsistent with functional constraints.Source: B. Task-Oriented and Semantic Grasping · p. 2

C. Real-to-Sim-to-Real for Manipulation

The paper frames task-oriented grasping as finding a pose that respects task semantics, robot kinematics, and nearby-object collisions. Its real-to-sim-to-real setting assumes a fixed-camera RGB-D observation and restricts objects to rigid, non-articulated items.

C. Real-to-Sim-to-Real for Manipulation

  • Real-to-sim tools enable a pipeline that builds a physics-based simulation from a real image, optimizes a grasp with domain randomization, and executes it in the real world.The paper situates this approach among systems that construct simulation-ready scenes from real observations.Source: C. Real-to-Sim-to-Real for Manipulation · pp. 2–3

III. PROBLEM STATEMENT

  • A suitable grasp pose must facilitate the task while respecting robot kinematics and avoiding collisions with other objects.The input is a natural-language task and an image of the tabletop environment; the output is a pose in SE(3).Source: §III · p. 3
  • The setting uses a mounted 7-DoF arm, a 1-DoF gripper, a fixed external RGB-D camera, and assumes a model of robot kinematics and dynamics.The arm executes waypoint-based control along joint-space trajectories.Source: §III · p. 3
  • Objects may be crowded or touching but must be rigid, non-articulated, and not fully occluded; the planner should succeed on the first trial without demonstrations or real-world training attempts.The target grasp should be stable, collision-free, and meaningful for the task.Source: §III · p. 3

IV. OPTIMIZING TASK-ORIENTED GRASPS

GraspTwin constructs a digital twin from an RGB-D image, uses VLMs to propose task-aligned grasps, and optimizes those proposals in simulation. The optimized pose is then executed by the robot.

IV. OPTIMIZING TASK-ORIENTED GRASPS

  • The method proceeds from digital-twin construction to VLM task-oriented initialization, then gradient-free optimization and real-robot execution, taking roughly 5 minutes on average.Optimization seeks robust, physically feasible, collision-free poses.Source: §IV · p. 3

A. Constructing the Digital Twin

  • The digital twin must reconstruct complete object meshes from one image and model robot-object interactions well enough for simulated grasp outcomes to predict real ones.The twin supports both multi-view VLM proposals and grasp testing.Source: A. Constructing the Digital Twin · p. 3
  • The system detects and labels objects, uses the task and labels to identify the target, then applies SAM-3D to estimate the object set from the RGB-D image.The target object receives the first index in the estimated set.Source: A. Constructing the Digital Twin · pp. 3–4
  • The system renders the target from multiple views for VLM grasp proposals, then uses batched Bayesian optimization with Thompson sampling to test candidate poses under object perturbations.The minimum-cost pose is executed by the real robot; nearby objects receive progressively lower-resolution meshes.Source: Figure 2 · p. 4, A. Constructing the Digital Twin · p. 4
  • The twin assigns objects dynamic parameters that may be inaccurate, and domain randomization is used to improve robustness to those dynamics.The simulation is initialized with a robot arm and extracted objects on a table.Source: A. Constructing the Digital Twin · p. 4

B. Identifying Task-Oriented Grasps

GraspTwin uses a digital twin and VLM reasoning to propose grasp positions and orientations suited to the task, then treats those proposals as starting points for optimization. Multi-view rendering and localization turn semantic grasp descriptions into 3D grasp candidates.

B. Identifying Task-Oriented Grasps

  • A VLM first describes a grasp position and approach orientation that would facilitate the task, such as grabbing a handle and keeping clear of the pouring path.The description is generated by reasoning across the task instruction and the intended grasp.Source: B. Identifying Task-Oriented Grasps · p. 4
  • Rendered views and a semantic grasp-position description yield candidate pixel locations that are back-projected with depth and camera parameters into 3D positions.Candidate locations are clustered by density, isolated predictions pruned, and cluster centroids used as grasp-point proposals.Source: B. Identifying Task-Oriented Grasps · p. 4
  • For orientation proposals, colored gripper wireframes aligned with bounding-box normals let the VLM select which side of the object to approach.Fine-grained orientation accuracy is left to downstream optimization.Source: B. Identifying Task-Oriented Grasps · p. 5
  • VLM proposals are treated as initial guesses because semantic reasoning alone may not ground them in the robot or environment.The digital twin and reconstructed meshes support rendering the target from multiple perspectives before proposals are refined.Source: B. Identifying Task-Oriented Grasps · pp. 4–5

C. Optimizing the Proposed Grasps

GraspTwin refines VLM-proposed poses in a domain-randomized digital twin. Bayesian optimization searches near task-aware proposals to improve grasp feasibility and robustness.

C. Optimizing the Proposed Grasps

  • Gradient-free Bayesian optimization refines VLM-proposed positions and orientations within a domain-randomized digital twin.It accounts for grasp success, joint limits, collisions, object interactions, and other robot or environmental constraints.Source: C. Optimizing the Proposed Grasps · p. 5
  • The optimization region is constrained to poses near task-aware proposals, preserving their semantic meaning while allowing low-level adjustments.The proposed grasp pose T is a position-orientation pair, T = (P, R).Source: C. Optimizing the Proposed Grasps · p. 5
  • Bayesian optimization is used because grasp success is sparse and non-convex.The optimization is seeded by task-aware proposals.Source: C. Optimizing the Proposed Grasps · p. 5

VLM, RVLM

The method searches locally around task-oriented proposals, scoring poses through simulated grasp attempts and domain-randomized rollouts. Batched Bayesian optimization selects candidates, and the lowest-cost evaluated pose is transferred to the real robot.

Optimization Region

  • The search space is the union of local regions around the proposed poses, with positional and angular radii setting each region's bounds.These bounds preserve high-level semantic meaning while allowing low-level pose adjustments.Source: VLM, RVLM · p. 5

Cost Function

  • The simulation assigns cost 0 to poses the motion planner cannot reach without collisions or joint-limit violations, and returns 1 for a successful lift and 0 otherwise.The simulated motion plans a collision-free inverse-kinematics reach, closes the gripper, and lifts vertically.Source: VLM, RVLM · p. 5
  • Domain randomization perturbs object x-y position and rotation around z, and the expected grasp cost represents success probability under that perturbation distribution.The expectation is approximated with randomized simulation rollouts; a negative sign makes minimizing cost correspond to maximizing successful, collision-free grasps.Source: VLM, RVLM · p. 5

Bayesian Optimization

  • Bayesian optimization with Thompson sampling draws batches of candidate poses to minimize cost over the search space.Previously tested poses and costs inform subsequent batches.Source: VLM, RVLM · p. 5

Tasks and Environments

  • Table I lists the experiment tasks: all 30 are used in simulated testing, while a subset of 10 is used for real-world evaluation.Source: Table I · p. 6

Bayesian Optimization and Sim-to-Real

  • Candidates are evaluated in parallel in the digital twin, and the evaluated pose with the lowest cost after the optimization rounds is returned.The resulting pose is used directly on the real robot, which plans a collision-free path and closes its gripper at the target pose.Source: VLM, RVLM · p. 6

V. EXPERIMENTS

The experiments evaluate GraspTwin in simulation and on a real robot across task-oriented grasping scenarios. They compare it with two baselines and two ablations using lift, collision, and task-alignment criteria.

V. EXPERIMENTS

  • The study uses 45 curated scenes and evaluates 450 grasps across five grasp pipelines in simulation.The scenes contain varied objects and natural-language task instructions.Source: §V · p. 6

Tasks and Environments

  • The simulated benchmark comprises 45 scenes with eight objects each and 30 tasks of varying difficulty.Most tasks are derived from FoundationGrasp, LERF-TOGO, and AffordGrasp.Source: §V · p. 6

Independent Variables

  • The five pipelines are GraspTwin, GraspMolmo, LERF-TOGO, and two ablations that remove optimization or replace the task-oriented proposal seed.Ours-O uses the raw VLM proposal; Ours-G seeds optimization with GraspMolmo's pose.Source: §V · p. 6

Dependent Variables

  • Success is assessed by whether the grasp lifts and holds the object, avoids collisions with non-target objects, and uses a task-meaningful grasp region.Source: §V · p. 6

Experimental Setup

  • The experiments use a 7-DoF Franka Emika Panda arm with a UMI gripper in simulation and the real-world setup.The algorithms run on two NVIDIA L40S GPUs.Source: §V · p. 6

A. Comparing to Baselines in Simulation

Across 30 simulated tasks, GraspTwin outperformed GraspMolmo in overall grasp success, while lift robustness remained comparable. The experiments used 450 grasp attempts per method.

A. Comparing to Baselines in Simulation

B. Testing Zero-Shot Transfer in Real-World Experiments

In real-world tests, GraspTwin substantially outperformed GraspMolmo and transferred with a smaller simulation-to-real change in overall grasp success. Its largest reported differences were in semantic alignment and obstacle avoidance.

B. Testing Zero-Shot Transfer in Real-World Experiments

  • The real-world evaluation tested 10 tasks four times each with GraspTwin and GraspMolmo, focusing on whether a proposed grasp succeeded on its first real-world attempt.The same procedure as the earlier experiments was followed.Source: B. Testing Zero-Shot Transfer in… · p. 7
  • GraspTwin varied by 9.2% in overall grasp success between simulation and real-world evaluation, while GraspMolmo lost 27.7%.The authors attribute GraspTwin's small transfer gap to the digital twin's accuracy and hypothesize that GraspMolmo's larger gap reflects its inability to fine-tune proposals.Source: B. Testing Zero-Shot Transfer in… · p. 7
  • GraspTwin achieved roughly a three-fold performance improvement over GraspMolmo, with task-oriented grasps twice as often and collision-free grasps more than twice as often.The semantic-alignment and obstacle-avoidance comparisons were the two largest differences reported.Source: B. Testing Zero-Shot Transfer in… · p. 7

C. Evaluating GraspTwin Components through Ablations

The ablations show that optimization and task-oriented proposals contribute distinct capabilities to GraspTwin. The conclusion reports improved simulated and real-world results, while noting that the pipeline takes roughly five minutes end to end.

C. Evaluating GraspTwin Components through Ablations

  • Removing optimization reduced overall task performance from 51.6% to 22.9% and lowered both lift robustness and collision-free feasibility.The authors say optimization grounds task-oriented VLM proposals in physically feasible grasps for robot execution.Source: C. Evaluating GraspTwin Components… · pp. 7–8
  • Replacing GraspTwin's semantic proposals with GraspMolmo proposals reduced task-oriented grasps, while adding optimization increased collision-free grasps without significantly increasing overall performance.The authors suggest that rendering multiple object views and querying a VLM provides a semantically meaningful initial guess for downstream optimization.Source: C. Evaluating GraspTwin Components… · pp. 7–8

VI. CONCLUSION

  • GraspTwin outperformed state-of-the-art baselines in simulations by 18 percentage points over the closest alternative and in real-world experiments by 33% absolute improvement.The conclusion describes the overall framework as using semantic proposals refined through parallel, domain-randomized tests in a digital twin.Source: §VI · p. 8
  • The end-to-end pipeline took roughly five minutes, with most time spent evaluating batched proposals in the physics simulator.The authors anticipate that concurrent rollouts and more computational hardware could significantly reduce execution time.Source: §VI · p. 8
Loading 2609.30543v1…