PaperlayerView exact revision on arXiv

Source-linked AI summary

DA-GRD: Decision-Aware Grasp-Relevant Disambiguation for tactile recovery under perception-to-execution mismatches

Haoran Wang, Yuteng Sun, Yuanjie Li, Ruofei Bai, Meng Yee, Chuah, Wenyu Liang, Jun Li, Wei-Yun Yau

arXiv:2609.29065v1cs.RO

TL;DR

When an object moves after visual grasp generation and vision is unavailable, the paper asks whether sparse tactile interactions can recover an executable grasp. DA-GRD selects probes using a weighted planar belief and stops when the remaining hypotheses support a common grasp; in MuJoCo it attains 84.7% overall physical lift success, above three recovery baselines.

  • Problem

    The paper asks whether sparse tactile interactions can recover an executable grasp after an unobserved object displacement makes a task-conditioned visual grasp spatially stale.

    Source: §I · p. 1

  • Method

    DA-GRD maintains a weighted SE(2) belief and selects tactile probes by their ability to eliminate pose hypotheses and improve agreement among candidate task grasps.

    Source: §I · p. 1, D. Grasp-Relevant Probe Selection · p. 4

  • Results

    84.7% overall physical lift success for Ours-PI versus 9.1% stale AnyGrasp, 21.2% original LGWS, and 63.7% LGWS adapted in MuJoCo; it uses 4.13 tactile probes on average.

    Source: C. Results and Analysis · p. 6, §V · p. 7

  • Takeaways & Limitations

    The results support that complete object localization is not always necessary when remaining pose hypotheses already imply similar grasp actions.

    Source: §V · p. 7

    The method is limited to planar SE(2) uncertainty and partial remembered geometry, which can be insufficient for complex non-planar objects.

    Source: §V · p. 7

Abstract

from arXiv · show

Grasping is a fundamental robotic capability that bridges perception and physical task execution. This paper studies grasp pose recovery under a perception-to-execution mismatch, where a grasp generated from visual perception may become spatially stale if the object moves before execution, using only sparse tactile interactions and no further visual observations. We propose DA-GRD, Decision-Aware Grasp-Relevant Disambiguation, which maintains a weighted planar belief over possible object configurations and selects tactile probes according to their ability to eliminate hypotheses and improve agreement among candidate task grasps. Rather than fully relocalizing the object, DA-GRD stops when the remaining hypotheses support a common executable grasp. In MuJoCo experiments on ten rigid objects with translations up to 5~cm and yaw perturbations up to $\pm45^\circ$, DA-GRD achieves an 84.7% physical lift success rate, compared with 9.1% for stale AnyGrasp, 21.2% for the original fix-scan baseline, and 63.7% for fix-scan method adapted with an SE(2) belief. DA-GRD also achieves a 57.3% Task conditioned Success rate. Across objects, it uses a success-average of 4.13 tactile probes over the ten per-object means, corresponding to a 72.5% reduction relative to the fixed 15-probe baselines. Real-world experiments on six objects achieve 71.7% physical lift success and 38.3% task-conditioned success with 4.20 probes on average. These results show that tactile sensing can recover task-relevant grasps under vision-off conditions with limited physical interaction, without requiring complete object localization.

I. INTRODUCTION

A task-conditioned visual grasp can become spatially stale when an object moves before execution and no further vision is available. DA-GRD uses tactile evidence to seek an executable grasp without requiring full object relocalization.

I. INTRODUCTION

  • The paper asks whether sparse tactile interactions can recover an executable grasp after a task-conditioned visual grasp becomes spatially stale.Object motion may follow human movement, prior robot interaction, clutter contact, support motion, or registration error; visual feedback may be unavailable under occlusion or in confined workspaces.Source: §I · p. 1
  • DA-GRD maintains a weighted planar belief and stops when the remaining hypotheses support a common executable grasp.Its goal is grasp-relevant disambiguation rather than complete pose convergence.Source: §I · p. 1, Figure 1 · p. 2
  • DA-GRD ranks probes by miss-exclusion mass, newly covered observed surface, and counterfactual improvement in common grasp support.It alternates probe feedback with grasp-coverage evaluation and executes a grasp once support is sufficient.Source: §I · p. 1, Figure 1 · p. 2
  • The framework combines initial language-conditioned grasp generation with tactile adjustment and a grasp-relevant SE(2) belief that aggregates multiple object-pose hypotheses into a committed grasp.The benchmark compares against stale AnyGrasp, fixed touch-scan LGWS, and LGWS adapted to an SE(2) belief; DA-GRD is reported to achieve higher success with fewer probes.Source: §I · pp. 1–2

II. RELATED WORK

Prior work addresses visual and language-conditioned grasp generation, tactile recovery, and active tactile exploration. DA-GRD begins from a task-conditioned visual grasp that becomes spatially stale and evaluates uncertainty by whether it changes the downstream grasp.

A. Visual and Language-Conditioned Grasping

  • Visual and language-conditioned methods generate grasp poses from point clouds or connect semantic instructions to robot actions, while closed-loop vision can compensate for motion when feedback remains available.DA-GRD addresses the later case in which the visual grasp is meaningful but becomes spatially stale before execution.Source: A. Visual and Language-Conditioned… · p. 2

B. Tactile Grasping and Recovery

  • Tactile recovery includes sequential planar touch scans with a particle filter before regrasping, while other work uses tactile exploration for grasp refinement or retrieval in confined spaces.The paper adapts the touch-localization stage of Murali et al. as a fixed-scan baseline and instead starts from a previously selected task-conditioned visual grasp.Source: B. Tactile Grasping and Recovery · p. 2

C. Active Tactile Exploration

  • Prior active tactile methods measure progress in object space, whereas DA-GRD measures whether remaining uncertainty changes the downstream grasp.Prior approaches include next-best touch, active Bayesian touch, tactile grasp re-planning, and active exploration for reconstruction.Source: C. Active Tactile Exploration · p. 2

III. METHODOLOGY

DA-GRD stores the initial task-conditioned grasp and partial geometry, then uses the failed nominal grasp to initialize a weighted SE(2) belief. It repeatedly evaluates candidate recovery grasps and probes against that belief.

III. METHODOLOGY

  • An RGB-D observation and language instruction ground the desired grasp region, and AnyGrasp generates the initial world-frame grasp and object-relative task grasp for visual memory.The remembered partial geometry is stored alongside the grasp.Source: A. Visual Memory and Belief… · p. 2
  • After unobserved planar displacement and with vision disabled, the robot first tries the original grasp; its hit or miss and free-space evidence initialize the weighted SE(2) belief.Hypotheses inconsistent with the nominal attempt are removed before recovery begins.Source: A. Visual Memory and Belief… · p. 3
  • Each pose hypothesis transforms the remembered task grasp into a candidate recovery grasp, while the belief also generates probe candidates; the robot executes a sufficiently covered grasp or probes and updates the belief.Fig. 2 illustrates hit evidence supporting surface-consistent hypotheses and miss evidence removing hypotheses inconsistent with observed free space.Source: A. Visual Memory and Belief… · p. 3, Figure 2 · p. 3

B. Pose-Belief Update

Each probe updates the weighted pose belief using hit or miss evidence. The update rejects pose hypotheses inconsistent with the observed contact or swept free space and retains successors consistent with accumulated constraints.

B. Pose-Belief Update

  • For a miss, DA-GRD rejects hypotheses predicting an object intersection with the swept sensing volume, renormalizes the remaining weights, and stores observed free space for later updates.For a vertical miss, only the volume physically swept by the sensing fingers is treated as free space.Source: B. Pose-Belief Update · p. 3
  • For a hit, the measured contact point must match a point on the remembered partial geometry under a candidate planar transform, within the contact-matching tolerance.The transform uses planar translation and yaw; a 3-D point is projected onto the tabletop xy plane for contact matching.Source: B. Pose-Belief Update · p. 3
  • A successor pose hypothesis is retained only if it remains consistent with prior hits, stored free-space constraints, belief support, and the configured SE(2) support.Downward contacts may match compatible observed surfaces, while horizontal first contacts must match an object boundary; small probe-induced motions are also considered.Source: B. Pose-Belief Update · p. 3

C. Probe Candidate Generation

DA-GRD regenerates tactile probes from the updated pose belief, then ranks them by how they can reduce pose uncertainty and improve agreement on an executable grasp.

C. Probe Candidate Generation

  • After each belief update, DA-GRD generates up to 30 horizontal and 30 vertical probe candidates; horizontal scans include 24 quantile-based routes plus spatially distributed routes.Horizontal scan lines adapt to the weighted projected particle distribution, while vertical probes descend along world −Z from candidate recovery grasps.Source: C. Probe Candidate Generation · pp. 3–4

D. Grasp-Relevant Probe Selection

  • Probe ranking combines miss-exclusion mass, unexplored-surface coverage, and the gain in best grasp coverage expected from a hypothetical miss.A candidate's miss-exclusion mass is the weight of hypotheses predicting intersection with its swept volume; surface novelty discourages repeated coverage.Source: D. Grasp-Relevant Probe Selection · p. 4
  • The selected probe balances pose elimination and grasp certainty against motion time and repeated attempts.DA-GRD favors probes that explore new surface regions while avoiding unnecessarily long or repetitive motions.Source: D. Grasp-Relevant Probe Selection · p. 4

E. Grasp Coverage and Local Refinement

DA-GRD evaluates candidate grasps directly against the weighted pose belief, refining a grasp in the region of compatible hypotheses rather than estimating one average object pose.

E. Grasp Coverage and Local Refinement

  • Posterior coverage measures how much of the current belief supports a candidate grasp, using compatibility tolerances for position, yaw, height, and gripper width.Each pose hypothesis induces a possible world-frame realization of the same remembered object-relative task grasp.Source: E. Grasp Coverage and Local Refinement · p. 4
  • High best-grasp coverage means many remaining object hypotheses support a common executable grasp even when object pose remains uncertain.The method evaluates refined grasps against the current belief rather than averaging object poses into a single estimate.Source: E. Grasp Coverage and Local Refinement · pp. 4–5
  • When a mutually compatible grasp cluster contains more than 0.50 of posterior weight, its weighted center initializes a local search using 2-mm translation and approximately 2° yaw increments.The search is bounded within 30 mm and 35° of the cluster center, and selects the candidate with the highest posterior coverage.Source: E. Grasp Coverage and Local Refinement · p. 4

A. Simulation Setup

The simulation evaluates tactile recovery after hidden planar object displacement, using initial RGB-D observations and tactile contact evidence without further visual feedback.

A. Simulation Setup

  • A language instruction, initial static RGB-D observation, and shared AnyGrasp initial grasp precede a hidden planar perturbation; cameras are then disabled.Contact sensors report target-object hits and contact positions, but not ground-truth pose, perturbation, contact normal, or hidden mesh state to the planner.Source: A. Simulation Setup · p. 5
  • The primary benchmark samples translation within ±50 mm per axis and yaw within ±45°; grasp execution requires coverage of at least 0.70 of the belief.Task-conditioned compatibility uses 10-mm position and 10° yaw tolerances, with 256 belief particles by default.Source: A. Simulation Setup · p. 5
  • All methods are evaluated under the same in-support full-SE(2) perturbations on a ten-object set spanning elongated, asymmetric, and approximately symmetric geometries.The object set includes the geometries shown in Fig. 5.Source: A. Simulation Setup · p. 5, Figure 5 · p. 5

B. Baselines

DA-GRD outperforms tactile-recovery baselines on physical and task-conditioned grasp success while using fewer probes. Its grasp-coverage stopping rule trades complete pose convergence for lower interaction when remaining hypotheses support a common executable grasp.

B. Baselines

  • The original LGWS uses 15 fixed XY scans, while LGWS adapted uses 15 fixed scans with an SE(2) belief; both apply the estimated pose to the stale grasp.Stale AnyGrasp executes the initial grasp without tactile recovery.Source: B. Baselines · p. 5
  • Ours-PI stops by grasp coverage, whereas Ours-PS continues until translational and yaw dispersion fall below 4 mm and 5°.Ours-PS retains active probe selection and local grasp refinement; Ours-PI’s coverage criterion is not sensitive to the precise grasp pose.Source: B. Baselines · pp. 5–6

C. Results and Analysis

  • Task-conditioned success requires a successful lift within tolerances around the reference task-conditioned grasp; lift-only success is outside those tolerances.Physical lift success sums task-conditioned success and lift-only episodes.Source: C. Results and Analysis · p. 6
  • 84.7% physical lift success for Ours-PI exceeds 9.1% for stale AnyGrasp, 21.2% for original LGWS, and 63.7% for LGWS adapted.Ours-PS achieves 78.2%; physical lift success requires lifting the target by 5 cm.Source: C. Results and Analysis · p. 6
  • 57.3% task-conditioned success for Ours-PI and 57.6% for Ours-PS compare with 4.9% for stale AnyGrasp, 6.3% for original LGWS, and 49.2% for LGWS adapted.Task-conditioned success requires the final successful grasp to remain consistent with the initial VLM–AnyGrasp grasp.Source: C. Results and Analysis · p. 6, Table II · p. 6
  • 4.13 probes on average in lift-success episodes represents a 72.5% reduction relative to LGWS, which always executes 15 scans.The average includes task-conditioned and lift-only successes, with a successful initial grasp contributing zero probes.Source: C. Results and Analysis · p. 6, B. Baselines · p. 6
  • Ours-PI has 6.5% higher overall physical success than Ours-PS, while their task-conditioned success is similar; Ours-PI uses approximately 46.36% fewer interactions.Ours-PS performs better on several regular-geometry objects with small surface-height variation.Source: C. Results and Analysis · p. 6
  • Planar SE(2) hypotheses and projected contact consistency can confuse different 3D contacts on airplane and binoculars, while incomplete remembered point clouds can preserve ambiguity.The authors identify contact height, surface normals, or a full 3D contact model as directions to address these failure cases.Source: C. Results and Analysis · p. 6

D. Real-World Experiment

On a physical UR5 setup with no visual updates during tactile recovery, DA-GRD achieves object-specific physical and task-conditioned success rates across six objects. Probe counts vary from 2.57 to 6.67 on average, consistent with the stopping policy continuing exploration when more precise alignment is needed.

D. Real-World Experiment

  • The initial visual observation initializes the task-conditioned grasp and is not updated during tactile recovery.The physical setup used a UR5, Robotiq 2F-140 gripper, finger-mounted FSR sensors, and a wrist-mounted force–torque sensor.Source: D. Real-World Experiment · pp. 6–7
  • Physical lift success was 70% for hammer, 80% for block, 70% for tape, 80% for eraser, 60% for mouse, and 70% for game controller.Each object was tested for ten episodes; Fig. 7 reports real-world success rates by object.Source: D. Real-World Experiment · p. 7, Figure 7 · p. 7
  • Task-conditioned success was 40% for hammer, 30% for block, 40% for tape, 40% for eraser, 30% for mouse, and 50% for game controller.Source: D. Real-World Experiment · p. 7
  • Average probes ranged from 2.57 to 6.67 across objects; the hammer required 2.57, while the block required 6.67.Probe counts varied consistently with whether remaining pose hypotheses induced similar executable grasps or required more precise alignment.Source: D. Real-World Experiment · p. 7
  • Real-world success was lower and more variable than simulation; unmodeled object motion, sensing and calibration errors, friction, and geometry mismatch affected updates and execution.The authors report that the belief update, active probe selection, and grasp-aware stopping policy transferred to a physical robot using sparse tactile sensing.Source: D. Real-World Experiment · p. 7

V. CONCLUSION

DA-GRD recovers stale grasps by stopping when remaining pose hypotheses support a common executable grasp, rather than requiring full pose convergence. In MuJoCo, it achieved strong lift and task-conditioned success with fewer tactile probes than fixed-scan baselines; experiments on a physical UR5 demonstrated transfer without further visual feedback.

V. CONCLUSION

  • DA-GRD maintains a weighted SE(2) belief, selects tactile probes actively, and stops when remaining pose hypotheses sufficiently support a common executable grasp.This avoids requiring full pose convergence by evaluating uncertainty directly in grasp space.Source: §V · p. 7
  • MuJoCo trials on ten objects yielded 84.7% overall physical lift success, 57.3% task-conditioned success rate, and 4.13 tactile probes on average, 72.5% fewer than fixed 15-scan LGWS baselines.Experiments on a physical UR5 with six objects further demonstrated transfer without additional visual feedback.Source: §V · p. 7
  • The method remains limited to planar SE(2) uncertainty and partial remembered geometry, which can be insufficient for complex non-planar objects; future work targets 6-DoF motion, richer 3D contact representations, and learned probe selection.These extensions are proposed for more efficient tactile recovery.Source: §V · pp. 7–8
Loading 2609.29065v1…