PaperlayerView exact revision on arXiv

Source-linked AI summary

AgenticTactileVLA: Contact-Guided Execution-Time Supervision for Generalizable Dexterous Manipulation without VLA Retraining

Elizaveta Semenyakina, Ivan Snegirev, Mikhail Kiselev, Miguel Altamirano Cabrera, Artem Lykov, Hajira Amjad, Dzmitry Tsetserukou

arXiv:2610.04391v1cs.RO

TL;DR

Objects with similar grasp strategies can require different physical contacts, while visual feedback degrades during closure. AgenticTactileVLA adds contact-guided supervision over a fixed VLA; on five held-out objects, it outperforms both the base VLA and unconditional close-to-stall control.

  • Problem

    Objects compatible with the same broad grasp can differ in geometry and compliance, changing required contacts while closure occludes visual feedback.

    Source: §I · p. 1

  • Method

    A bounded execution supervisor uses finger-position and motor-effort feedback to adapt a fixed VLA’s grasp realization without additional tactile sensors or object-specific VLA retraining.

    Source: §I · p. 2, §VI · p. 8

  • Results

    84.0% completion for the supervisor versus 61.3% for the base VLA and 72.0% for unconditional close-to-stall control on five held-out objects.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The results support execution-time adaptation toward more scalable object-level generalization of fixed VLA policies without object-specific retraining or additional tactile hardware.

    Source: §VI · p. 8

    The support predicate is conservative, especially for compliant objects, and can reject configurations that would still retain the object.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-language-action policies may predict a transferable manipulation strategy yet fail to realize it reliably on the encountered object: objects compatible with the same grasp differ in geometry and compliance, and visual feedback degrades under closure occlusion. AgenticTactileVLA is presented as an execution-time supervisor that shifts part of object-specific adaptation from prediction to physical interaction. A fixed VLA provides the approach and hand targets; the supervisor decides whether to remain transparent, refine finger flexion, retain or release the corrected configuration, return control to the VLA for retry, or select a compliant hand-control regime. It uses finger-position and motor-effort feedback as proprioceptive contact evidence and requires neither tactile sensors nor VLA retraining. On a Unitree G1 with a BrainCo Revo2 hand, a randomized matched-block evaluation on five objects held out from VLA training yields 61.3% completion for the base VLA, 72.0% for unconditional close-to-stall control, and 84.0% for the supervisor under a shared budget; the gain is positive on every object and persists under moderate pose perturbations. Ablations show the gain is not explained by extended closure alone, and that selective triggering reduces correction episodes by 65.7% with no detected change in completion. A retention audit shows acceptance predicts retention in 88.9% of held-out cases, while compliant objects expose conservative false rejection. A thin-walled-cup study demonstrates contextual routing to compliant control, matching an always-compliant reference. These results suggest that contact-guided execution-time adaptation can improve the object-level generalization of a fixed VLA to held-out objects by adapting physical realization without object-specific retraining.

I. INTRODUCTION

VLA grasp strategies can be transferable yet fail at physical execution as object geometry, compliance, and closure occlusion complicate contact; this paper asks whether a fixed VLA’s physical realization can be adapted during execution without object-specific retraining.

I. INTRODUCTION

  • Objects compatible with the same broad grasp can differ in geometry and compliance, changing required contacts and interaction conditions while closure occludes visual feedback.A suitable approach may still leave fingers unsupported, produce an unsuitable terminal configuration, or over-compress a fragile object.Source: §I · p. 1
  • The supervisor assesses physical support at nominal closure, selectively corrects unsupported fingers, retains accepted corrections, or releases the override for a VLA-generated retry.Whole-body stabilization remains active while body-reference progression is paused.Source: §I · p. 1
  • The evaluation compares the supervisor with unconditional close-to-stall control under a shared execution budget and uses component ablations to separate selective supervision from simply closing farther or longer.The stated evaluation also includes pose perturbations, retention diagnostics, and fragile-object interaction tests.Source: §I · p. 2
  • For fragile objects, the system can select compliant rather than nominal position control before closure, using explicit handling cues or image and task context.The VLM selects only the interaction mode, not controller parameters or material estimates.Source: §I · p. 2
  • AgenticTactileVLA adds bounded, stateful execution supervision to a fixed VLA, adapting finger references, grasp decisions, retries, and interaction mode without tactile sensors or object-specific retraining.The intended scope is adapting feasible policy-generated strategies, not recovering from arbitrary incompatible approaches.Source: §I · p. 2

A. Contact Sensing from Motor Feedback

The paper places contact evidence and physical adaptation in an external execution layer over a fixed VLA, linking grasp assessment to finger correction, retention or retry, and interaction control.

B. Tactile Feedback for VLA Adaptation

  • Unlike tactile-conditioned VLA approaches that refine action generation, AgenticTactileVLA leaves the VLA fixed and places adaptation in a nonlearned external layer.The supervisor remains transparent when the policy-generated closure already provides sufficient support.Source: B. Tactile Feedback for VLA Adaptation · p. 2

C. Execution Supervision and Interaction Control

  • The supervisor handles closure-to-motion decisions—finger correction, grasp retention or release, and VLA retry—while a separate layer selects nominal or compliant control before closure.The policy continues to provide the manipulation strategy.Source: C. Execution Supervision and… · p. 2

III. METHOD

  • The fixed VLA generates body-motion tokens and hand targets; the supervisor monitors their physical realization and can pause body-reference progression while whole-body stabilization remains active.When the grasp is sufficiently realized, execution remains unchanged.Source: §III · p. 2, Figure 2 · p. 3

A. Contact Sensing from Motor Feedback

A. System Overview and Execution Supervisor

  • Grasp assessment follows the commanded closure’s terminal phase; insufficient support pauses body-reference progression for local correction, with accepted corrections retained and rejected ones returning authority to the VLA.A fresh observation conditions the next policy chunk after authority transitions.Source: A. System Overview and Execution… · pp. 3–4

B. Selective Intervention

Selective intervention transfers hand execution to the supervisor only at the commanded grasp’s terminal assessment point when accumulated support evidence is insufficient.

B. Selective Intervention

  • Intervention is evaluated only at the grasp-assessment point; missing contact before the VLA reaches its terminal grasp phase does not trigger correction.The point follows the terminal phase of the commanded grasp.Source: B. Selective Intervention · p. 4
  • When intervention is triggered, η_t transfers hand execution from the nominal VLA command to the execution agent.The transfer follows insufficient support at terminal assessment.Source: B. Selective Intervention · p. 4
  • Position-support evidence is accumulated for monitored flexion channels, and at least three support latches allow nominal execution to continue.Otherwise, the supervisor enters adjustment and overrides flexion targets; the threshold is an operational rule, not a guarantee of grasp stability.Source: B. Selective Intervention · p. 4

C. Proprioceptive Contact Estimation

The detector estimates contact from proprioceptive finger-position and motor-effort signals. It combines command–response mismatch with effort evidence, but highly compliant objects can remain difficult to detect.

C. Proprioceptive Contact Estimation

  • Commanded-versus-measured finger motion is the main resistance cue, with motor effort providing complementary evidence during adjustment.When commands continue advancing but measured motion is substantially smaller, the finger is interpreted as encountering resistance.Source: C. Proprioceptive Contact Estimation · p. 4
  • Support latches when position-based stall or effort-assisted detection activates; reopening resets latches, and missed activation can reflect compliance or insufficient excitation.Effort-assisted detections accumulate only during adjustment.Source: C. Proprioceptive Contact Estimation · p. 4
  • Detector thresholds were calibrated on 40 closures across five training objects, while held-out objects and evaluation trials were excluded from tuning.Thresholds, preload, timing parameters, and controller gains were frozen before evaluation.Source: C. Proprioceptive Contact Estimation · p. 4
  • Highly compliant objects remain challenging because deformation can preserve command tracking despite physical contact, weakening the mismatch required for detection.Source: C. Proprioceptive Contact Estimation · p. 4

D. Object-Adaptive Finger Correction and Acceptance

The supervisor refines unsupported fingers independently, then uses total support to retain a corrected grasp or return control to the VLA for another approach.

D. Object-Adaptive Finger Correction and Acceptance

  • Independent finger stopping turns the proposed terminal hand configuration into object-dependent references while preserving the policy-selected thumb-opposition target.Supported fingers stop at measured position plus a normalized preload of 0.02; unsupported fingers advance in increments of 0.004 every 20 ms.Source: D. Object-Adaptive Finger Correction… · p. 5
  • Adjustment lasts 0.6–4.0 seconds and ends when monitored fingers latch support or exhaust permitted travel, or the timeout is reached.Source: D. Object-Adaptive Finger Correction… · p. 5
  • The verdict reflects total support: FORMED retains the corrected configuration, while EMPTY or CROOKED releases the override so the VLA can reopen and retry.CROOKED means insufficient support evidence, not geometric misalignment; releasing the override does not itself command reopening.Source: D. Object-Adaptive Finger Correction… · p. 5
  • Re-entry is blocked until reopening, so each closure has at most one correction episode and retries consume the shared trial budget.Rearming requires commanded and measured mean flexion to remain at least 0.05 below terminal-assessment values for 0.12 seconds.Source: D. Object-Adaptive Finger Correction… · p. 5

E. Language- and Vision-Conditioned Mode Selection

The system chooses nominal or compliant control before closure, using explicit handling cues or image-and-task context when cues are absent.

E. Language- and Vision-Conditioned Mode Selection

  • Explicit cues such as “gently”, “carefully”, or “fragile” activate compliant interaction mode.Source: E. Language- and Vision-Conditioned… · p. 5
  • Without an explicit cue, Qwen2.5-VL-7B-Instruct selects between two predefined modes from image and task context, with one cached decision per grasp episode.Language cues take priority, and missing, late, or malformed VLM responses default to nominal control.Source: E. Language- and Vision-Conditioned… · p. 5
  • The VLM selects only the interaction mode, not material properties or controller parameters, and selection occurs before closure because fragile objects may deform before resistance is detectable.Source: E. Language- and Vision-Conditioned… · p. 5
  • For the mode-selection experiment, every condition used the same VLA instruction, while handling cues were reserved for the interaction parser.The VLM received a fixed image-and-task prompt without handling cues.Source: E. Language- and Vision-Conditioned… · p. 5

F. Compliant Control via Current-Limited Interaction

Compliant interaction uses a current-limited controller, while the experiments examine supervision, closure effects, pose sensitivity, contact verification, and fragile-object mode selection.

F. Compliant Control via Current-Limited Interaction

  • The paper identifies normalized hand-interface positions and smoothed finger velocity as controller inputs, while actuator command is device-level rather than calibrated motor current.Source: F. Compliant Control via… · p. 5
  • The compliant controller runs every 20 ms, bounds its actuator command by Imax, and drives fingers toward a predefined grasp configuration.Kp = 5.0 and Kd = 0.05 are controller-interface gains, not physical stiffness or damping coefficients; limits of 0.15 and 0.20 are not calibrated current or fingertip force.Source: F. Compliant Control via… · p. 5
  • Compliant closure and holding are supported, but autonomous transport is not; fragile-object experiments assess integrity during closure and holding only.Compliant mode is activated before closure to avoid an initial high-effort squeeze from nominal position control.Source: F. Compliant Control via… · p. 5

IV. EXPERIMENTAL RESULTS

  • The evaluation examines task-level supervision benefit, unconditional finger closure, initial-pose perturbations, proprioceptive verification, and contextual routing for fragile objects.Source: §IV · p. 5

A. Experimental Protocol

The evaluation uses a fixed VLA and five held-out objects in randomized, interleaved matched blocks under a shared execution budget. The supervisor achieves the highest completion, outperforming both the base VLA and unconditional close-to-stall control.

A. Experimental Protocol

  • The five test objects were excluded from VLA training, detector calibration, and controller development; trials used randomized, interleaved matched blocks and a shared budget.A trial ended at successful placement, 60 seconds, or three arm approaches, whichever came first; no started trial was excluded.Source: A. Experimental Protocol · pp. 5–6

B. Main Comparison

  • 84.0% completion with the supervisor exceeds 61.3% for the base VLA and 72.0% for unconditional close-to-stall control.The matched-block comparisons favor the supervisor over both alternatives after Holm correction.Source: Table II · p. 6, B. Main Comparison · p. 6
  • Unconditional closure exceeds the base VLA in 12/4 discordant blocks, but does not significantly outperform it; added closure alone does not explain the supervisor’s gain.The supervisor intervenes in 27/75 trials, compared with 101 extension episodes for unconditional closure across the same 75 trials.Source: B. Main Comparison · p. 6
  • The supervisor gains 3–4 successful trials over the base VLA and 1–2 over unconditional closure for each of the five objects.The primary comparison covers 75 matched blocks, with 15 trials per held-out object.Source: B. Main Comparison · p. 6

C. Component Ablation

The ablations favor the complete selective supervisor over position-only stall control, while pose perturbations preserve its advantage within the tested range. Proprioceptive acceptance is associated with retention, but compliant objects reveal conservative false rejection.

C. Component Ablation

  • 86% completion with S-full exceeds 72% for position-only stall control, indicating that extended closure alone does not explain the gain.The difference is significant after Holm correction (p = 0.047).Source: C. Component Ablation · p. 6, Table III · p. 6
  • Selective supervision reduces correction episodes from 134 to 46, a 65.7% reduction, with no detected completion difference from U-full.Completion is 84% for U-full and 86% for S-full; p = 0.804.Source: C. Component Ablation · p. 6
  • Removing effort evidence reduces completion to 79%, and disabling verdict-dependent release reduces it to 77%, but neither individual contrast is statistically resolved.The corrected p-values are 0.378 and 0.235, respectively.Source: C. Component Ablation · p. 6

D. Pose-Perturbation Evaluation

  • The supervisor remains strongest under nominal, translated, and rotated poses, with gains over the base VLA of 22, 28, and 28 percentage points, respectively.The tested perturbations were translations within 20 mm and yaw changes within 15°; pooled, the supervisor exceeds unconditional closure in 26/6 discordant blocks (p < 0.001).Source: D. Pose-Perturbation Evaluation · p. 7, Table IV · p. 7

E. Proprioceptive Verification and Failure Modes

  • Soft-ball deformation can mask the command–response mismatch used as contact evidence, producing false rejection even when video confirms physical enclosure.Five of six such trials eventually completed, including two after a policy-generated retry; diagnostic verdicts are not contact-detection accuracy because independent per-finger labels are unavailable.Source: E. Proprioceptive Verification and… · p. 7
  • On held-out objects, accepted configurations retain the object in 88.9% of cases, while 25.6% of retainable configurations are falsely rejected.The audit indicates acceptance is stronger evidence of a useful grasp than rejection is evidence of an unusable one.Source: E. Proprioceptive Verification and… · p. 7, Table V · p. 7

F. Contextual Interaction-Mode Selection

The study tests whether contextual selection of a compliant interaction mode can preserve fragile cups while retaining nominal control for rigid objects. Routing outcomes are compared with nominal and always-compliant control, but the cup is a training object.

F. Contextual Interaction-Mode Selection

  • Instruction-based and combined routing preserve 5/5 cups, while VLM-only routing preserves 4/5 and nominal position control damages all five.In the remaining VLM-only trial, compliant mode was selected after closure began; Fig. 4 contrasts damage under nominal control with preserved integrity under compliant control.Source: F. Contextual Interaction-Mode Selection · p. 7, Figure 4 · p. 8, Table VI · p. 8
  • Always-compliant control also preserves 5/5 cups, so the cup study shows contextual routing works but not that it beats always-compliant execution.The cup is a training object, so this result does not establish generalization to held-out fragile objects.Source: F. Contextual Interaction-Mode Selection · p. 8
  • On three additional rigid objects, intended nominal mode is selected in 14/15 trials, demonstrating routing between predefined regimes but not a general material classifier.The evaluated modes include position control, instruction-based routing, VLM-based routing, combined routing, and always-compliant control.Source: F. Contextual Interaction-Mode Selection · pp. 7–8

V. DISCUSSION AND LIMITATIONS

The discussion reports gains from execution-time supervision alongside selective intervention and contact-support audits, while noting limits in the evidence and recovery scope. The conclusion frames this approach as adaptation of a fixed VLA’s physical realization without object-specific retraining.

V. DISCUSSION AND LIMITATIONS

  • Selective intervention reduces correction episodes by 65.7% with no detected completion change, and the benefit persists under moderate pose variation.The comparison is to the always-on controller.Source: §V · p. 8
  • Accepted configurations retain objects in 70/76 cases (92.1% pooled; 88.9% on held-out objects), while 18/44 rejected configurations would still retain them.The support predicate is conservative, especially for compliant objects, and false rejections can be recovered through policy-mediated retry.Source: §V · p. 8
  • Instruction-based and combined routing each preserved 5/5 fragile cups; VLM-only routing preserved 4/5 due to late mode selection, while intended nominal mode was selected in 14/15 rigid-object trials.The fragile-object study uses a training object.Source: §V · p. 8
  • The ablation does not isolate effort evidence from verdict-dependent release, and the supervisor uses proprioception rather than calibrated force and cannot recover arbitrary approach errors.Validation on unseen objects and diverse embodiments remains necessary.Source: §V · p. 8

VI. CONCLUSION

  • AgenticTactileVLA preserves the predicted strategy while adapting physical realization through selective intervention, finger correction, grasp acceptance or retry, and interaction-mode selection.The conclusion presents execution-time adaptation as a complement to a reusable fixed VLA without object-specific retraining or additional tactile hardware.Source: §VI · p. 8, §V · p. 8
  • 84.0% task completion on five held-out objects exceeds 61.3% for the base VLA and 72.0% for unconditional close-to-stall control.The gain is positive on every held-out object, persists under pose perturbations, and additional closure explains only part of the improvement.Source: §VI · p. 8, §V · p. 8
Loading 2610.04391v1…