PaperlayerView exact revision on arXiv

Source-linked AI summary

TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation

Kaidi Zhang, Heng Zhang, Zhengtong Xu, Zhiyuan Zhang, Md Rakibul Islam Prince, Xiang Li, Xiaojing Han, Yuhao Zhou, Arash Ajoudani, Yu She

arXiv:2603.12665v3cs.RO

TL;DR

Visual-language-action models struggle with occlusion and contact-rich manipulation because vision and language alone miss physical interaction cues. TacVLA adds tactile tokens with contact-aware gating, and experiments show clear improvements over VLA and diffusion-based policies.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have demonstrated significant advantages in robotic manipulation. However, their reliance on vision and language often leads to suboptimal performance in tasks involving visual occlusion, fine-grained manipulation, and physical contact. To address these challenges, we propose TacVLA, a fine-tuned VLA model by incorporating tactile modalities into the transformer-based policy to enhance fine-grained manipulation capabilities. Specifically, we introduce a contact-aware gating mechanism that selectively activates tactile tokens only when contact is detected, enabling adaptive multimodal fusion while avoiding irrelevant tactile interference. The fused visual, language, and tactile tokens are jointly processed within the transformer architecture to strengthen cross-modal grounding during contact-rich interaction. Extensive experiments on constraint-locked disassembly, in-box picking and robustness evaluations demonstrate that TacVLA outperforms baselines, %including existing VLA models and diffusion policies, improving the performance by averaging 20\% success rate in disassembly and 60\% in in-box picking, achieving a 2.1$\times$ improvement under visual occlusion, and showing recovery behavior under human disturbance. Videos are available at https://sites.google.com/view/tacvla.

I. INTRODUCTION

TacVLA addresses the limitations of vision-language policies in occluded and contact-rich manipulation by adding tactile sensing with adaptive multimodal fusion. It is evaluated on disassembly, in-box picking, and robustness benchmarks.

Motivation

  • Vision-language policies struggle when occlusion blocks cameras or when visual signals cannot capture contact forces, friction, and surface textures.These limitations hinder dynamic adjustment to resistance or slippage in contact-rich tasks.Source: §I · pp. 1–2

Approach

  • TacVLA integrates a compact tactile array into a transformer-based VLA policy for contact-rich manipulation.Tactile signals are represented as low-dimensional tokens for efficient multimodal processing.Source: §I · p. 2
  • A contact-aware gating mechanism activates tactile tokens only when physical contact is detected, reducing irrelevant tactile interference during non-contact phases.The mechanism enables adaptive cross-modal interaction within the transformer architecture.Source: §I · p. 2

Evaluation

  • The hardware setup uses a 7 DoF Franka robotic platform with tactile sensors and two cameras for visual input.Source: Figure 2 · p. 2
  • TacVLA is evaluated on four constraint-locked disassembly tasks, in-box picking, and robustness benchmarks involving visual occlusion and dynamic physical interactions.The disassembly tasks use diverse geometric constraints, while the evaluation also examines robustness conditions.Source: Figure 3 · p. 2, §I · p. 2

II. RELATED WORK

Related work establishes tactile sensing as complementary to vision for contact-rich manipulation, especially under visual uncertainty. TacVLA adopts a compact tactile array to provide efficient contact-state information within a VLA model.

VLA for Physical Interaction

Evaluation Scope

Tactile Sensing for Contact-rich Manipulation

C. Multimodal Fusion for Vision Language Tactile Models

Tactile information is contact-dependent and locally informative, making static fusion vulnerable to token competition and modality imbalance. TacVLA therefore arbitrates tactile participation according to contact state.

Fusion Challenge

  • Tactile signals differ from global visual observations and language tokens because they are contact-dependent and locally informative.This modality heterogeneity makes naive feature concatenation suboptimal in transformer architectures.Source: C. Multimodal Fusion for Vision… · pp. 3–4

Prior Fusion Strategies

TacVLA Fusion

  • TacVLA introduces token-level modality arbitration conditioned on contact state to selectively activate tactile representations during informative physical phases.Source: C. Multimodal Fusion for Vision… · p. 4
  • The model combines unified tokenization with contact-aware gating for adaptive grounding in contact-rich manipulation.Source: §III · p. 4

A. Overall Architecture

TacVLA extends a transformer-based VLA with modality tokenizers, a pretrained VLM backbone, an action expert, and contact-aware tactile gating. Visual, language, proprioceptive, and tactile tokens are jointly contextualized before action prediction.

Architecture

  • TacVLA consists of modality tokenizers, a pretrained VLM backbone, an action expert, and a contact-aware gating module.Source: A. Overall Architecture · p. 4

Tokenization

  • Front and wrist-camera images produce visual tokens, while language instructions and robot proprioception are tokenized for the policy.Source: A. Overall Architecture · p. 4
  • A 15 × 8 tactile array is encoded by a lightweight MLP into 36 tactile tokens with fixed 2D sine-cosine positional embeddings.Source: A. Overall Architecture · p. 4

Multimodal Processing

  • All modality tokens are concatenated into a shared sequence and processed with non-causal attention so vision, language, and touch can cross-attend.Source: A. Overall Architecture · p. 4

Action Prediction

  • The fused representation is provided as a prefix to an action expert that predicts continuous action sequences using a flow-matching objective.Source: A. Overall Architecture · p. 4

B. Contact-aware Gating Tactile Fusion

TacVLA uses contact-aware gating to keep tactile tokens inactive during non-contact phases and activate them only when physical contact is detected. This preserves a fixed token structure while restricting tactile influence to physically informative moments.

C. Training Procedure

TacVLA is trained and evaluated on a real-world contact-rich manipulation platform using synchronized multimodal data and consistent fine-tuning settings.

  • TacVLA is fine-tuned with LoRA on an OpenPI pi05 base checkpoint for 10,000 gradient steps.Source: C. Training Procedure · p. 5
  • Robustness evaluations test visual occlusion and human disturbance alongside disassembly and in-box picking tasks.Source: §IV · p. 5
  • The platform uses a 7-DoF Franka Emika Panda arm, two RGB cameras, and a finger-mounted tactile array.Source: A. Experimental Setup · p. 5
  • Evaluation covers contact-rich constraint-locked disassembly tasks requiring pressing, twisting, or sliding motions.Source: 1) Disassembly · p. 5

2) In-box Picking:

The in-box picking task tests retrieval under severe visual occlusion, while the broader evaluation compares TacVLA with Pi0.5 and diffusion-policy baselines across manipulation tasks.

2) In-box Picking:

TacVLA improves in-box picking under severe visual occlusion and outperforms tactile diffusion-policy baselines, while contact-aware gating supports stable interaction behavior.

  • The finetuned Pi0.5 baseline achieves only 10% success in the severely occluded in-box picking task.Source: 2) In-box Picking · p. 6
  • TacVLA achieves 70% success under heavy occlusion and proceeds to lifting only after physical contact is detected.Source: 2) In-box Picking · p. 6
  • TacVLA’s contact verification during exploration improves robustness under visual occlusion and uncertain physical interaction.Source: 2) In-box Picking · p. 6
  • TacVLA reaches 83.75% average disassembly success, compared with 48.75% for diffusion policy and 31.25% for 3D diffusion policy.Source: 3) Benchmark with Diffusion Policy · p. 6
  • Under severe occlusion, diffusion policies achieve 0% and 5% success and often struggle with localization, stable contact, and consistent trajectories.Source: 3) Benchmark with Diffusion Policy · p. 6
  • Removing contact-aware gating reduces average success by approximately 25% under severe visual occlusion.Source: Table III · p. 7, Figure 7 · p. 7

D. Robustness Evaluation

TacVLA is evaluated under severe visual occlusion and human disturbance to test robustness when visual feedback is unavailable or task states change unexpectedly. It maintains higher disassembly success under camera blocking and recovers from an object being returned to the box.

Evaluation Setup

  • The robustness evaluation targets contact-rich manipulation under camera occlusion and dynamic environmental changes.These tests challenge performance when reliable visual feedback is unavailable and when the manipulated object is unexpectedly repositioned.Source: D. Robustness Evaluation · p. 7, 1) Block Camera · p. 7

Block Camera

  • TacVLA raises average disassembly success under visual occlusion from approximately 30% to over 60%.Across four tasks, TacVLA reaches 70%, 65%, 45%, and 70%, compared with 40%, 40%, 5%, and 35% for finetuned Pi0.5.Source: 1) Block Camera · p. 7
  • Under severe visual occlusion, TacVLA outperforms the vision-only baseline on all four disassembly tasks.The largest task-level improvement is +40% on Task 3.Source: 1) Block Camera · p. 7

Human Disturbance

  • During human disturbance, TacVLA detects that the object was returned to the box, re-grasps it, and continues execution.The disturbance tests state-change detection, strategy adaptation, and recovery from external perturbations.Source: 1) Block Camera · p. 7

E. Ablation Study

The ablation study shows that contact-aware gating is important for using tactile feedback effectively. Removing gating lowers performance across disassembly, in-box picking, and severe visual-occlusion evaluations, with qualitative failures before stable contact.

Disassembly

  • Removing gating reduces average disassembly success from 83.75% to 71.25%.On Task 3, the no-gating variant reaches 60% versus 65% for Pi0.5 without tactile input.Source: 1) Disassembly · p. 7

Qualitative Failures

  • The no-gating variant exhibits misalignment, repeated re-grasp attempts, stalled states, and failed lifts during object approach.These behaviors are consistent with tactile-token injection interfering with visual localization before stable contact.Source: 1) Disassembly · p. 7

In-Box Picking

  • Without gating, the in-box picking success rate drops from 70% to 40%.Failure cases commonly involve repeated grasp attempts without successful completion.Source: 2) In-box Picking Tasks · p. 7

Interpretation

  • Contact-aware gating enables a more stable transition from visual localization to physical interaction under degraded visual conditions.The ablation attributes the limitation of unconditional tactile fusion to ineffective use of contact-state information.Source: 2) In-box Picking Tasks · p. 7

2) Human Disturbance:

The visual-occlusion and broader robustness results support TacVLA’s use of tactile sensing and state-dependent fusion for contact-rich manipulation. The paper also identifies limits in the gating heuristic, tactile resolution, and task scope.

Block Camera

  • Under severe visual occlusion, removing gating reduces average success by approximately 25% compared with TacVLA.This result highlights the importance of state-dependent tactile activation when visual information is unreliable.Source: 3) Block Camera on Disassembly · pp. 7–8
  • Contact-aware gating is reported as critical for leveraging tactile feedback when visual information is unreliable.Continuous tactile-token participation is insufficient to compensate for visual degradation without state-dependent activation.Source: 3) Block Camera on Disassembly · p. 8

Limitations

  • TacVLA’s limitations include binary threshold gating, limited tactile spatial resolution, and evaluation focused on short-horizon contact-centric tasks.The paper identifies learnable modality weighting, finer contact-geometry reasoning, and longer-horizon evaluation as future directions.Source: §V · p. 8

Conclusion

  • TacVLA improves performance across disassembly, in-box picking, and robustness evaluations over pure VLA and diffusion-based policies.The conclusion summarizes clear improvements from tactile augmentation and contact-aware gating.Source: §VI · p. 8

Related papers

Loading 2603.12665v3…