Source-linked AI summary
TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation
Kaidi Zhang, Heng Zhang, Zhengtong Xu, Zhiyuan Zhang, Md Rakibul Islam Prince, Xiang Li, Xiaojing Han, Yuhao Zhou, Arash Ajoudani, Yu She
TL;DR
Visual-language-action models struggle with occlusion and contact-rich manipulation because vision and language alone miss physical interaction cues. TacVLA adds tactile tokens with contact-aware gating, and experiments show clear improvements over VLA and diffusion-based policies.
Problem
VLA models can struggle with visual occlusion and fine-grained contact-rich manipulation because visual signals miss physical properties such as contact forces, friction, and surface textures.
Method
TacVLA integrates low-dimensional tactile tokens into a transformer-based VLA policy and uses contact-aware gating to activate them only when physical contact is detected.
Results
TacVLA achieves 83.75% average success on disassembly and 70% success on in-box picking, outperforming VLA and diffusion-based policies.
Takeaways & Limitations
State-dependent fusion of vision, language, and touch supports robust cross-modal grounding for contact-rich manipulation.
Takeaways & Limitations
The evaluation focuses on short-horizon, contact-centric tasks, while the tactile sensor has limited spatial resolution and gating uses a binary threshold heuristic.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have demonstrated significant advantages in robotic manipulation. However, their reliance on vision and language often leads to suboptimal performance in tasks involving visual occlusion, fine-grained manipulation, and physical contact. To address these challenges, we propose TacVLA, a fine-tuned VLA model by incorporating tactile modalities into the transformer-based policy to enhance fine-grained manipulation capabilities. Specifically, we introduce a contact-aware gating mechanism that selectively activates tactile tokens only when contact is detected, enabling adaptive multimodal fusion while avoiding irrelevant tactile interference. The fused visual, language, and tactile tokens are jointly processed within the transformer architecture to strengthen cross-modal grounding during contact-rich interaction. Extensive experiments on constraint-locked disassembly, in-box picking and robustness evaluations demonstrate that TacVLA outperforms baselines, %including existing VLA models and diffusion policies, improving the performance by averaging 20\% success rate in disassembly and 60\% in in-box picking, achieving a 2.1$\times$ improvement under visual occlusion, and showing recovery behavior under human disturbance. Videos are available at https://sites.google.com/view/tacvla.
I. INTRODUCTION
TacVLA addresses the limitations of vision-language policies in occluded and contact-rich manipulation by adding tactile sensing with adaptive multimodal fusion. It is evaluated on disassembly, in-box picking, and robustness benchmarks.
- Motivation: Vision-language policies struggle when occlusion blocks cameras or when visual signals cannot capture contact forces, friction, and surface textures.These limitations hinder dynamic adjustment to resistance or slippage in contact-rich tasks.
- Approach: TacVLA integrates a compact tactile array into a transformer-based VLA policy for contact-rich manipulation.Tactile signals are represented as low-dimensional tokens for efficient multimodal processing.
- Approach: A contact-aware gating mechanism activates tactile tokens only when physical contact is detected, reducing irrelevant tactile interference during non-contact phases.The mechanism enables adaptive cross-modal interaction within the transformer architecture.
- Evaluation: The hardware setup uses a 7 DoF Franka robotic platform with tactile sensors and two cameras for visual input.
- Evaluation: TacVLA is evaluated on four constraint-locked disassembly tasks, in-box picking, and robustness benchmarks involving visual occlusion and dynamic physical interactions.The disassembly tasks use diverse geometric constraints, while the evaluation also examines robustness conditions.
II. RELATED WORK
Related work establishes tactile sensing as complementary to vision for contact-rich manipulation, especially under visual uncertainty. TacVLA adopts a compact tactile array to provide efficient contact-state information within a VLA model.
- VLA for Physical Interaction: VLA models integrate visual perception, language understanding, and action generation to support generalized robotic manipulation.
- Evaluation Scope: The experiments target contact-rich fine-grained manipulation and robustness to visual occlusion in disassembly and in-box picking.
- Tactile Sensing for Contact-rich Manipulation: Tactile sensing complements vision by providing contact mechanics, pressure distribution, local geometry, and slip-related feedback.These signals help detect pose, alignment, and contact-transition errors during physical interaction.
- Tactile Sensing for Contact-rich Manipulation: High-resolution vision-based tactile sensors such as GelSight provide detailed contact observations but introduce additional sensing complexity.
- Tactile Sensing for Contact-rich Manipulation: Visuotactile policies have improved robustness under visual uncertainty and during fine-grained contact phases.
- Tactile Sensing for Contact-rich Manipulation: TacVLA uses a compact tactile array that provides efficient contact-state information for integration into vision-language-action models.
C. Multimodal Fusion for Vision Language Tactile Models
Tactile information is contact-dependent and locally informative, making static fusion vulnerable to token competition and modality imbalance. TacVLA therefore arbitrates tactile participation according to contact state.
- Fusion Challenge: Tactile signals differ from global visual observations and language tokens because they are contact-dependent and locally informative.This modality heterogeneity makes naive feature concatenation suboptimal in transformer architectures.
- Prior Fusion Strategies: Most prior structured fusion methods use static mechanisms that keep tactile inputs continuously active throughout trajectories.
- TacVLA Fusion: TacVLA introduces token-level modality arbitration conditioned on contact state to selectively activate tactile representations during informative physical phases.
- TacVLA Fusion: The model combines unified tokenization with contact-aware gating for adaptive grounding in contact-rich manipulation.
A. Overall Architecture
TacVLA extends a transformer-based VLA with modality tokenizers, a pretrained VLM backbone, an action expert, and contact-aware tactile gating. Visual, language, proprioceptive, and tactile tokens are jointly contextualized before action prediction.
- Architecture: TacVLA consists of modality tokenizers, a pretrained VLM backbone, an action expert, and a contact-aware gating module.
- Tokenization: Front and wrist-camera images produce visual tokens, while language instructions and robot proprioception are tokenized for the policy.
- Tokenization: A 15 × 8 tactile array is encoded by a lightweight MLP into 36 tactile tokens with fixed 2D sine-cosine positional embeddings.
- Multimodal Processing: All modality tokens are concatenated into a shared sequence and processed with non-causal attention so vision, language, and touch can cross-attend.
- Action Prediction: The fused representation is provided as a prefix to an action expert that predicts continuous action sequences using a flow-matching objective.
B. Contact-aware Gating Tactile Fusion
TacVLA uses contact-aware gating to keep tactile tokens inactive during non-contact phases and activate them only when physical contact is detected. This preserves a fixed token structure while restricting tactile influence to physically informative moments.
- Naively concatenating tactile tokens throughout a trajectory can introduce unnecessary attention tokens during non-contact phases.
- Contact is detected when the number of taxels exceeding a predefined pressure threshold surpasses a fixed count.
- The contact flag controls attention masking and embedding gating for tactile tokens.
- When contact is absent, tactile tokens are excluded from attention and zeroed to reduce pre-contact sensor-noise influence.
- Fixed token topology with state-dependent routing lets tactile information influence policy decisions only when physically informative.
C. Training Procedure
TacVLA is trained and evaluated on a real-world contact-rich manipulation platform using synchronized multimodal data and consistent fine-tuning settings.
- TacVLA is fine-tuned with LoRA on an OpenPI pi05 base checkpoint for 10,000 gradient steps.
- Robustness evaluations test visual occlusion and human disturbance alongside disassembly and in-box picking tasks.
- The platform uses a 7-DoF Franka Emika Panda arm, two RGB cameras, and a finger-mounted tactile array.
- Evaluation covers contact-rich constraint-locked disassembly tasks requiring pressing, twisting, or sliding motions.
2) In-box Picking:
The in-box picking task tests retrieval under severe visual occlusion, while the broader evaluation compares TacVLA with Pi0.5 and diffusion-policy baselines across manipulation tasks.
- The front camera cannot see inside the box, while the wrist camera has limited illumination and occlusion.
- Diffusion policies use multiview images and the same tactile measurements, with modality features concatenated to condition action generation.
- The evaluation dataset contains four disassembly tasks and one in-box picking task with human teleoperation demonstrations.
- TacVLA achieves an average success rate of 83.75%, versus 63.75% for the finetuned Pi0.5 baseline.
- Robustness figures compare TacVLA against finetuned Pi0.5 and tactile input without gating under camera disturbance.
2) In-box Picking:
TacVLA improves in-box picking under severe visual occlusion and outperforms tactile diffusion-policy baselines, while contact-aware gating supports stable interaction behavior.
- The finetuned Pi0.5 baseline achieves only 10% success in the severely occluded in-box picking task.
- TacVLA achieves 70% success under heavy occlusion and proceeds to lifting only after physical contact is detected.
- TacVLA’s contact verification during exploration improves robustness under visual occlusion and uncertain physical interaction.
- TacVLA reaches 83.75% average disassembly success, compared with 48.75% for diffusion policy and 31.25% for 3D diffusion policy.
- Under severe occlusion, diffusion policies achieve 0% and 5% success and often struggle with localization, stable contact, and consistent trajectories.
- Removing contact-aware gating reduces average success by approximately 25% under severe visual occlusion.
D. Robustness Evaluation
TacVLA is evaluated under severe visual occlusion and human disturbance to test robustness when visual feedback is unavailable or task states change unexpectedly. It maintains higher disassembly success under camera blocking and recovers from an object being returned to the box.
- Evaluation Setup: The robustness evaluation targets contact-rich manipulation under camera occlusion and dynamic environmental changes.These tests challenge performance when reliable visual feedback is unavailable and when the manipulated object is unexpectedly repositioned.
- Block Camera: TacVLA raises average disassembly success under visual occlusion from approximately 30% to over 60%.Across four tasks, TacVLA reaches 70%, 65%, 45%, and 70%, compared with 40%, 40%, 5%, and 35% for finetuned Pi0.5.
- Block Camera: Under severe visual occlusion, TacVLA outperforms the vision-only baseline on all four disassembly tasks.The largest task-level improvement is +40% on Task 3.
- Human Disturbance: During human disturbance, TacVLA detects that the object was returned to the box, re-grasps it, and continues execution.The disturbance tests state-change detection, strategy adaptation, and recovery from external perturbations.
E. Ablation Study
The ablation study shows that contact-aware gating is important for using tactile feedback effectively. Removing gating lowers performance across disassembly, in-box picking, and severe visual-occlusion evaluations, with qualitative failures before stable contact.
- Disassembly: Removing gating reduces average disassembly success from 83.75% to 71.25%.On Task 3, the no-gating variant reaches 60% versus 65% for Pi0.5 without tactile input.
- Qualitative Failures: The no-gating variant exhibits misalignment, repeated re-grasp attempts, stalled states, and failed lifts during object approach.These behaviors are consistent with tactile-token injection interfering with visual localization before stable contact.
- In-Box Picking: Without gating, the in-box picking success rate drops from 70% to 40%.Failure cases commonly involve repeated grasp attempts without successful completion.
- Interpretation: Contact-aware gating enables a more stable transition from visual localization to physical interaction under degraded visual conditions.The ablation attributes the limitation of unconditional tactile fusion to ineffective use of contact-state information.
2) Human Disturbance:
The visual-occlusion and broader robustness results support TacVLA’s use of tactile sensing and state-dependent fusion for contact-rich manipulation. The paper also identifies limits in the gating heuristic, tactile resolution, and task scope.
- Block Camera: Under severe visual occlusion, removing gating reduces average success by approximately 25% compared with TacVLA.This result highlights the importance of state-dependent tactile activation when visual information is unreliable.
- Block Camera: Contact-aware gating is reported as critical for leveraging tactile feedback when visual information is unreliable.Continuous tactile-token participation is insufficient to compensate for visual degradation without state-dependent activation.
- Limitations: TacVLA’s limitations include binary threshold gating, limited tactile spatial resolution, and evaluation focused on short-horizon contact-centric tasks.The paper identifies learnable modality weighting, finer contact-geometry reasoning, and longer-horizon evaluation as future directions.
- Conclusion: TacVLA improves performance across disassembly, in-box picking, and robustness evaluations over pure VLA and diffusion-based policies.The conclusion summarizes clear improvements from tactile augmentation and contact-aware gating.