Source-linked AI summary
AT-VLA: Adaptive Tactile Injection for Enhanced Feedback Reaction in Vision-Language-Action Models
Xiaoqi Li, Muhe Cai, Jiadong Xu, Juan Zhu, Hongwei Fan, Yan Shen, Guangrui Ren, Hao Dong
TL;DR
Contact-rich manipulation requires physical-interaction feedback that VLAs may overlook, but adding tactile input can interfere with pretrained representations and slow inference limits rapid response. AT-VLA injects tactile signals adaptively and pairs slow visual-language reasoning with fast tactile control. The authors report superior real-world contact-rich manipulation performance and closed-loop response within 0.04 s.
Problem
VLAs can overlook tactile feedback needed for precise contact-rich manipulation, while adding this rarely pretrained modality may disrupt pretrained capabilities and slow inference limits timely response.
Source: §1 · p. 2Method
AT-VLA uses a learnable tactile gate with adaptive cross-attention and a dual-stream design that separates slow visual-language processing from fast tactile control.
Source: §1 · p. 2Results
Takeaways & Limitations
AT-VLA maintains performance across sensory conditions, including real-world circumstances where tactile signals may be unstable or unavailable.
Source: §1 · p. 2On the Unscrew Lid task, AT-VLA performs slightly worse than VTLA and RDP; unlike those baselines, it does not always ensure a sufficiently firm grasp before rotation.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have significantly advanced the capabilities of robotic agents in executing diverse tasks; however, they still face challenges in contact-rich manipulation scenarios that require precise physical interactions. To address this limitation, recent studies have attempted to incorporate tactile signals during downstream tasks, enabling pretrained VLAs to interpret tactile feedback. Nevertheless, introducing new modalities during finetuning, which are rarely present in the pretrain stage, may disrupt the pretrained capabilities of VLAs. In addition, the inherently slow inference speed of VLAs hampers real-time responsiveness and limits the effective utilization of tactile feedback for action adjustment. To overcome these challenges, we propose Adaptive Tactile Vision-Language-Action (AT-VLA), which introduces a novel Adaptive Tactile Injection mechanism. This mechanism dynamically determines the appropriate timing and locations for tactile injection, incorporating only when it significantly contributes to action generation, thereby minimizing interference with pretrained representations. Furthermore, to enable rapid and accurate tactile responses, we propose a Tactile Reaction Dual-Stream mechanism, which decouples sensory processing into a slow visual-language stream for low-frequency perceptual reasoning and a fast tactile control stream for high-frequency physical interaction understanding, achieving real-time close-loop responses within 0.04 s. Real-world experiments thoroughly validate the effectiveness of AT-VLA in contact-rich manipulation tasks. The project page is available at: https://sites.google.com/view/at-vla.
1. Introduction
Contact-rich manipulation demands precise interaction feedback that VLAs often overlook, while introducing tactile signals can disrupt pretrained capabilities and slow inference limits timely responses. AT-VLA adaptively injects tactile input and separates slow visual-language reasoning from fast tactile control.
1. Introduction
- VLAs often overlook tactile interaction feedback needed for precise control and safe contact-rich manipulation.Downstream tactile integration may also disrupt pretrained perception or object grounding, while slow inference limits timely closed-loop responses.Source: §1 · p. 2
- Adaptive Tactile Injection uses a learnable gate to decide when tactile features enter action generation, preserving pretrained behavior in non-contact phases.Adaptive cross-attention conditionally switches how tactile information is incorporated.Source: §1 · p. 2
- The mechanism minimizes tactile signals’ impact on the vanilla model to retain robust action generation, including target-object localization.Tactile integration does not require changing the model structure or feature dimensionality relative to the inactive state.Source: §1 · p. 2
- Tactile Reaction Dual-Stream processes visual-language inputs slowly and tactile feedback rapidly, conditioning actions on recent tactile input and the latest reasoning output.It enables closed-loop control within 0.04 s inference speed.Source: §1 · p. 2
- AT-VLA achieves superior performance on real-world contact-rich tasks and maintains strong performance without tactile signals at inference, comparable to the vanilla VLA.The authors describe this as modality-agnostic robustness when tactile signals are unstable or unavailable.Source: §1 · p. 2
2. Related Work
Prior VLA work combines perception, reasoning, and action generation, while interaction-feedback policies add tactile or force sensing for manipulation. AT-VLA’s related-work distinction is using tactile feedback as the fast stream and managing its integration with pretrained VLA representations.
2.1. Vision Language Action Model
- VLA action generation commonly uses diffusion-based models, and GO-1 provides the action-modeling approach followed in this paper.Related work also describes dual-system strategies pairing slow high-level reasoning with faster action generation.Source: §2.1 · pp. 2–3
- Unlike dual-system approaches using visual or point-cloud fast streams, AT-VLA uses high-frequency tactile feedback to react to contact events.The cited motivation is rapid reaction to contact and safer physical interactions.Source: §2.1 · p. 3
2.2. Interaction Feedback Policies
- Prior interaction-feedback policies use signals such as marker offsets, tactile input, or joint torque, force, and end-effector tactile sensing for manipulation.The cited examples include RDP’s marker-offset feedback, TLA’s tactile-and-language insertion policy, and TA-VLA’s tactile adapter placement comparison.Source: §2.2 · p. 3
- The paper distinguishes its approach from tactile-gate methods that append tactile tokens and process modalities at the same frequency.It argues that appending an unseen modality may disrupt pretrained token-sequence modeling and that VLA latency motivates decoupled processing frequencies.Source: §2.2 · p. 3
3. Method
AT-VLA uses a tactile gate and adaptive cross-attention to introduce tactile feedback selectively, while a dual-stream design pairs slower visual-language reasoning with faster tactile control. Its training objectives and inference behavior accommodate both gated and ungated operation.
3.1. Framework of AT-VLA
- The policy maps camera images, language, tactile feedback, and robot proprioception to a 14-DoF action chunk for both arms.The tactile input is resultant contact force with 3D normal and 3D tangential components; the instantiation adds a lightweight MLP tactile encoder to pretrained GO-1.Source: §3.1 · pp. 3–4
3.2. Adaptive Tactile Injection
- Directly adding tactile tokens during fine-tuning can shift attention away from the target object and impair grasp localization, motivating adaptive injection.The attention-map comparison examines action reasoning with and without tactile feedback.Source: §3.2 · p. 4, Figure 3 · p. 4
- A learned tactile gate classifies contact state, and adaptive cross-attention switches its query from the state token to the tactile token when contact is detected.When the gate is inactive, the input and architecture remain identical to the vanilla VLA; the gate is trained with manually labeled contact/non-contact frames and binary cross-entropy.Source: §3.2 · p. 4
3.3. Effective Tactile Reaction Dual-Stream
- The dual-stream design uses slow visual-language features as cross-attention keys and values and fast tactile input as the action expert’s query conditions.The fast stream uses the latest tactile feedback while conditioning on the periodically updated slow-stream output within the action-chunk horizon, supporting 0.04 s inference.Source: §3.3 · pp. 4–5
3.4. Training Objectives and Inference Pipeline
- Training combines action and gating losses, while inference uses synchronous streams when the gate is inactive and a 3:1 fast-to-slow frequency ratio when it is active.The reported gating-loss weight is 0.01; in the active state, attention queries switch to tactile tokens.Source: §3.4 · p. 5
4. Experiment
Across real-world contact-rich tasks, AT-VLA outperforms the tested baselines overall while preserving comparable pre-contact performance to pretrained VLAs. Evaluations also examine missing tactile input, component contributions, and tactile formats.
4.1. Setup
- The evaluation uses four contact-rich and two non-contact-rich tasks, with 30–50 demonstrations and 15 test trials per task.Tasks include Unzip Bag, Stamp, Wipe Vase, Unscrew Lid, Pick and Place, and Open Drawer.Source: §4.1 · p. 5
4.2. Contact-rich Task Evaluation
- AT-VLA outperforms all baselines overall and exceeds tactile-based policies in contact-rich manipulation, except on Unscrew Lid, where grip slippage can cause failures.Compared with GO-1 and π0.5, its pre-contact performance is comparable; the authors attribute its Unscrew Lid exception partly to baselines receiving an ideal grasp before rotation.Source: §4.2 · p. 6, Table 1 · p. 6
4.3. Modality-agnostic Evaluation
- Without tactile input at inference, AT-VLA performs comparably to GO-1 and π0.5 on Pick-and-Place and Open Drawer, and remains reliable on the representative Stamp task.The tactile-present and tactile-absent variants share model weights; the former serves as an upper bound.Source: Table 2 · p. 6, §4.3 · p. 7
4.4.1. Contribution of Each Component
- Direct tactile incorporation lowers average success by 9% versus vanilla VLA, while adding the Tactile Gate yields a 17% improvement over vanilla and the Reaction Dual-Stream adds 11%.The direct-input failures often occur during grasping; the gated setting preserves vanilla queries when the gate is inactive, and the dual-stream setting separates visual and tactile input frequencies.Source: §4.4.1 · pp. 7–8, Table 3 · p. 8
4.4.2. Different Tactile Format
- Directly feeding tactile tokens throughout manipulation degrades performance across visual-tactile image, marker 2D, and force 6D formats, including failures in object grasping.The authors suggest that higher-dimensional tactile input may perturb pretrained representations more, and emphasize balancing tactile influence against the pretrained model.Source: §4.4.2 · p. 8
- Using the authors’ proposed method to incorporate tactile input, Ex5 outperforms Ex4 by 27% and Ex7 surpasses Ex6 by 38%, demonstrating robustness across different tactile formats.Force 6D achieves the best performance, followed by marker 2D and then the visual-tactile image.Source: §4.4.2 · p. 8
5. Conclusion
AT-VLA combines adaptive tactile injection with dual-stream control and outperforms state-of-the-art VLA and tactile-based policies in real-world contact-rich manipulation. Its tactile reactions operate in a closed loop within 0.04 s.
5. Conclusion
- Real-world experiments show AT-VLA significantly outperforms state-of-the-art VLA and tactile-based policies in contact-rich manipulation tasks.The conclusion also characterizes the system as adaptable and modality-agnostic.Source: §5 · p. 8
- AT-VLA achieves closed-loop tactile reactions within 0.04 s by decoupling slow perceptual reasoning from fast physical control.Source: §5 · p. 8
- AT-VLA adaptively integrates tactile sensing while balancing pretrained visual-language knowledge with newly learned tactile representations.The conclusion describes the framework as preserving model integrity while enhancing action precision.Source: §5 · p. 8