Source-linked AI summary
VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback
Jianxin Bi, Kevin Yuchen Ma, Ce Hao, Mike Zheng Shou, Harold Soh
TL;DR
VLA models lack mechanisms to interpret tactile signals, limiting contact-rich manipulation. VLA-Touch addresses this gap without fine-tuning the base VLA by combining semantic tactile feedback for planning with tactile action refinement, achieving improved planning and manipulation outcomes in real-world experiments.
Problem
VLA models lack intrinsic mechanisms to process tactile inputs, limiting their use of touch for planning and control in contact-rich tasks.
Method
VLA-Touch combines a pretrained tactile-language model for high-level planning with an interpolant-based controller that refines VLA actions using tactile signals, without fine-tuning the base VLA.
Results
Dual-level tactile feedback improves planning efficiency by up to 40%, manipulation success by up to 35% over vision-only VLA baselines, and task success by up to 35% over single-level feedback.
Takeaways & Limitations
The experiments support dual-level tactile feedback and multimodal vision-touch fusion as useful components for contact-rich manipulation.
Takeaways & Limitations
The implementation has gripper-sensor discrepancies, focuses mainly on object-position and target-location generalization, and operates at 8 Hz rather than fully using tactile signals at ≥25 Hz.
Abstract
from arXiv · showhide
Tactile feedback is generally recognized to be crucial for effective interaction with the physical world. However, state-of-the-art Vision-Language-Action (VLA) models lack the ability to interpret and use tactile signals, limiting their effectiveness in contact-rich tasks. Incorporating tactile feedback into these systems is challenging due to the absence of large multi-modal datasets. We present VLA-Touch, an approach that enhances generalist robot policies with tactile sensing \emph{without fine-tuning} the base VLA. Our method introduces two key innovations: (1) a pipeline that leverages a pretrained tactile-language model that provides semantic tactile feedback for high-level task planning, and (2) a diffusion-based controller that refines VLA-generated actions with tactile signals for contact-rich manipulation. Through real-world experiments, we demonstrate that our dual-level integration of tactile feedback improves task planning efficiency while enhancing execution precision. Code is open-sourced at \href{https://github.com/jxbi1010/VLA-Touch}{this URL}.
1 Introduction
VLA-Touch adds tactile feedback to existing VLA pipelines at both planning and control levels without fine-tuning the base VLA. Its real-world experiments report improved planning efficiency, manipulation success, and task success from combining both levels.
- Visual perception alone struggles with compliance, texture, and contact events, while tactile integration into large-scale foundation models remains underexplored.
- Tactile feedback supports planning about visually ambiguous object properties and control of friction and contact events in contact-rich manipulation.
- The modular framework incorporates touch into existing pipelines without fine-tuning base VLA models.
- The system combines a VLM task planner, a pretrained tactile-language model for linguistic object-property feedback, and a tactile-augmented controller for refining VLA actions.
- Planning efficiency improves by up to 40%, manipulation success by up to 35% over vision-only VLA baselines, and task success by up to 35% over single-level tactile feedback.
- The modular scaffold enables experimentation with tactile-input effects, while richer feedback and tighter module integration remain future directions.
2 Related Work
Related work spans visual-language-action models, tactile foundation models, and tactile sensors. VLA-Touch builds on tactile-language modeling to add semantic tactile interpretation to VLA task planning.
- VLA models combine visual perception and language understanding to generate robot actions across diverse manipulation tasks.
- Tactile foundation models learn generalizable representations from large-scale tactile datasets and are increasingly extended toward language and multimodal reasoning.
- VLA-Touch uses a pretrained tactile-language model to generate semantic interpretations of grasped objects for task planning.
- Tactile sensors measure contact forces, surface textures, and material properties using taxel-based or vision-based sensing designs.
3 Methodology
The methodology combines cyclical tactile-assisted planning with tactile refinement of VLA action chunks. A language model updates instructions from Octopi feedback, while an interpolant-based controller conditions action refinement on visual, proprioceptive, and tactile inputs.
- The implementation uses RDT as a base VLA, while the integration methodology is intended to apply across VLA architectures.
- Tactile-Assisted Task Planning: A cyclical planner generates manipulation instructions, executes VLA action chunks, gathers observations, and updates planning from tactile feedback.
- Tactile-Assisted Task Planning: GPT-4o is prompted to produce primitive single-object actions and use feedback about retrieved information or execution results for subsequent planning.
- Tactile-Assisted Task Planning: Octopi processes six Gelsight tactile frames to infer properties such as roughness and hardness, then converts them into linguistic feedback for planning.
- VLA Policy Refinement with Tactile Sensing: BRIDGeR refines VLA-generated actions using stochastic interpolants initialized from the VLA action distribution rather than Gaussian noise.
- VLA Policy Refinement with Tactile Sensing: The controller conditions refinement on a source action chunk, RGB and proprioceptive state, and aggregated tactile force signals, then outputs a refined action sequence.
- VLA Policy Refinement with Tactile Sensing: Controller training pairs VLA-generated actions with expert sequences, while inference refines non-overlapping action-window segments sequentially.
4 Experiments
The experiments evaluate tactile feedback for task planning, contact-rich manipulation, and their dual-level integration using real-world tasks and ablations. Results show that linguistic tactile feedback improves planning, while a tactile-conditioned interpolant controller improves manipulation and benefits from both visual and tactile inputs.
- Experiment Design: The experiments compare tactile-assisted planning, tactile-enhanced manipulation controllers, and single-level ablations across real-world contact-rich tasks.Planning compares GPT-4o with no tactile input, raw tactile images, or Octopi-generated linguistic feedback; manipulation compares RDT variants with residual or interpolant controllers.
- Task Planning Results: Linguistic tactile descriptions outperform raw tactile images for VLM-based planning, reaching 90% success for force prediction and 75% for hardness prediction.Raw tactile images achieve 100% for surface roughness, but only 50% for force and 60% for hardness.
- Manipulation Results: The interpolant controller improves base RDT task success rates by 42% for Cup, 140% for Wipe, and 67% for Peel.It also outperforms the residual controller by 67% for Cup, 100% for Wipe, and 42% for Peel.
- Manipulation Results: The interpolant controller achieves 12/20 successful picks for Cup and completes full wiping in 12/20 trials.The controller improves grasping and pressure modulation relative to the compared policies.
- Manipulation Results: During peeling, the interpolant controller removes 13 pieces with only 3 short fragments, compared with 12 pieces and 5 short fragments for the residual controller.The result is reported as evidence of improved contact maintenance and cutting consistency.
- Dual-Level Feedback: Removing planning-level tactile feedback reduces success by 44% for Cup, 58% for Wipe, and 14% for Peel, while removing control-level tactile feedback reduces it by 33%, 58%, and 43%, respectively.The ablation attributes failures to incorrect object selection without planning feedback and failed grasping or inaccurate contact control without control feedback.
- Dual-Level Feedback: Removing tactile feedback reduces success rates by 50% for cup placement, 42% for wiping, and 50% for peeling.The controller ablation also indicates that visual feedback remains important for spatial positioning and grasp planning, while tactile feedback is critical during contact-rich phases.
5 Conclusion
VLA-Touch improves VLA policies through dual-level tactile feedback without fine-tuning the base VLA with tactile data. The framework combines semantic tactile feedback for planning with tactile-guided action refinement for contact-rich manipulation.
- VLA-Touch adds dual-level tactile feedback to VLA policies without fine-tuning the VLA model with tactile data.The framework targets both high-level planning and contact-rich manipulation.
- A pretrained tactile-language model provides semantic tactile feedback for high-level task planning.
- An interpolant-based controller refines VLA-generated actions using tactile signals for contact-rich manipulation.
- Three real-world experiments demonstrated the effectiveness of the dual-level tactile feedback system.
- Ablation studies confirmed that both visual and tactile inputs are crucial for contact-rich tasks.
6 Limitation and Future Works
The implementation remains bounded by gripper–dataset mismatch, limited cross-task generalization, and an 8 Hz controller that does not fully exploit high-frequency tactile signals. Future work targets broader task scalability, improved grasping, and event-triggered inference.
- Limitations: Gripper control differs from the Octopi pretraining setup, creating contact-measurement discrepancies that can compromise tactile interpretation, particularly for hardness.
- Limitations: Experiments mainly test generalization across object positions and target locations, while cross-task generalization remains future work requiring significantly more training.
- Limitations: The Interpolant controller operates at 8 hz and does not fully leverage high-frequency tactile signals of ≥25 Hz for dynamic interactions.
- Future Works: Future work proposes active grasping that combines visual-tactile feedback to dynamically refine grasp poses and improve touch inference.
- Future Works: Future work also proposes event-triggered inference and task-agnostic policy refinement to improve temporal efficiency and scale to more real-world tasks.
Task 1: Cup
The cup task uses tactile feedback to infer whether a cup is empty or full from lifting-related force information, then places the cup according to the inferred state.
- The task asks the robot to identify whether a cup is full or empty and place it onto a plate if it is not empty.
- The robot first grips the cup gently and requests its weight to distinguish between the two states.
- Tactile images are also used to interpret the cup state; minimal change in the sensor pattern indicates that the cup is likely empty.
- A tactile shear-force magnitude of 0.6571505252217286 is compared with reference magnitudes of around 0.55 for an empty cup and around 1.1 for a full cup.
- After determining the cup state, the robot places the cup onto the plate or returns it to its original position.
Task 2: Wipe
The sponge task compares tactile roughness information for left and right sponges and selects the smoother one. Tactile measurements and image deformation support choosing the left sponge.
- The task asks the robot to touch two sponges, identify their roughness, and pick up the smoother one.
- The robot begins by touching the left sponge to obtain its roughness level, then touches the right sponge for comparison.
- Minimal deformation indicates that the left sponge is relatively smooth, while significant deformation indicates that the right sponge is rougher.
- The left sponge has hardness 2.973193645477295 and roughness 6.088855266571045.
- The right sponge has hardness 4.499370098014 and roughness 7.802529811859131.
- The robot therefore picks up the sponge on the left.
Task 3: Mango
The mango task uses sequential tactile checks to compare softness and guide selection. The reported interactions include cases where both mangoes are judged firm and either may be picked.
- Sequential tactile assessment: The robot first touches one mango to assess softness, then uses tactile feedback to guide which mango to pick.In one interaction, it touches the right mango and subsequently selects it as softer and likely riper.
- Sequential tactile assessment: After assessing the left mango as firm, the robot touches the right mango for a corresponding softness assessment.The dialogue records hardness and roughness values for the left mango before the second tactile check.
- Selection outcome: When both mangoes are judged firm, the robot concludes that either mango can be picked.This decision follows tactile observations for both sides rather than visual inspection alone.
B Ablation on Tactile Feedback Format from Octopi
The mango peeling ablation compares two tactile feedback formats under a subtle hardness distinction. The results favor method (b), while also identifying a limitation of natural-language representations of touch.
- Success-rate comparison: Method (b) reached 75% success, compared with 60% for method (a), in the mango peeling task.The passage attributes the result to a setting where the hardness difference is more subtle.
- Interpretation: The comparison highlights limitations of representing tactile physical properties through natural language.This interpretation is stated alongside the two success rates for the mango peeling task.
C Interpolant Controller
The interpolant controller refines VLA-generated action chunks by combining source-policy actions with visual, proprioceptive, and tactile information. It uses a stochastic interpolation process to target expert behavior for contact-rich manipulation.
- Controller formulation: The stochastic interpolant bridges source and target action distributions instead of denoising from standard Gaussian noise alone.Using informative source policies is described as enabling better performance with fewer diffusion steps.
- Stochastic interpolation: The interpolant process begins at the source action and ends at the target action, with the noise schedule vanishing at both boundaries.The boundary conditions are I(0,...)=a0 and I(1,...)=a1, while γ(0)=γ(1)=0.
- Intended benefit: The formulation is intended to combine VLA robustness with tactile-enabled precision for contact-rich manipulation.The paper explicitly describes this as the benefit of incorporating tactile sensing into refinement.
- Controller formulation: The controller starts from VLA actions and refines them toward an expert policy that incorporates tactile feedback.The VLA policy supplies the source behavior, while the expert policy defines the tactile-aware target.
- Inference procedure: During inference, the controller refines truncated VLA action chunks and executes the refined actions at up to 8 Hz.The VLA uses 64-step chunks, with the first 48 steps refined before subsequent observation updates.