Source-linked AI summary
OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing
Zhengxue Cheng, Yiqian Zhang, Anni Tang, Keyu Wang, Wenkang Zhang, Haoyu Li, Hengdi Zhang, Li Song
TL;DR
Existing VLA models largely overlook tactile perception despite its importance for contact-rich manipulation and the heterogeneity of tactile sensors. OmniVTLA addresses this gap with dual-path tactile encoding, semantic alignment, and the ObjTac dataset. It reports higher manipulation success rates than VLA baselines, along with reduced completion time and smoother trajectories.
Problem
Existing VLA models overlook tactile perception and struggle with the heterogeneous tactile data relevant to contact-rich manipulation.
Method
OmniVTLA combines a pretrained ViT, a semantically-aligned tactile ViT, and the 135K-sample ObjTac tri-modal dataset to align tactile, visual, and language representations.
Results
OmniVTLA improves pick-and-place success rates by 21.9% to 96.9% with a gripper and by 6.2% to 100% with a dexterous hand, while also reducing completion time and smoothing trajectories.
Takeaways & Limitations
The results support integrating semantically aligned tactile sensing into VLA policies for contact-rich robotic manipulation.
Abstract
from arXiv · showhide
Recent vision-language-action (VLA) models build upon vision-language foundations, and have achieved promising results and exhibit the possibility of task generalization in robot manipulation. However, due to the heterogeneity of tactile sensors and the difficulty of acquiring tactile data, current VLA models significantly overlook the importance of tactile perception and fail in contact-rich tasks. To address this issue, this paper proposes OmniVTLA, a novel architecture involving tactile sensing. Specifically, our contributions are threefold. First, our OmniVTLA features a dual-path tactile encoder framework. This framework enhances tactile perception across diverse vision-based and force-based tactile sensors by using a pretrained vision transformer (ViT) and a semantically-aligned tactile ViT (SA-ViT). Second, we introduce ObjTac, a comprehensive force-based tactile dataset capturing textual, visual, and tactile information for 56 objects across 10 categories. With 135K tri-modal samples, ObjTac supplements existing visuo-tactile datasets. Third, leveraging this dataset, we train a semantically-aligned tactile encoder to learn a unified tactile representation, serving as a better initialization for OmniVTLA. Real-world experiments demonstrate substantial improvements over state-of-the-art VLA baselines, achieving 96.9% success rates with grippers, (21.9% higher over baseline) and 100% success rates with dexterous hands (6.2% higher over baseline) in pick-and-place tasks. Besides, OmniVTLA significantly reduces task completion time and generates smoother trajectories through tactile sensing compared to existing VLA. Our ObjTac dataset can be found at https://readerek.github.io/Objtac.github.io
1 Introduction
OmniVTLA addresses the limited tactile perception and semantic alignment of existing VLA models by combining vision, language, and touch with a dual tactile encoder and the ObjTac dataset. Real-world experiments report higher success rates, shorter completion times, and smoother trajectories than typical VLA baselines.
- Motivation: Existing VLA models primarily rely on vision and language, overlooking tactile feedback needed for contact-rich manipulation.Tactile sensing provides complementary information about properties such as texture and hardness.
- Method: OmniVTLA uses a dual tactile-encoder path to align tactile signals with visual and language concepts.The architecture combines a pretrained ViT with a semantically-aligned tactile ViT.
- Results: 21.9% higher success rates reach 96.9% with a gripper, while 6.2% higher rates reach 100% with a dexterous hand.On contact-rich peg insertion, OmniVTLA reaches an 83.3% success rate, 33.3% above the baseline.
- Dataset: 135K tri-modal samples cover 56 objects across 10 categories in the ObjTac dataset.The dataset contains textual, visual, and tactile information and supports training a semantically-aligned tactile encoder.
- Results: OmniVTLA reduces task completion time and produces smoother trajectories than existing VLA baselines.The reported trajectory behavior is faster when clear and slower during contact approach.
2 Related Works
Prior work spans tactile representation learning, vision-tactile manipulation, and generalist VLA models, but these areas leave a gap in semantically aligned tactile encoder design. OmniVTLA positions dual-path tactile encoding as a bridge between tactile sensing and VLA-based manipulation.
- Comparison: Table 1 compares VLA models by their language, vision, tactile, and action modalities.The table’s abbreviations are L for language, V for vision, T for tactile, and A for action.
- Tactile Sensing: Tactile research has progressed from low-level signal processing toward representations transferable across tasks, sensors, and modalities.Recent methods model object properties, tactile dynamics, or 3D contact geometry.
- Vision-Tactile Fusion: Vision-tactile policies achieve task-specific contact-rich manipulation but remain limited in semantic reasoning and generalization compared with VLA models.The cited gap motivates semantic fusion between vision, tactile sensing, and language.
- Vision-Language-Action Models: VLA models use language, vision, and action modeling to support generalist robotic policies and knowledge transfer from large-scale pretraining.Subsequent work expands action generation through flow-based or diffusion-based approaches.
- Tactile-Enhanced VLA: Tactile-enhanced VLA approaches have explored sensor fusion and force-aware control without fully resolving tactile encoder design.OmniVTLA addresses this gap with a dual-encoder path and unified cross-modal representation learning.
3 Methods
OmniVTLA integrates visual, language, and tactile inputs through a dual-path tactile encoder and trains semantic alignment using the ObjTac dataset.
- Dual-Encoder Tactile Path: Its tactile pathway explores ViT-based encoders because tactile data differ substantially from visual data.The framework uses a pretrained ViT and a semantically-aligned tactile ViT as dual tactile encoder paths.
- Overall Architecture: OmniVTLA processes language, images, and tactile data through tokenizers, a backbone, and an action head.Image and tactile inputs are projected into latent tokens before action prediction.
- ObjTac Dataset: ObjTac contains paired visual and tactile data for 56 objects across 10 material categories, with 135K collected samples.The dataset includes interaction trials, force recordings, and first-person visual recordings.
- Semantic-Aligned Tactile Encoder: The semantic-alignment training pipeline combines tri-modal alignment losses with cross-sensor matching loss.The resulting encoder grounds tactile properties such as material, roughness, and hardness in visual and linguistic contexts.
- Encoder Evaluation: The encoder comparison includes vision-pretrained CLIP and SigLIP, unaligned Tac-ViT, AnyTouch, and SA-ViT.CLIP and SigLIP provide same-parameter-count vision-encoder comparisons, while Tac-ViT removes the alignment objective.
4 Experiments
Experiments evaluate tactile encoder variants through offline trajectory prediction and real-world manipulation with grippers and dexterous hands. OmniVTLA generally improves success, completion time, and trajectory smoothness, especially in contact-rich and soft-object tasks.
- Offline Validation: Offline validation ranks VTLA-Pre-Tac lowest in trajectory MSE and OmniVTLA second lowest on average across diverse objects.The reported trend supports using large-scale tactile data rather than only teleoperated data.
- Real-World Pick and Place: 96.9% average success rate is achieved by OmniVTLA with the two-finger gripper, while completion time falls 24.2% from 657 to 498 steps versus VLA.Parameter-matched comparisons indicate the gains are attributed to tactile encoder architecture rather than parameter count.
- Real-World Pick and Place: 6.2% higher success rate brings OmniVTLA to 100% with the four-finger dexterous hand, while completion time decreases 6% from 343 to 322 steps.On unseen Plastic and Square objects, OmniVTLA reaches 100% success versus 87.5% for VLA.
- Real-World Pick and Place: 18.7% higher average success rate and 19.9% lower average completion time are obtained when tactile sensing is added to the DP baseline.Average success rate rises from 59.4% to 78.1%, while average completion time falls from 851 to 682 steps.
- Peg Insertion: 83.3% average success rate is achieved on peg insertion, outperforming VLA by 33.3%, while VTLA-SA reaches the fastest completion time of 678 steps.The evaluation covers two in-distribution shapes and one out-of-distribution shape.
- Trajectory and Qualitative Analysis: VTLA-SA achieves the best end-effector and gripper smoothness, while OmniVTLA achieves the second-best gripper smoothness during contact-rich interaction.Qualitative results show tactile feedback supporting stable grasping and position adjustment when vision is occluded.
- Generalization to Soft Objects: On unseen Sponge and Towel objects, VTLA-SA and OmniVTLA achieve the top two maximum gripper widths, suggesting reduced excessive deformation during soft grasping.VTLA-FS and VTLA-Pre have high success rates largely because of very tight gripper closure.
5 Conclusion
OmniVTLA combines vision, tactile, and language through a dual-encoder architecture and the ObjTac dataset for cross-modal contrastive learning. It improves manipulation success, completion time, and trajectory smoothness over VLA baselines, while remaining limited in messy clutter and requiring further study of training-attempt and visual-ambiguity effects.
- Conclusion: OmniVTLA uses a dual-encoder architecture and ObjTac for cross-modal contrastive learning to interpret tactile data in task-relevant contexts.The model combines visual and language modalities with semantic-aligned tactile encoding.
- Conclusion: OmniVTLA reduces completion time and produces smoother trajectories through tactile-guided learning.These conclusions are reported across the paper’s real-world manipulation evaluations.
- Limitations: The reported experiments use repeated-attempt training data, while messy-clutter performance does not improve because success depends on visual navigation rather than tactile feedback.The paper calls for further study of single-attempt training, general tasks, and visual ambiguity.
6 Appendix
The appendix documents ObjTac, training information, and qualitative visualizations of tactile manipulation. It also illustrates failure modes of VLA and successful force balancing, contact stabilization, and smoother trajectories with OmniVTLA.
- Dataset: ObjTac contains 56 objects organized into 10 classes, with the complete inventory provided in Table 9.The dataset object list is presented as the appendix’s object-inventory reference.
- Data Collection: Data collection is divided into Touch and Grasp processes, with the Grasp process targeting grasp success, failure, stability, release, and slip events.The Grasp process is described as planned, while Touch collection records synchronized sensor and video data.
- Data Collection: The Touch process produces 252 video recordings, 135,000 video frames, and 270,000 force data points across all 56 objects.Interactions last 10–60 seconds and are sampled at 60 Hz, with synchronized 720p video at 30 FPS.
- Training Details: Table 10 lists training details for VTLA, Pi0, VTA, and DP.The appendix identifies the table as the training-detail reference.
- Qualitative Results: Peg-insertion visualizations contrast VLA failures from unbalanced force with OmniVTLA successes associated with balanced forces.The figure presents several failed and successful cases rather than aggregate metrics.
- Qualitative Results: Qualitative pick-and-place visualizations show VLA and VTLA-Pre failures contrasted with OmniVTLA’s successful grasping and stable contact.The visualization concerns manipulation of a short can moved to a plate.
- Action Trajectories: Action-trajectory visualizations plot X, Y, and Z spatial motion alongside gripper closure, with OmniVTLA showing smoother motion and VLA showing jitter and instability.The comparison includes occasional dropping in the VLA trajectory.