PaperlayerView exact revision on arXiv

Source-linked AI summary

VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation

Chaofan Zhang, Peng Hao, Xiaoge Cao, Xiaoshuai Hao, Shaowei Cui, Shuo Wang

arXiv:2505.09577v1cs.RO

TL;DR

Language-conditioned robotic manipulation remains comparatively underexplored for contact-rich tasks requiring tactile feedback alongside vision. VTLA integrates visual, tactile, and language inputs with temporally enhanced tokens and DPO-based preference learning, and achieves over 90% success on unseen peg shapes while showing promising real-world transfer.

  • Problem

    Existing language-conditioned manipulation methods provide limited support for contact-rich tasks because visual-only VLA systems lack tactile feedback and tactile-only approaches lack global visual perception.

  • Method

    VTLA integrates visual, tactile, and linguistic information using VGTE for temporal reasoning and DPO to provide regression-like preference supervision.

  • Results

    Over 90% assembly success on unseen peg shapes was achieved, with VTLA outperforming traditional imitation learning methods and TLA/VLA baselines.

  • Takeaways & Limitations

    VTLA supports generalized visual-tactile skills for contact-rich insertion manipulation and demonstrates promising Sim2Real transfer.

  • Takeaways & Limitations

    VTLA uses an off-the-shelf vision encoder for tactile inputs, which may lose tactile-specific features and leaves dedicated tactile-language alignment for future work.

Abstract

from arXiv · show

While vision-language models have advanced significantly, their application in language-conditioned robotic manipulation is still underexplored, especially for contact-rich tasks that extend beyond visually dominant pick-and-place scenarios. To bridge this gap, we introduce Vision-Tactile-Language-Action model, a novel framework that enables robust policy generation in contact-intensive scenarios by effectively integrating visual and tactile inputs through cross-modal language grounding. A low-cost, multi-modal dataset has been constructed in a simulation environment, containing vision-tactile-action-instruction pairs specifically designed for the fingertip insertion task. Furthermore, we introduce Direct Preference Optimization (DPO) to offer regression-like supervision for the VTLA model, effectively bridging the gap between classification-based next token prediction loss and continuous robotic tasks. Experimental results show that the VTLA model outperforms traditional imitation learning methods (e.g., diffusion policies) and existing multi-modal baselines (TLA/VLA), achieving over 90% success rates on unseen peg shapes. Finally, we conduct real-world peg-in-hole experiments to demonstrate the exceptional Sim2Real performance of the proposed VTLA model. For supplementary videos and results, please visit our project website: https://sites.google.com/view/vtla

1 Introduction

VTLA addresses the limitations of existing vision- or tactile-only approaches by integrating visual, tactile, and linguistic information for contact-rich manipulation. It introduces temporally enhanced visual-tactile tokens and preference learning, with real-world experiments supporting the framework’s effectiveness.

  • Existing vision-tactile systems often rely on proprietary models and task-specific datasets, limiting generalization and perceptual-motor adaptability.
  • TLA models lack global visual perception, while most VLA systems lack tactile feedback for contact-rich manipulation.
  • VTLA integrates visual, tactile, and linguistic information for contact-rich manipulation and cross-modal language-conditioned skill learning.
  • VGTE emphasizes visual tokens and performs temporal fusion before tokenization to improve cross-modal temporal reasoning.
  • DPO provides regression-like preference supervision intended to reduce overfitting to ground-truth actions and improve generalization.
  • Real-world insertion experiments show that VTLA outperforms current methods, supporting the effectiveness of its designed modules.

2 Related Works

Related work spans vision-tactile fusion, vision-language-action manipulation, and tactile-language modeling. These studies establish relevant foundations while leaving tactile data and language-conditioned tactile action modeling comparatively underused.

  • Vision-Tactile Learning: Vision-tactile fusion has been applied to slip detection, grasp outcome prediction, liquid pouring, and peg-in-hole assembly.
  • VLM for Robot Manipulation: VLM-based manipulation methods formulate robotic actions as token sequences and use multimodal or staged training for robotic control.
  • Tactile-Language Model in Robotics: Tactile-language research has advanced material and texture understanding, but tactile data remains underutilized in robotic manipulation.
  • Tactile-Language Model in Robotics: TLA links tactile modalities and language-conditioned interaction policies, extending tactile-language modeling toward contact-rich manipulation.

3 Vision-Tactile-Language-Action Model

VTLA combines visual, tactile, and linguistic inputs for peg-in-hole manipulation, using temporally enhanced tokens, supervised fine-tuning, and DPO-based preference learning. Its pipeline uses simulation data and staged optimization to address temporal reasoning and the mismatch between token classification and continuous control.

  • Simulation and dataset: The simulation setup captures wrist-camera and fingertip visuotactile observations during randomized peg-in-hole insertion attempts.The task includes randomized 3-DOF misalignment, collision-based failure, up to 15 attempts, and insertion-depth success criteria.
  • Simulation and dataset: The VTLA dataset contains 28,000 assembly samples across five peg-hole shapes with 0.6–2.0 mm clearances and domain randomization.Each sample includes left/right tactile image sequences, a visual image, and an action label.
  • Instruction tuning: VTLA organizes tactile, visual, and textual inputs into instruction-formatted sequences whose action labels provide ground-truth supervision.Tactile and visual images are inserted sequentially into dialogue-style inputs, followed by task instructions and action prediction.
  • Instruction tuning: Vision-guided temporal enhancement emphasizes visual information and temporally encodes tactile observations before multimodal tokens are processed by the language model.Visual inputs are positioned closer to action prediction, while tactile sequences are represented using a Vision Transformer to capture temporal dependencies.
  • Instruction tuning: Stage 1 fine-tunes VTLA with next-token prediction after vision and tactile encoders and modality adapters generate tokens for the language model.The vision encoder and modality adapter are frozen, while the language model is tuned using NTP loss.
  • Preference learning: Stage 2 applies DPO to action preferences generated from the fine-tuned model, treating responses closer to ground truth as preferred over rejected alternatives.The preference objective uses trainable and frozen VTLA models initialized from the Stage 1 model and promotes higher likelihood for preferred responses.

4 Experiments

Experiments compare VTLA with imitation-learning and multimodal baselines in simulation and real-world insertion, including preference-learning ablations and Sim2Real generalization. VTLA achieves stronger performance across these evaluations, including robust real-world insertion on varied peg shapes and clearances.

  • Experiment Setup: Experiments compare Diffusion Policy, VLA, TLA, and VTLA across dataset, simulation, and real-world insertion settings.The evaluation includes dataset metrics, insertion success and attempt steps, and real-robot experiments.
  • Comparison with Baseline Methods: 80+% vs 22% success rate: LLM-based models outperform Diffusion Policy in the 0.6 mm-clearance simulation insertion experiment.The comparison is reported for square-peg insertion in simulation.
  • Comparison with Baseline Methods: 47.3% vs 46.1% GCR-ID and 90% vs 80% success rate: VTLA outperforms VLA with tactile encoding on dataset and 0.6 mm-clearance task performance.The reported gains are attributed to temporally aware tactile tokens in VTLA.
  • Ablation Study: 16% improvement in GCR and approximately 10% lower L1 error: VTLA-DPO-1k improves OOD performance over the corresponding non-DPO setting.The gains are reported across all action dimensions for OOD data.
  • Real-world Robotic Insertion: 100% assembly success rate: VTLA generalizes to unseen peg-hole shapes in real-world insertion with 0.6 mm clearance.VTLA slightly outperforms its performance on in-distribution peg shapes.
  • Real-world Robotic Insertion: 1.85 steps vs 2.3 steps: VTLA is more insertion-efficient than VLA in the real-world 0.6 mm-clearance comparison.Both vision-based methods achieve success rates over 90%, while TLA achieves only 30–40%.

5 Conclusion

VTLA combines visual-tactile-language-action modeling for contact-rich insertion and generalizes across assembly conditions. It achieves over 90% success in challenging fingertip peg-in-hole tasks and 95% success in real-world insertion using only simulated training data, while tactile-language alignment and modality fusion remain open limitations.

  • VTLA surpasses traditional imitation learning, TLA, and VLA models, achieving an assembly success rate exceeding 90%.The result is reported for the challenging fingertip peg-in-hole task.
  • VTLA generalizes across different assembly clearances and peg shapes.
  • 95% real-world assembly success is achieved by VTLA trained using only simulated data.This result demonstrates promising Sim2Real transfer performance in the real-world insertion task.
  • VTLA remains limited by tactile-language alignment and deep visual-tactile fusion for language models.The paper identifies dedicated tactile-language alignment and deeper modality fusion as future work.

Appendix

The supplementary appendix adds methodological, experimental, and comparative details omitted from the main manuscript. It covers simulation domain randomization, the real-world insertion setup, and performance under poor lighting.

  • The supplementary material provides additional method and experimental details excluded from the main manuscript because of page limitations.
  • Section A: Section A describes domain randomization for the simulated dataset.
  • Section B: Section B details the real-world robot setup for the insertion task.
  • Section C: Section C compares VTLA and VLA under poor lighting conditions.

A Domain Randomization on Simulated Dataset

The simulated dataset uses domain randomization across physical, task, tactile, and visual factors to improve zero-shot Sim2Real transfer. These variations target uncertainties in real sensors, environments, task initialization, and camera configurations.

  • Domain randomization varies simulation, task, visual, and tactile parameters to enhance zero-shot Sim2Real transfer.The settings are summarized in Table 8.
  • Physical parameters: Physical randomization includes Young’s modulus, Poisson ratio, and friction coefficient to model sensor aging and environmental variability.Real-world friction depends on material properties, humidity, and other factors that are difficult to model precisely.
  • Task-related: Task initialization randomizes the peg’s in-hand position and gripper width to improve policy adaptability.These variables represent episode-to-episode variation in grasping position and applied force.
  • Tactile images: Simulated tactile images receive color jittering in brightness, contrast, saturation, and hue to reduce the tactile sim-real gap.
  • Vision images: Vision randomization changes lighting and image geometry, including scale, translation, rotation, and shear, to reduce visual sim-real discrepancies.Lighting intensities and directions are randomized, while image transformations address wrist-camera pose differences.

B Real-World Robot Setup for Insertion Task

The real-world insertion system uses a UR3 arm, Robotiq gripper, wrist-mounted vision, and fingertip tactile sensing. The robot grasps a peg, approaches a target hole with randomized pose, and records multimodal observations during collisions.

  • The setup uses a 6-DoF UR3 arm with a Robotiq 2F-85 gripper for real-world insertion experiments.
  • An Intel RealSense D405 wrist camera captures images, while two GelStereo 2.0 fingertip sensors capture tactile observations during peg-hole collisions.
  • The robot grasps the peg from a holder and approaches the target hole with a randomized pose.Figure 5 depicts the robot platform, visual-tactile observations, and a dialogue round.

C VTLA vs. VLA under Poor Lighting Condition

Under poor lighting and 0.6 mm assembly clearance, VTLA successfully performs peg-in-hole insertion while VLA struggles under the same conditions.

  • VTLA successfully performs real-world peg-in-hole insertion under poor lighting with 0.6 mm assembly clearance.
  • VLA struggles to complete the same insertion task under poor lighting and 0.6 mm clearance.The initial peg-hole misalignment is consistent with the VTLA comparison.
  • The comparison indicates that combining visual and tactile observations improves performance in contact-rich manipulation when visual quality is degraded.
Loading 2505.09577v1…