Source-linked AI summary
TLA: Tactile-Language-Action Model for Contact-Rich Manipulation
Peng Hao, Chaofan Zhang, Dingzhe Li, Xiaoge Cao, Xiaoshuai Hao, Shaowei Cui, Shuo Wang
TL;DR
Contact-rich language-conditioned manipulation lacks tactile-centered models and action datasets. TLA combines sequential tactile encoding, language grounding, and a 24k-example peg-in-hole dataset, outperforming traditional imitation learning and exceeding 85% success on unseen clearances and peg shapes. Its main limitation is that temporal tactile information is encoded spatially rather than rigorously modeled as a sequence.
Problem
Most vision-language-action models focus on visual tasks, while existing tactile-language datasets often exclude robotic actions or support only perception and limited grasping tasks.
Method
TLA uses cross-modal fine-tuning to combine language with sequential tactile representations for contact-rich robot-action generation, supported by a 24k tactile-action dataset for fingertip peg-in-hole assembly.
Results
TLA outperforms traditional imitation learning in action accuracy and multi-step assembly success, achieving over 85% success on unseen 1.6 mm and 1.0 mm clearances after training at 2.0 mm.
Takeaways & Limitations
TLA demonstrates language-grounded tactile skill generalization across assembly clearances and peg geometries in fingertip peg-in-hole manipulation.
Takeaways & Limitations
TLA encodes temporal tactile cues through spatial arrangements, which may not fully exploit sequential tactile information.
Abstract
from arXiv · showhide
Significant progress has been made in vision-language models. However, language-conditioned robotic manipulation for contact-rich tasks remains underexplored, particularly in terms of tactile sensing. To address this gap, we introduce the Tactile-Language-Action (TLA) model, which effectively processes sequential tactile feedback via cross-modal language grounding to enable robust policy generation in contact-intensive scenarios. In addition, we construct a comprehensive dataset that contains 24k pairs of tactile action instruction data, customized for fingertip peg-in-hole assembly, providing essential resources for TLA training and evaluation. Our results show that TLA significantly outperforms traditional imitation learning methods (e.g., diffusion policy) in terms of effective action generation and action accuracy, while demonstrating strong generalization capabilities by achieving over 85\% success rate on previously unseen assembly clearances and peg shapes. We publicly release all data and code in the hope of advancing research in language-conditioned tactile manipulation skill learning. Project website: https://sites.google.com/view/tactile-language-action/
I. INTRODUCTION
Vision-language-action models have advanced language-conditioned robotics, but their primarily visual focus leaves tactile sensing underdeveloped for contact-rich manipulation. TLA addresses this gap with a tactile-action dataset and language-grounded model, achieving stronger action accuracy and generalization in peg-in-hole assembly.
- Tactile sensing supports precise contact adjustment in fine assembly and improves manipulation flexibility and robustness.
- Most vision-language-action models focus on visual tasks and lack tactile modalities needed for contact-rich manipulation.
- TLA addresses missing tactile-action resources with a fingertip peg-in-hole instruction dataset and a language-grounded policy-learning model.
- TLA encodes sequential tactile images into a composite image and combines the representation with language-grounded reasoning to generate robot actions.
- 24k tactile sequences and corresponding robot action trajectories support TLA training, while the model achieves over 85% assembly success on 1.6 mm and 1.0 mm clearances after training at 2.0 mm.
II. RELATED WORK
Prior tactile-language work mainly targets perception and does not provide robotic manipulation policies. TLA instead combines simulated tactile-action data for peg-in-hole assembly with a tactile-language-action policy framework.
- Vision-language-action systems commonly tokenize or fuse multimodal information for robotic policies, while some rely on Internet-scale video pretraining.
- Existing tactile-language datasets focus on material or texture understanding, and tactile manipulation work remains limited in scope.
- TLA connects tactile perception to robot actions through a pretrained language model and is presented as a tactile-only language-action model.
- The dataset simulates fingertip peg-in-hole insertion with GelStereo 2.0 sensors and FEM-based visuotactile deformation in NVIDIA Isaac Gym.
- During data collection, tactile image sequences record collision states used to infer 3-DOF corrective actions in x, y, and rz.
- The dataset uses 2.0 mm assembly-clearance pegs, clips actions for policy stability, and converts interactions into instruction-formatted training examples.
IV. TACTILE-LANGUAGE-ACTION MODEL
TLA is built on Qwen2-VL, using its vision transformer to encode tactile inputs and its Qwen2 language model to process multimodal information and generate robot actions.
- TLA uses a ViT for tactile encoding and the Qwen2 language model for multimodal understanding and robot-action prediction.
A. Tactile Encoder
The tactile encoder converts temporally aligned fingertip image sequences into a spatially arranged representation that can be processed by a vision transformer. This design facilitates tactile feature extraction while representing temporal variation spatially.
- The tactile encoder processes two temporally aligned image sequences whose images change with the robot’s contact state.
- TLA synthesizes the tactile sequences into one image, converting temporal information into spatial structure for ViT-based feature extraction.
- Eight left- and right-fingertip images are arranged in a 3 × 3 grid with one white cell and resized to 616 × 616.
- The Qwen2-VL ViT processes the concatenated tactile image, while an MLP compresses tactile features into tokens for the language model.
B. Action Prediction with Language Model
TLA feeds tactile and language tokens into Qwen2 to predict robot actions, while retaining numerical encoding and simplifying decimal-valued actions for training.
- Qwen2 7B serves as TLA’s backbone, receiving tactile tokens and language tokens produced from multimodal inputs.
- The model retains its numerical encoding scheme so pre-trained numerical knowledge can support robotic manipulation.
- Decimal-heavy robot actions are scaled and rounded to integers to reduce redundant tokenizer information and ease training.
C. Training and Inference
TLA is trained with frozen tactile encoding and language-model adaptation using next-token prediction, then generates executable actions sequentially during inference.
- Training: The tactile encoder is frozen and Qwen2 7B is efficiently fine-tuned with LoRA using ground-truth actions as next-token labels.
- Training: The next-token objective predicts each ground-truth action token conditioned on previous action tokens and tactile-image and instruction tokens.
- Inference: During inference, TLA uses beam search to generate action probabilities sequentially until an end token, then converts outputs into executable floating-point actions.
- Evaluation: The experiments assess tactile-only action generation, comparison with imitation learning, generalization across task variations, and control across peg geometries and constraints.
A. Baseline and Metrics
The study compares TLA variants with behavior cloning and diffusion policy using action correctness and L1 error metrics.
- Baselines: Behavior Cloning uses a ResNet-50 policy that maps tactile images to robot actions through supervised training.
- Baselines: Diffusion Policy learns the peg-in-hole assembly policy through a conditional denoising diffusion process.
- TLA variants: Single-Peg TLA is trained on square-peg insertion data, whereas Multi-Peg TLA uses square- and triangular-peg insertion data.
- Metrics: L1 distance evaluates each action dimension using predicted actions and ground truth across samples.
B. Comparison on Single-Peg Inserting Tasks
On the single-peg test set, TLA produces more correct and more precise actions than baselines, while visualizations show strong planar guidance but weaker y-axis translation-rotation predictions.
- Experimental Setup: The single-peg evaluation uses 8k square-peg insertion samples, split into 6k training and 2k test examples.
- Results: TLA has more correct actions than baselines and reduces x-direction L1 by 78% versus the second-best method.Its predicted step lengths are also described as more accurate, implying fewer operation steps.
- Visualization: In Fig. 4, arrow origins encode pose deviations and arrow directions encode predicted action movements across paired action dimensions.
- Visualization: TLA guides planar translation toward the target hole, but translation-rotation views contain more incorrect y-axis actions.The passage attributes this imbalance to 2D tactile images representing x-axis information better than y-axis information.
C. Comparison on Multi-Peg Inserting Tasks
TLA is evaluated on multi-peg insertion using square and triangular pegs for training, with additional square, triangular, round, and hexagonal pegs for evaluation. It achieves the lowest L1 error on seen pegs and retains closely aligned performance on unseen geometries, unlike conventional imitation-learning baselines.
- Experimental Setup: The study trains on 16k square and triangular peg samples and evaluates on 8k additional samples spanning seen and unseen peg geometries.Training uses equal numbers of square and triangular pegs; evaluation includes 4k seen-geometry and 4k round/hexagonal-peg samples.
- Results: TLA achieves the lowest L1 error on in-distribution seen pegs, indicating more precise and stable predicted step lengths.The reported L1 result complements action correctness by measuring the precision and stability of step sizes during execution.
- Results: TLA maintains performance closely aligned between in-distribution and out-of-distribution settings for unseen peg geometries.The result indicates strong generalization when the peg shape changes beyond the training geometries.
- Results: Unlike traditional imitation-learning baselines, TLA does not suffer significant degradation under distribution shift to unseen peg shapes.The comparison reports pronounced performance drops for conventional baselines, while TLA transfers learned manipulation skills to novel configurations.
D. Robotic Insertion Tasks
Robotic insertion experiments test TLA across assembly clearances and peg shapes using repeated physical insertion attempts. TLA achieves superior success and efficiency, adapts to challenging clearances and shapes, and can recover from an initial failed insertion using tactile observations, although triangular insertion exposes a y-axis limitation.
- Experimental Setup: The robotic evaluation tests square-peg clearances of 2.0 mm, 1.6 mm, and 1.0 mm, plus square, triangular, round, and hexagonal pegs at 2.0 mm clearance.Each task is repeated 50 times, and success rate and average steps in successful episodes are reported.
- Results: TLA surpasses the second-best method by 50% in success rate and achieves superior manipulation efficiency across assembly clearances.The comparison includes success rate and execution steps, with TLA remaining superior even at the challenging 1.0 mm clearance.
- Results: Both TLA variants achieve better success rates and fewer steps than baseline methods across peg types.The multi-peg results support generalization across square, triangular, round, and hexagonal insertion tasks.
- Visualization: After an initial failed insertion, TLA uses tactile observations from previous attempts to adjust its actions and succeeds after two further rounds.The visualization illustrates iterative correction during contact-rich insertion.
- Failure Analysis: TLA fails on triangular insertion when the smaller y-axis deviation tolerance coincides with poor 2D tactile representation along that axis.For clearance d, the allowable deviation is 3d along x but only 2.3d along y, increasing the difficulty of y-axis contact interpretation.
VI. DISCUSSION AND LIMITATIONS
The paper presents TLA as a language-grounded model for generalized tactile skills in contact-rich manipulation and reports stronger assembly performance across clearances and peg shapes. It also identifies limitations in temporal tactile encoding, tactile representation choices, action decoding, and validation of sim-to-real transfer.
- Discussion: TLA uses cross-modal fine-tuning to acquire generalized tactile manipulation skills through language grounding.The model is designed for contact-rich manipulation scenarios.
- Discussion: TLA outperforms traditional imitation-learning methods in fingertip tactile peg-in-hole assembly with superior assembly success rates.The reported advantage is accompanied by generalization across assembly clearances and peg shapes.
- Limitations: The model does not rigorously capture tactile temporal information because temporal cues are encoded through spatial arrangements.The paper notes that this representation may not fully exploit the sequential nature of tactile data.
- Limitations: Sim-to-real generalization remains to be evaluated in real-world environments under real-world uncertainties.The authors plan deployment experiments to test transfer from simulation.