Source-linked AI summary
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, Alex Wong
TL;DR
Multimodal tactile learning is limited by expensive data collection and heterogeneous sensor outputs. UniTouch aligns touch with pretrained visual embeddings and uses sensor-specific tokens to learn across sensors, achieving zero-shot capabilities and competitive performance across multiple tasks. Its scope is limited to vision-based tactile sensors, and its representation is not easily interpretable.
Problem
Expensive tactile data collection and non-standardized sensor outputs make it difficult to learn multimodal tactile representations across sensors.
Method
UniTouch aligns touch embeddings with a shared multimodal space through contrastive learning and adds sensor-specific tokens to learn from multiple vision-based tactile sensors simultaneously.
Results
UniTouch unifies zero-shot tactile tasks and achieves competitive (or even better) performance than previously proposed approaches across multiple tasks.
Takeaways & Limitations
UniTouch provides a general-purpose interface that connects touch with vision, language, and sound and expands the range of zero-shot touch-sensing tasks.
Takeaways & Limitations
UniTouch is limited to vision-based tactile sensors, while its learned representation is a black box that is not easily interpretable.
Abstract
from arXiv · showhide
The ability to associate touch with other modalities has huge implications for humans and computational systems. However, multimodal learning with touch remains challenging due to the expensive data collection process and non-standardized sensor outputs. We introduce UniTouch, a unified tactile model for vision-based touch sensors connected to multiple modalities, including vision, language, and sound. We achieve this by aligning our UniTouch embeddings to pretrained image embeddings already associated with a variety of other modalities. We further propose learnable sensor-specific tokens, allowing the model to learn from a set of heterogeneous tactile sensors, all at the same time. UniTouch is capable of conducting various touch sensing tasks in the zero-shot setting, from robot grasping prediction to touch image question answering. To the best of our knowledge, UniTouch is the first to demonstrate such capabilities. Project page: https://cfeng16.github.io/UniTouch/
1. Introduction
Touch supports important cross-modal associations, but learning unified tactile representations is hindered by costly data collection and substantial variation across sensors. UniTouch aligns touch with visual embeddings to connect touch with other modalities and supports zero-shot tasks across sensing, retrieval, generation, and language interaction.
- Touch supports physical interaction and cross-modal predictions, with applications including robotic manipulation, material estimation, assistive technology, and texture recognition.
- Expensive active probing limits tactile training data, while non-standardized sensor designs and materials create divergent outputs that constrain generalization.
- UniTouch aligns tactile signals with visual signals, allowing touch to connect with language, sound, and other modalities through pretrained models such as CLIP.
- UniTouch unifies zero-shot touch understanding, cross-modal retrieval, touch-to-image generation, image stylization, tactile question answering, and X-to-touch generation.
- Competitive (or even better) performance than previously proposed approaches is reported across multiple tasks.
2. Related Work
Related work spans specialized tactile sensors and representations, self-supervised learning for individual sensors, and multimodal methods that align visual content with language, audio, or other modalities.
- Tactile sensing: Vision-based tactile sensors such as GelSight use an elastomeric gel, camera, and illumination to produce high-resolution touch information through deformation and photometric stereo.
- Tactile sensing: Early tactile sensors measured low-dimensional signals including force, pressure, vibration, and temperature, while recent work increasingly focuses on vision-based sensing.
- Representation learning with touch: Initial tactile representation methods targeted specific tasks, including contact-rich manipulation and fabric classification through shared visual-tactile spaces.
- Representation learning with touch: Recent approaches learn general tactile representations using self-supervision, contrastive multiview coding, and contrastive pretraining for sensors such as GelSight and DIGIT.
- Multimodal representation learning: Multimodal representation learning has connected visual content with text, extended CLIP to 3D, and learned shared audio-visual representations from video correspondence.
3. Method
UniTouch learns a unified tactile representation by aligning touch with pretrained visual embeddings and adapting one encoder to heterogeneous sensors. The resulting shared space supports zero-shot and cross-modal tactile applications.
- 3.1. Binding touch with images: UniTouch aligns tactile embeddings with pretrained image embeddings already connected to language and audio, avoiding paired touch-text or touch-audio data.
- 3.1. Binding touch with images: Contrastive learning pulls corresponding touch-image pairs together and pushes mismatched pairs apart using bidirectional alignment losses.
- 3.2. Learning from multiple sensors at once: Learnable sensor-specific tokens capture calibration and background differences while leaving model capacity for shared tactile properties such as texture and geometry.
- 3.2. Learning from multiple sensors at once: The multi-sensor batch sampler concentrates σ·B examples from one dataset and samples the remaining (1−σ)·B examples from other datasets, emphasizing intra-sensor hard negatives while retaining inter-sensor diversity.
- 3.3. Applications: By placing touch in a joint latent space, UniTouch supports zero-shot classification, grasp-stability prediction, touch-language question answering, touch-conditioned image synthesis, and x-to-touch generation.
4. Experiments
Experiments evaluate UniTouch across touch understanding, retrieval, generation, Touch-LLM, and multi-sensor learning. Across these settings, the model shows strong or competitive zero-shot performance and benefits from sensor-specific tokens and sampling.
- Experimental setup: UniTouch is evaluated on zero-shot touch understanding, cross-modal retrieval, image synthesis, Touch-LLM, X-to-touch generation, and multiple tactile sensors.The experiments use four visuotactile datasets from three sensor types and test generalization on out-of-domain datasets with unseen sensors.
- Touch representation: UniTouch outperforms baselines by a large margin in linear-probed material classification across in-domain and out-of-domain datasets and sensors.The reported improvements are consistent across datasets, while the authors attribute the gains to alignment with a large-scale pretrained embedding space and the proposed multi-sensor training design.
- Touch representation: UniTouch consistently outperforms existing baselines in grasping stability prediction, including on the out-of-domain ObjectFolder 1.0 dataset.The task predicts whether a robotic gripper can successfully hold an object before lifting, with failure defined as slipping by more than 3cm.
- Zero-shot touch understanding: Zero-shot prompting yields performance comparable to supervised methods for material classification and grasping stability prediction.The results support using touch-aligned representations with text prompts for touch understanding and robotics tasks beyond the original vision-language model training scope.
- Cross-modal retrieval with touch: UniTouch achieves state-of-the-art performance across vision, language, and sound modalities and outperforms paired-modality supervised methods by a large margin in cross-modal retrieval.This result is presented as evidence that touch can be aligned with other modalities without explicit paired training data or additional supervision.
- Cross-modal generation and interpretation: For touch-to-image generation, UniTouch produces images with higher CVTP and material consistency than compared methods despite a slightly lower FID than Vision-from-touch.The method also supports tactile-driven image stylization, Touch-LLM tasks, and X-to-touch generation, which reaches 55.3% material classification consistency.
- Ablation study: Adding sensor-specific tokens improves performance by 17%, while the proposed sampling strategy improves it by 19% in the multi-sensor ablation.The baseline drops from 43.1% to 21.4% when applied across multiple sensors and datasets; combining both designs addresses inter-sensor and intra-sensor hard samples.
5. Discussion
UniTouch unifies tactile sensing across vision-based sensors and supports zero-shot tasks spanning touch understanding and cross-modal generation. The authors frame it as a step toward integrating touch into multimodal foundation models while acknowledging scope and interpretability limits.
- Discussion: UniTouch aligns touch embeddings into a shared multimodal space and uses sensor-specific tokens to learn across heterogeneous vision-based tactile sensors.The representation is trained with contrastive learning and is intended to preserve sensor-specific details while learning common tactile knowledge.
- Discussion: UniTouch unifies existing tactile tasks and extends touch sensing to zero-shot understanding, cross-modal retrieval, and image, text, and audio applications.Reported applications include material recognition, grasp stability prediction, and cross-modal generation or understanding.
- Discussion: The work is presented as a step toward multimodal touch experiences beyond vision-touch pairing and toward integrating tactile sensing into multimodal foundation models.
- Discussion: The current scope excludes non-vision-based tactile sensors, and the learned representation remains difficult to interpret.The limitation follows from differing output formats among tactile sensors and the representation's black-box nature.
A.1. Datasets and Metrics
The evaluation uses publicly available visuo-tactile datasets spanning real-world, robot-collected, simulated, and out-of-domain settings. These datasets cover multiple sensors, interaction types, materials, and downstream tactile tasks.
- A.1. Datasets and Metrics: Touch and Go provides 13,900 real-world touch instances across approximately 4,000 objects and 20 material types using GelSight data.It supports material classification, touch-conditioned image synthesis, Touch-LLM, and X-to-touch generation.
- A.1. Datasets and Metrics: Feeling of Success contains 9.3k paired vision-touch images from robot grasps captured with GelSight sensors for grasping stability prediction.The dataset is split by objects in an 8:1:1 ratio because no official train, validation, and test split exists.
- A.1. Datasets and Metrics: YCB-Slide contains 180k DIGIT tactile frames from sliding interactions with 10 YCB objects, using simulated visual inputs.
- A.1. Datasets and Metrics: ObjectFolder 1.0 and 2.0 provide simulated tactile data for 100 and 1000 objects, respectively, while ObjectFolder Real provides 20k real visuo-tactile pairs from 100 household objects.ObjectFolder Real is used as an out-of-domain material-classification dataset.
- A.1. Datasets and Metrics: SSVTP contains 4.9k human-collected DIGIT visuo-tactile pairs, primarily involving garments and metal materials, with six annotated material categories.
A.2. Implementation Details
Implementation details describe zero-shot touch-conditioned generation and Touch-LLM integration. The systems reuse pretrained diffusion and language models while injecting aligned UniTouch embeddings as conditioning signals.
- A.2. Implementation Details: Tactile-driven image stylization combines touch and image embeddings with weights 0.3 and 0.7, respectively.The method uses DDIM with guidance scale 9, 50 denoising steps, and 768×768 edited images.
- A.2. Implementation Details: Touch-LLM projects the global UniTouch feature to LLaMA's token-embedding dimension and injects it into the language model through repeated token features.The projected touch feature is added across text tokens and layers using a zero-initialized learnable gate.
- A.2. Implementation Details: The X-to-touch model trains an image-to-touch diffusion model, then supports zero-shot text-to-touch and audio-to-touch by replacing image conditioning with already aligned modalities.
A.3. Evaluation Details
Evaluation details define task-specific procedures for touch-to-image generation, Touch-LLM, and X-to-touch generation. The protocols use perceptual, cross-modal, semantic, and language-based assessments.
- A.3. Evaluation Details: Touch-to-image generation is evaluated with FID, CVTP, and Material Classification Consistency.FID compares real and generated image-activation distributions; CVTP measures visual-tactile embedding similarity; Material Classification Consistency compares material predictions.
- A.3. Evaluation Details: Touch-LLM outputs are evaluated by GPT-4 using the prompt, a human-crafted reference response, and each model's response.
- A.3. Evaluation Details: X-to-touch is evaluated on Touch and Go through material-classification consistency across generated touches from different modalities.Text-to-touch uses 400 human-labeled captions, while audio-to-touch uses manually selected ObjectFolder 2.0 audio for qualitative evaluation.
A.4. Additional Experiments
Additional experiments examine in-batch sampling and demonstrate zero-shot tactile image synthesis, including touch-to-image stylization and X-to-touch generation examples.
- In-batch sampling: σ = 0.75 provides a good trade-off between preserving intra-sensor discrimination and using intra-sensor negatives for zero-shot material classification.The evaluation compares σ values of 0, 0.5, 0.75, and 1 across six datasets.
- Image synthesis with touch: UniTouch embeddings successfully guide zero-shot image synthesis for touch-to-image generation and tactile-driven image stylization.Stable Diffusion is used with the aligned UniTouch embedding for these tasks.
- X-to-touch generation: The experiments include X-to-touch generation from image, text, and audio inputs, with additional examples shown on the Touch and Go dataset.The examples include tactile images generated from corresponding visual, textual, and auditory inputs.
- Touch-LLM: Additional Touch-LLM examples evaluate touch-image question answering, including a highlighted failure case.The evaluation template instructs GPT-4 to judge candidate responses against questions and human-authored reference answers.