Source-linked AI summary
Connecting Touch and Vision via Cross-Modal Prediction
Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba
TL;DR
The paper addresses the scale mismatch between whole-scene vision and localized touch by learning cross-modal predictions between the two modalities. It collects paired data with robots and trains a conditional adversarial model using touch scale and location, obtaining promising results for known and unseen objects.
Problem
Vision observes an entire scene while touch senses only a small object region, making cross-modal prediction between the modalities challenging.
Method
The paper collects paired vision-tactile sequences and uses a conditional GAN that incorporates touch scale and location for prediction in both directions.
Results
Human perceptual studies and objective evaluations show promising cross-modal prediction results, including for known and unseen objects.
Takeaways & Limitations
The learned vision-touch connection may support applications such as object recognition, grasping in low light, and physical scene understanding.
Abstract
from arXiv · showhide
Humans perceive the world using multi-modal sensory inputs such as vision, audition, and touch. In this work, we investigate the cross-modal connection between vision and touch. The main challenge in this cross-domain modeling task lies in the significant scale discrepancy between the two: while our eyes perceive an entire visual scene at once, humans can only feel a small region of an object at any given moment. To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data as input. To accomplish our goals, we first equip robots with both visual and tactile sensors and collect a large-scale dataset of corresponding vision and tactile image sequences. To close the scale gap, we present a new conditional adversarial model that incorporates the scale and location information of the touch. Human perceptual studies demonstrate that our model can produce realistic visual images from tactile data and vice versa. Finally, we present both qualitative and quantitative experimental results regarding different system designs, as well as visualizing the learned representations of our model.
1. Introduction
The paper connects vision and touch through cross-modal prediction, addressing the large scale mismatch between whole-scene vision and localized tactile sensing. It introduces paired prediction tasks, a robotic vision-tactile dataset, and a conditional adversarial model that incorporates touch scale and location.
- Vision and touch share physical cues such as geometry, texture, roughness, and hardness that can support cross-modal transfer.
- The paper introduces vision →touch synthesis of temporal tactile signals and touch →vision prediction of the touched object and object part.
- 12,000 touches on 195 objects produced 3 million paired visual and tactile images for training cross-modal models.
- The model incorporates touch scale and location to address the vision-tactile scale gap and uses data rebalancing to reduce mode collapse.
- Evaluations include human perceptual studies, touch-location accuracy, GelSight deformation, ablations, and learned-representation visualizations.
2. Related Work
Prior work learns shared representations and predicts across modalities, but vision-touch research lacked large-scale paired data. This paper addresses that gap through automated robotic data collection and builds on image-to-image translation methods.
- Cross-modal learning has been applied to shared embeddings and prediction across words, images, audio, video, and text.
- Vision-touch prediction was limited by the lack of large-scale paired cross-domain datasets.
- The paper uses robots to automate collection of paired vision-touch data.
- The model builds on image-to-image translation with adversarial training to generate photorealistic outputs in a target domain.
3. VisGel Dataset
VisGel is a synchronized vision-tactile dataset collected with robotic arms, GelSight, and a webcam across diverse everyday objects. Its design supports tactile measurements and cross-modal modeling at scale.
- The dataset contains 195 everyday objects spanning food, tools, kitchen items, fabrics, and stationery.
- KUKA robotic arms, a GelSight sensor, and a webcam collect and synchronize tactile images with scene videos using timestamps.
- GelSight measures contact texture and geometry at high spatial resolution through a deformable reflective membrane and an internal camera.
- The collection uses 165 training objects and 30 seen plus 30 novel test objects, with scenes containing 4–10 randomly placed objects.
- The dataset provides synchronized tactile and RGB images for 195 objects and is described as the largest vision-touch dataset.
4. Cross-Modal Prediction
The method extends conditional GANs for vision–touch prediction by addressing scale mismatch, temporal alignment, reference conditioning, and data imbalance. It predicts tactile signals from vision and visual outputs from touch, with reference images and temporal cues improving correspondence and rebalancing improving diversity.
- 4.1. Conditional GANs: The baseline conditional GAN produces unrealistic, artifact-prone outputs that often fail to align with the input signals.
- 4.2. Improving Photorealism and Accuracy: Reference images reduce the scale-gap challenge by helping localize touch in vision-to-touch prediction and constrain extrapolation in touch-to-vision prediction.Without references, the model must infer a tiny touched region from a full scene or extrapolate from a tiny tactile patch to an entire image.
- 4.2. Improving Photorealism and Accuracy: Data rebalancing targets mode collapse caused by approximately 60% non-contact recordings and improves qualitative results and evaluation performance.The method samples training pairs using a rarity score derived from the difference between current and reference tactile images.
- 4.2. Improving Photorealism and Accuracy: Using five nearby frames addresses temporal mismatch between predicted and input sequences.The sampled sequence is x̄_t = {x_t−4, x_t−2, x_t, x_t+2, x_t+4}, with grayscale inputs except for the RGB reference image.
- 4.2. Improving Photorealism and Accuracy: The full model uses temporal frame sequences, reference vision and tactile images, and an encoder–decoder generator for cross-modal prediction in both directions.The generator combines two ResNet encoders with a decoder, while the discriminator receives the same temporal and reference conditioning.
5. Experiments
Experiments evaluate cross-modal prediction with perceptual studies, contact-timing and deformation metrics, qualitative examples, and representation visualizations. The model generally outperforms baselines, benefits from temporal cues, and produces plausible predictions for both directions.
- Evaluation: The evaluation measures tactile realism, contact timing, marker deformation, touch-location plausibility, and learned representations across vision-to-touch and touch-to-vision tasks.Human perceptual studies complement objective measures and qualitative analyses.
- Vision →Touch: 8,000 judgments over 1,250 results show the full method outperforming baselines for tactile realism on seen and unseen objects.Participants selected which predicted tactile video better matched the visual input in a real-versus-fake test.
- Vision →Touch: Temporal cues improve contact-moment and marker-deformation prediction, while the single-frame model performs worst on deformation.Without temporal cues, predictions can miss or shift the contact event because of temporal misalignment.
- Representation analysis: Learned representations highlight meaningful geometric cues, including sharp edges, flat surfaces, and distinct geometric patterns.These visualizations indicate that internal units attend to modality-relevant structure.
- Touch →Vision: The model produces plausible touch locations, and its self-supervised predictions are comparable to a fully supervised method.Touch-location similarity is judged by humans because multiple positions can produce similar tactile sensations.
6. Discussion
The paper connects vision and touch with conditional adversarial networks and reports promising cross-modal predictions for known and unseen objects. It suggests potential relevance to downstream vision and robotics applications.
- Discussion: Conditional adversarial networks produce promising vision-touch predictions for both known and unseen objects.The discussion identifies possible applications including object recognition, grasping in low light, and physical scene understanding.