Source-linked AI summary
ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition
Shan Luo, Wenzhen Yuan, Edward Adelson, Anthony G. Cohn, Raul Fuentes
TL;DR
The paper addresses texture recognition from vision and tactile images despite differences between the modalities and redundancy in their learned features. It proposes Deep Maximum Covariance Analysis to learn a shared latent space from paired data, achieving recognition above 90% and improving either modality over unimodal learning.
Problem
The paper addresses how to learn shared vision-tactile features while reducing modality differences and redundancy for cloth-texture recognition.
Method
Deep Maximum Covariance Analysis learns modality representations with neural networks and projects them into a joint latent space using covariance-based dimensionality reduction for weakly paired data.
Results
Both vision and tactile sensing achieve recognition accuracy above 90% with DMCA, while the shared representation improves either modality compared with unimodal learning.
Takeaways & Limitations
A shared representation can transfer complementary information between vision and touch for cloth-texture recognition, including when only one modality is available at classification time.
Abstract
from arXiv · showhide
Vision and touch are two of the important sensing modalities for humans and they offer complementary information for sensing the environment. Robots could also benefit from such multi-modal sensing ability. In this paper, addressing for the first time (to the best of our knowledge) texture recognition from tactile images and vision, we propose a new fusion method named Deep Maximum Covariance Analysis (DMCA) to learn a joint latent space for sharing features through vision and tactile sensing. The features of camera images and tactile data acquired from a GelSight sensor are learned by deep neural networks. But the learned features are of a high dimensionality and are redundant due to the differences between the two sensing modalities, which deteriorates the perception performance. To address this, the learned features are paired using maximum covariance analysis. Results of the algorithm on a newly collected dataset of paired visual and tactile data relating to cloth textures show that a good recognition performance of greater than 90\% can be achieved by using the proposed DMCA framework. In addition, we find that the perception performance of either vision or tactile sensing can be improved by employing the shared representation space, compared to learning from unimodal data.
I. INTRODUCTION
Vision and touch provide complementary information, but their differing sensing conditions make shared representation learning difficult. The paper targets this problem in cloth-texture recognition using a deep fusion framework and paired visual-tactile data.
- Vision captures global appearance and texture at a distance, whereas touch provides detailed local texture and haptic information through physical interaction.
- The key challenge is learning a joint latent space that shares features across vision and touch while mitigating modality differences.
- Cloth textures provide a test case because tactile sensing reveals detailed yarn patterns while vision captures related patterns that may be blurry.
- Modality-specific factors, including visual color variation, can hinder recognition, motivating extraction of shared information while eliminating such factors.
- The paper proposes a deep fusion framework using deep neural networks and maximum covariance analysis to learn a joint vision-tactile latent space.
II. RELATED WORKS
Earlier tactile-vision research commonly treated touch as a supporting source for vision, often combining tactile measurements or hand-designed tactile features with visual representations.
- Previous systems typically used tactile sensing to confirm contact or refine visual object hypotheses after extracting visual features.
- Hand-designed tactile features, including image moments, were combined with visual features in earlier multimodal approaches.
B. Multi-modal deep learning
Multimodal deep-learning research learns relationships between modalities such as vision, sound, and text, while texture recognition has mostly remained unimodal. This paper applies multimodal learning to paired visual and tactile texture data.
- Deep multimodal methods have learned cross-modal relationships between vision and sound, images and captions, and other modality pairs.
- Most texture-recognition studies use either visual or tactile data alone, whereas this work explores both tactile images and vision.
- The study uses a stationary high-resolution GelSight sensor, making the task harder than standard approaches based on dynamic contact sensing.
III. GELSIGHT TOUCH SENSOR
GelSight is a high-resolution tactile sensor that records object surface geometry and texture through deformation of an elastomer membrane and camera-based illumination.
- GelSight uses a camera, elastomeric gel, reflective membrane, and RGB LED illumination to capture contacted surface deformation.
- The elastomer deforms to represent surface geometry and texture, which the camera records under directional illumination.
- A photometric stereo algorithm reconstructs a three-dimensional height map from the recorded tactile image.
- The implementation uses a silicone-rubber elastomer, a 960×720 webcam, and a surface-topography recovery system running at over 10 Hz.
- The sensor is less affected by materials’ optical characteristics and can measure rich physical properties through its compliant elastomer gel.
IV. VITAC CLOTH DATASET
The ViTac Cloth dataset pairs visual and tactile measurements for cloth-texture recognition across 100 everyday clothing pieces.
- The dataset contains 100 clothing pieces spanning varied clothing types, fabrics, and textures.
- It includes 1,000 digital camera images, with ten images per cloth collected under different in-plane rotations.
- GelSight tactile data was collected by pressing the sensor normally onto each cloth surface, averaging around 30 contacts per cloth.
- Both modalities were recorded while the cloth lay flat, providing paired visual and tactile data.
V. DEEP MAXIMUM COVARIANCE ANALYSIS
DMCA learns modality-specific deep representations and projects them into a shared latent space by maximizing cross-modal covariance, while inferring pairings for weakly paired data.
- DMCA processes camera and GelSight images through separate nonlinear neural networks before learning their shared representation.
- The method maps both modality datasets into a common lower-dimensional space using functions f and f′ that depend on cross-modal relationships.
- MCA extracts coupled cross-modal variability by constructing a covariance matrix between representations and applying singular value decomposition.
- Because visual and tactile images cannot be fully paired during collection, DMCA introduces a pairing matrix Π to match instances from the two modalities.
- The optimization alternates between singular-value decomposition for projection matrices and linear assignment for instance pairing.
- The learned hidden representations can be high-dimensional, including D = 4,096 for camera-image features, motivating dimensionality reduction.
VI. EXPERIMENTS AND ANALYSIS
The experiments evaluate unimodal and cross-modal cloth-texture recognition on the ViTac Cloth dataset using separate training and test data from the two sensing modalities.
- The evaluation measures standard unimodal classification and cross-modal classification across vision and tactile sensing.
- Cross-modal evaluation trains a model with one sensing modality and applies it to data from the other modality.
- The dataset is divided into training and test subsets using a 9:1 ratio.
- Because the modalities are not fully paired, the experiments use weak pairing based on the cloth surface from which each sample was recorded.
A. Unimodal cloth texture recognition
Unimodal deep networks recognize cloth textures from either tactile or camera images, with camera-based recognition slightly outperforming tactile-based recognition in the reported setup.
- The unimodal model fine-tunes AlexNet and replaces its final layer with 100 outputs corresponding to the texture classes.
- 85.9% accuracy is obtained when camera data are used for both training and testing.
- 83.4% accuracy is obtained when GelSight data are used for both training and testing.
- Cross-validation is used to address overfitting, with a learning rate of 0.001, batch size 128, and 20 training epochs.
B. Cross-modal cloth texture recognition
Cross-modal recognition is possible because camera and GelSight data share image-like texture patterns, but models trained on one modality transfer poorly to the other. Direct cross-modal accuracy reaches only 16.7% for GelSight tests and 14.8% for camera tests.
- 16.7% accuracy is achieved when GelSight test data are evaluated with a model trained on vision data.The corresponding camera-test evaluation using a GelSight-trained model achieves 14.8%.
- 14.8% accuracy is achieved when camera test data are evaluated with a model trained on GelSight data.
- Cross-modal recognition performs much worse than unimodal recognition despite similar cloth patterns appearing in camera and GelSight images.The reported differences include visual scaling, rotation, translation, color variance, illumination, and force-dependent tactile impressions.
C. Shared representation learning for cloth recognition
DMCA learns shared representations from paired vision and tactile data while testing each modality separately. Increasing the shared-space dimension raises performance before it levels off, reaching about 90% for GelSight and 92.6% for camera tests.
- Both camera and GelSight data are available during learning, while only one modality is used during later application to new data.This setting tests whether bimodal learning produces useful low-dimensional representations for unimodal deployment.
- Around 90% classification accuracy is achieved for GelSight test data as the DMCA shared-space dimension increases and then levels off.The shared representation uses complementary visual features to support tactile cloth-texture discrimination.
- 92.6% classification accuracy is achieved for camera test images after performance levels off above dimension 20.The shared representation uses complementary tactile features to support visual cloth-texture discrimination.
- DMCA shared representations improve recognition when either tactile or visual data are used at test time.The results support using the other modality during learning to improve single-modality recognition.
VII. CONCLUSION AND FUTURE WORK
The paper introduces a joint latent space for camera vision and tactile data and evaluates it on the ViTac cloth-texture dataset. DMCA yields over 90% recognition for both modalities and improves unimodal perception through shared representations.
- DMCA learns a joint latent space shared by camera vision and tactile data.
- More than 90% recognition accuracy is achieved for both vision and tactile sensing using DMCA.
- Shared representations improve either modality’s perception compared with learning from unimodal data.Future work includes applying the framework to other modalities and incorporating temporal information.