Source-linked AI summary
Connecting Look and Feel: Associating the visual and tactile properties of physical materials
Wenzhen Yuan, Shaoxiong Wang, Siyuan Dong, Edward Adelson
TL;DR
The paper asks how machines can infer physical properties needed for interacting with materials. It jointly trains CNNs on color, depth, and GelSight data to associate modalities through fabric embeddings, finding that visual and tactile information can support cross-modal matching and improve visual recognition.
Problem
Machines need to understand physical properties, not only categories or descriptors, to interact more actively with physical objects.
Method
The study jointly trains separate CNNs on color, depth, and GelSight images so each modality produces an embedding representing fabric properties.
Results
The networks match visual and tactile images from the same fabric, and joint touch-depth training outperforms a single-modal model on the size-limited dataset.
Takeaways & Limitations
Fabric properties can be associated across vision and touch, while extra modality information can improve single-modality material matching.
Takeaways & Limitations
The physical-property labels do not cover all properties influencing drape and contain human bias.
Abstract
from arXiv · showhide
For machines to interact with the physical world, they must understand the physical properties of objects and materials they encounter. We use fabrics as an example of a deformable material with a rich set of mechanical properties. A thin flexible fabric, when draped, tends to look different from a heavy stiff fabric. It also feels different when touched. Using a collection of 118 fabric sample, we captured color and depth images of draped fabrics along with tactile data from a high resolution touch sensor. We then sought to associate the information from vision and touch by jointly training CNNs across the three modalities. Through the CNN, each input, regardless of the modality, generates an embedding vector that records the fabric's physical property. By comparing the embeddings, our system is able to look at a fabric image and predict how it will feel, and vice versa. We also show that a system jointly trained on vision and touch data can outperform a similar system trained only on visual data when tested purely with visual inputs.
1 MIT, 2 Tsinghua University
The listed affiliations include MIT and Tsinghua University.
- The authors’ affiliations include MIT and Tsinghua University.
1. Introduction
The paper addresses how machines can infer physical material properties for interaction, using fabrics whose visual and tactile signals reflect shared underlying parameters. It collects complementary visual and tactile observations and learns cross-modal associations through neural-network embeddings.
- Machines need physical-property understanding, beyond passive image categorization, to interact more actively with objects.
- Fabric appearance and touch are linked because shared mechanical parameters shape both visual forms and tactile interactions.
- Neural networks learn an embedding intended to capture fabric properties and use embedding distances to associate visual and tactile inputs.
- The study compares color, depth, and GelSight images gathered from fabrics draped over a cylinder or pressed at a fold.
- The main task is deciding whether two color, depth, or tactile images came from the same fabric.
2. Related work
Prior work establishes visual and tactile cues for material perception, fabric representation challenges, GelSight sensing, and shared-embedding methods. The paper builds on these strands while associating three modalities.
- Human studies found that color and three-dimensional folds are important visual cues for perceiving fabric properties.
- Fabric tracking and representation are difficult because deformable shapes and motion are uncertain and complex.
- GelSight provides high-resolution tactile images used for material-class recognition and estimating hardness of soft objects.
- Siamese and joint neural networks learn embeddings or shared spaces across inputs, while adding modalities can increase learning difficulty but provide more information.
3. Dataset
The dataset contains 118 diverse fabric samples with visual, tactile, and human-labeled physical-property information. Samples are imaged under repeated draping and pressing conditions and grouped into eight property-based clusters.
- The dataset contains 118 apparel, bedding, and functional fabrics with varied colors and patterns.
- Each fabric is photographed in ten cylindrical drapes using both color and depth cameras.
- GelSight captures tactile image sequences while pressing fabrics on a flat surface, using both flat and folded samples.
- Human and physical measurements label thickness, stiffness, stretchiness, and density, though the labels omit some properties and include human bias.
- K-means groups fabrics into eight clusters based on their measured physical parameters.
4. Associating Vision and Touch
The paper associates depth, color, and tactile fabric observations by mapping each modality through separate CNNs into comparable embeddings. Joint, auxiliary, and multi-input architectures train these representations so distances indicate fabric similarity across inputs and modalities.
- Cross-modal Net: Separate CNNs map depth, color, and GelSight images into fc7 embedding vectors representing each fabric.The cross-modal network uses AlexNet-based modality-specific CNNs, with fc7 as the embedding vector E.
- Cross-modal Net: Contrastive training pulls embeddings from the same fabric together and separates embeddings from different fabrics using a margin.The loss uses Y = 0 for inputs from the same fabric and Y = 1 for different fabrics; dissimilar inputs contribute when their distance is below m = 2.
- Auxiliary Net: The auxiliary network adds fabric-cluster classification losses to supervise the shared embeddings and bring similar fabrics closer.Three cross-entropy cluster losses are combined with the contrastive loss, with clusters based on human labels.
- Multi-input Net: The multi-input network aggregates three GelSight presses by taking the element-wise maximum of their embedding vectors.The design uses repeated tactile inputs to exploit information from multiple presses.
- Training: The networks are trained for 25,000 iterations with Adam while fixing AlexNet parameters before fc7.Training uses learning rate 0.001 and batch size 128.
- Cross-modal Net: At test time, embedding distances compare inputs from the same or different modalities to estimate whether they depict the same fabric.Each depth, color, or GelSight image is processed by its corresponding network before cross-input distance comparison.
5. Experiments
The experiments evaluate cross-modal fabric matching and embedding quality across vision and touch inputs. Joint and richer-input models generally improve matching and represent fabric similarities more accurately, especially within visually or mechanically similar groups.
- 5.1. Infer Touch from Vision: The auxiliary net with three tactile frames performs best, followed by the one-frame auxiliary net, while the plain-press basic model performs worst; depth matching exceeds color matching.All networks outperform average chance in matching experiments.
- 5.2. Representing Fabrics by Embeddings: Embedding distances usually confuse fabrics with similar properties, and the Multi-input Net produces the best confusion distribution.Examples include satin matches among satin samples and fleece confusion with other thick, soft blankets.
- 5.2. Representing Fabrics by Embeddings: Within fabric clusters, multiple touch presses, folded-fabric inputs, and human-derived clustering information improve representation and reduce confusion between individual fabrics.The Multi-input Net concentrates confusion matrices most effectively in the evaluated thin-stiff and thick-stiff clusters.
- 5.1. Infer Touch from Vision: Joint training on touch and depth images outperforms a single-modal SNN model on the size-limited dataset.The authors attribute this to complementary modality information reducing overfit and helping optimization.
6. Conclusion
The work uses deep learning to associate visual and tactile information for recognizing fabrics through embeddings that represent physical properties. Joint visual–tactile training improves visual information matching.
- Joint training with visual and tactile information greatly improves performance on visual information matching.
- The system uses depth, color, and GelSight images to recognize fabrics through embeddings representing physical parameters rather than discrete labels.
- Embedding distances indicate whether inputs likely come from the same fabric or from similar fabrics.
- Fabric folds during touching, presumed human-label clusters, and multiple touch inputs help the network learn better embedding vectors.