PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions

Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak

arXiv:2604.11579v1cs.CV

TL;DR

Tactile localization requires identifying image regions with the same material properties as a tactile input, but global visuo-tactile alignment misses these local correspondences. This paper learns dense local alignment, expands training diversity with in-the-wild multi-material images and material-diversity pairing, and substantially outperforms prior methods across new and existing benchmarks.

  • Problem

    Existing visuo-tactile methods mainly learn global alignment, limiting their ability to identify where a tactile property appears within a visual scene.

  • Method

    The approach learns dense local visuo-tactile alignment, uses diverse scene-level image pairing, and constructs two tactile-grounded material segmentation datasets.

  • Results

    The model substantially outperforms prior visuo-tactile methods and baselines across new and established tactile-localization benchmarks.

  • Takeaways & Limitations

    Fine-grained local alignment and visual dataset diversity support robust touch-conditioned material segmentation, including under weak tactile inputs.

  • Takeaways & Limitations

    The model struggles to localize large highly transparent regions and can mistake painted areas for separate visual categories despite tactile equivalence.

Abstract

from arXiv · show

We address the problem of tactile localization, where the goal is to identify image regions that share the same material properties as a tactile input. Existing visuo-tactile methods rely on global alignment and thus fail to capture the fine-grained local correspondences required for this task. The challenge is amplified by existing datasets, which predominantly contain close-up, low-diversity images. We propose a model that learns local visuo-tactile alignment via dense cross-modal feature interactions, producing tactile saliency maps for touch-conditioned material segmentation. To overcome dataset constraints, we introduce: (i) in-the-wild multi-material scene images that expand visual diversity, and (ii) a material-diversity pairing strategy that aligns each tactile sample with visually varied yet tactilely consistent images, improving contextual localization and robustness to weak signals. We also construct two new tactile-grounded material segmentation datasets for quantitative evaluation. Experiments on both new and existing benchmarks show that our approach substantially outperforms prior visuo-tactile methods in tactile localization.

1. Introduction · 2. Related Work · 3. Methodology

The paper defines tactile localization as identifying image regions matching a tactile input’s material properties and proposes Seeing Through Touch, which learns local visuo-tactile alignment for tactile-grounded material segmentation. It addresses limitations in prior global-alignment methods and low-diversity datasets through dense feature interactions, diverse material pairing, in-the-wild images, and two new segmentation datasets.

  • 1. Introduction: Tactile localization identifies image regions sharing a given tactile input’s material properties, framing material segmentation as tactile-conditioned rather than purely visual.The task requires discovering cross-modal correspondences from tactile input instead of using a query pixel within the same image.
  • 1. Introduction: Seeing Through Touch learns fine-grained local visuo-tactile alignment through dense cross-modal feature interactions and produces tactile saliency maps for material segmentation.Prior approaches primarily align pooled representations or CLS tokens, capturing coarse correspondence across entire samples.
  • 1. Introduction: Existing datasets are dominated by close-up, texture-centric images with nearly identical frames, providing limited contextual diversity and few effective image–tactile pairs.The Touch-and-Go images often show one material filling almost the entire scene, making the benchmark ineffective for tactile localization.
  • 1. Introduction: The paper constructs two tactile-grounded material segmentation datasets because existing resources lack image–tactile pairs with corresponding segmentation maps.One dataset extends Touch-and-Go with pixel-level material segmentations, while the other is newly curated.
  • 2. Related Work: Related visuo-tactile representation learning has progressed from jointly trained CNNs toward self-supervised and multimodal contrastive learning, while localization methods pursue region correspondence from tactile signals.The proposed task differs from pixel-query material segmentation because tactile input comes from another modality.
  • 3.1. Preliminaries: The model uses backbone-plus-aligner encoders to map visual and tactile inputs into shared C-channel feature maps, then averages tactile features and computes spatial inner-product similarity maps.The final cross-modal similarity score is obtained by max pooling the similarity map.
  • 3.2. Training Pairs: Training pairs exploit material diversity by sampling tactile and image frames across same-category instances and pairing tactile frames with labeled out-domain images of matching material categories.This extends beyond highly similar frames within a single touch instance and leverages all additional labeled images for alignment.
  • 3.3. Material Diversity-Based Pairing: The method collects scene-level multi-material web images using LLM-generated contextual queries, filters confusing categories with CLIP and human refinement, and trains DINOv3-based encoders with aligners on 224 × 224 inputs.The image backbone is frozen while the image aligner and entire tactile encoder are optimized; training uses 4 A5000 GPUs and an effective batch size of 64.

4. Experiments

Experiments evaluate tactile localization on three benchmarks using mAP and mIoU, showing consistent gains from local alignment, in-the-wild visual diversity, and material-diversity pairing. Additional analyses demonstrate robustness to weak tactile signals and reliable interactive localization across changing tactile inputs.

  • Test Sets and Metrics: Evaluation uses TG-Test, Web-Material, and OpenSurfaces, with mAP and mIoU as the primary metrics.TG-Test contains 579 samples across 18 categories; Web-Material contains 675 samples across 18 categories; OpenSurfaces contains 211 samples across 13 categories.
  • Main Results: +5.21 mIoU on Web-Material and +2.63 mIoU on OpenSurfaces result from incorporating additional in-the-wild images.The gains are attributed to greater visual diversity and enhanced semantic alignment between modalities.
  • Qualitative Results: Qualitative comparisons show that local-alignment models accurately localize tactile correspondences, unlike TVL and vision-only attention maps.Material-diversity pairing further improves localization and supports accurate detection of small regions such as Plants in OpenSurfaces.
  • Weak-Signal Robustness: Models are evaluated on Start, Middle, and End tactile frames because contact strength is weaker at the beginning and end of each touch.The analysis tests whether the encoder can capture weak tactile signals, while material-diversity pairing is assessed for robustness across signal-strength variations.
  • Interactive Localization: Interactive qualitative results show that the model shifts its localized region appropriately when the tactile input changes, while TVL fails to do so.The material-diversity pairing variant further improves interactive localization accuracy.
  • Interactive Localization: TVL and STT-CLS fall from 32.16 and 34.74 mIoU to 1.0 and 4.0 IIoU in interactive localization.These results indicate that global-alignment methods nearly fail when distinguishing multiple tactile signals, whereas the local-alignment objective is essential.

5. Conclusion and Discussion

The paper introduces a tactile-localization framework that learns fine-grained alignment between tactile signals and visual scenes. It combines dense local cross-modal interactions, in-the-wild multi-material images, and material-diversity pairing to address limitations of global-alignment methods and existing datasets.

  • Framework: The framework learns fine-grained alignment between tactile signals and visual scenes for tactile localization.Its core mechanism is dense local cross-modal feature interaction.
  • Framework: Dense local cross-modal feature interactions replace the global-alignment objectives used by existing visuo-tactile methods.The approach is designed to overcome the limitations of global-alignment objectives.
  • Data strategy: The approach addresses current dataset constraints through in-the-wild multi-material images and a material diversity-based pairing strategy.These components are presented as ways to overcome limitations of current visuo-tactile datasets.

7 . Clarifying Touch Instances in Touch-and-Go … 8. Details on Our Web-Material Dataset

The appendix clarifies how Touch-and-Go frames are regrouped into temporally coherent Touch Instances and details the construction and scale of the web-material dataset. The dataset combines LLM-generated, viewpoint-diverse web queries with CLIP filtering, minimal human verification, and annotated training, test, and interactive-localization samples.

  • 7. Clarifying Touch Instances in Touch-and-Go: The Touch-and-Go dataset contains approximately 246k visuo-tactile image pairs and 13.9k detected touches.Its official contrastive-learning split contains 91,982 training and 29,879 testing samples and excludes “Inconclusive” examples.
  • 7. Clarifying Touch Instances in Touch-and-Go: A Touch Instance is a temporally contiguous sequence of frames from one video sharing the same category label.The dataset contains consecutive sequences corresponding to a single press-and-release interaction.
  • 7. Clarifying Touch Instances in Touch-and-Go: Touch Instances are recovered by regrouping samples by video ID, sorting frame indices, and merging uninterrupted consecutive frames with identical labels.Each instance is represented by its frame range, duration, and category.
  • 8.1.1. Image Collection: For each tactile category, an LLM generates descriptive context queries to retrieve relevant and diverse web images.The queries extend beyond simple class names.
  • 8.1.1. Image Collection: Concept queries combine category, object, and place terms, supplemented by close-up prompts to increase scene and viewpoint diversity.The two formats are {category} + {object} + {place} and “A close-up shot of {category} + {object}.”
  • 8.1.1. Image Collection: A custom-built Gradio tool manages category-specific query generation, image collection, and duplicate removal.These operations are shown in the top and middle panels of Figure 10.
  • 8.1.2. Image Filtering: After CLIP-based filtering, two human annotators minimally verify whether images contain objects corresponding to the target category.The annotation interface is provided through the Gradio tool.
  • 8.2. Dataset Distribution: The web-material dataset contains 32,107 training samples across 18 categories, with 675 annotated test samples and 200 masks in Interactive Localization.The Interactive Localization set uses 100 samples with an additional segmentation category annotated in each scene.

9. Material Diversity-based Pairing · 10. Implementation Details

The paper compares three material-diversity pairing strategies that combine Touch-and-Go and Web-Material data. It also specifies staged training with AdamW, pretrained DINOv3 Small encoders, backbone freezing, and curriculum learning across 100 and 50 epochs.

  • 9. Material Diversity-based Pairing: Three pairing strategies leverage Touch-and-Go aligned visuo-tactile data and Web-Material vision-only images with diverse material contexts.The strategies are illustrated according to Touch Instance and Material Diversity pairing schemes.
  • 9. Material Diversity-based Pairing: Touch Instance Pairing matches visual and tactile samples from the same touch instance in Touch-and-Go.
  • 9. Material Diversity-based Pairing: In-domain Pairing matches samples from different Touch-and-Go touch instances within the same category.
  • 9. Material Diversity-based Pairing: Out-domain Pairing combines Web-Material visual images with Touch-and-Go tactile information using matching categories.
  • 10. Implementation Details: Training uses AdamW with β = (0.9, 0.95), weight decay 0.05, base learning rate 1 × 10−5, and effective batch size 64.
  • 10. Implementation Details: Both encoders use pretrained DINOv3 Small models, with both backbones frozen for the first 3 epochs while only aligners are optimized.The tactile backbone is then unfrozen while the image backbone remains frozen for the rest of training.
  • 10. Implementation Details: Because Web-Material has greater variation in target-material appearance, especially object scale and spatial location, the model uses curriculum learning.The model first trains on Touch-and-Go alone for basic alignment, then introduces the more challenging data for further optimization.
  • 10. Implementation Details: The curriculum stages use 100 and 50 epochs, respectively.

11. Details on Evaluation

The evaluation uses refined tactile-grounded test sets designed to provide comprehensive material masks and reduce leakage or redundancy. For datasets without paired tactile signals, category prototypes average tactile features across start, middle, and end frames.

  • Touch-and-Go Dataset: TG-Test contains 579 Touch Instances after leakage-prevention refinement, chunk extraction, and redundancy filtering, with segmentation masks annotated for every test sample.The test-set filtering particularly targets redundancy in the test set.
  • OpenSurfaces Test Set: The refined OpenSurfaces test set groups images and masks by material category, removes negligible mask coverage, and manually verifies comprehensive matching-material masks.The original OpenSurfaces dataset is unsuitable because it annotates only selected salient regions rather than all regions matching the ground-truth material.
  • Web-Material Test Set: The Web-Material test set contains 657 samples, each annotated with a mask for its corresponding tactile category.This set, like OpenSurfaces, lacks corresponding tactile signals.
  • Tactile Prototype Construction: For Web-Material and OpenSurfaces, the method forms category tactile prototypes by averaging features from start, middle, and end frames across corresponding Touch Instances.The final category prototype averages the three frame-specific prototypes, providing a simple strategy when selecting one tactile frame is non-trivial.

12. Comparison with Cascaded System

The paper compares its end-to-end model with a cascaded system combining UniTouch tactile classification and Grounded-SAM visual grounding. Despite the strong segmenter, the cascaded approach underperforms, indicating tactile localization requires a standalone solution rather than classification followed by vision-only segmentation.

  • Cascaded versus End-to-End Methods: The cascaded baseline classifies tactile input with UniTouch [43] before feeding the predicted category into Grounded-SAM, yet underperforms the proposed model.This result shows that tactile localization cannot be reduced to tactile classification followed by vision-only segmentation.

13. Material Classification on the Original Split · 14. Ablation on Tactile Backbone

On the original Touch-and-Go split, the method achieves the highest material-classification accuracy among compared tactile representation-learning methods. Ablation results further show that initializing the tactile backbone with DINO vision pretraining outperforms T3 initialization in both local and out-domain settings.

  • 13. Material Classification on the Original Split: Table 6 reports material-classification linear-probing accuracy.
  • 13. Material Classification on the Original Split: The evaluation uses the original Touch-and-Go train-test split for fair comparison.
  • 13. Material Classification on the Original Split: The method achieves the highest accuracy among the compared tactile representation-learning methods.This result indicates discriminative material representations while enabling tactile localization.
  • 14. Ablation on Tactile Backbone: Table 7 evaluates tactile-backbone initialization in local and out-domain settings.
  • 14. Ablation on Tactile Backbone: Because tactile signals come from vision-based tactile sensors, the tactile backbone can use either tactile- or vision-pretrained initialization.
  • 14. Ablation on Tactile Backbone: DINO initialization outperforms T3 initialization in both local and out-domain settings.The result suggests that texture and local representations from DINO’s large-scale pre-training are effective for interpreting tactile frames.

15. Additional Qualitative Results

Additional qualitative results show consistent tactile localization across diverse test environments, interactive queries, real-world robotic scenarios, illumination changes, and material replacements. The analysis also identifies limitations with transparent objects and visually dominant painted surfaces.

  • Localization Results on Test Sets: The model maintains consistent localization performance across TG-Test, Web-Material, and OpenSurfaces environments and material textures.Additional results are presented for all three test sets.
  • Interactive Localization Results: Using two tactile signals on one image, the model precisely distinguishes and localizes different material properties within a single scene.The two localization outputs are represented in red and green.
  • Real-World Scenarios: The model accurately localizes tactilely queried target items in warehouse collection and robotic recycling scenes despite distorted 360◦ imagery or conveyor-belt clutter.The use cases include one-touch material-based item collection and material-based robotic recycling.
  • Illumination Change Results: Predictions remain robust under strong artificial color-contrast variations and natural daytime-to-nighttime illumination differences.These qualitative tests are shown in Figures 20 and 21.
  • Material Replacement Results / Failure Cases: Material replacement results show accurate target-region localization when geometry is unchanged, while failures occur for transparent objects and painted surfaces whose visual cues dominate tactile equivalence.The model can localize distinct tactile materials despite identical object shape, but may localize only unpainted concrete regions on painted surfaces.
Loading 2604.11579v1…