AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors

Ruoxuan Feng; Jiangyu Hu; Wenke Xia; Tianci Gao; Ao Shen; Yuhao Sun; Bin Fang; Di Hu

Source-linked AI summary

AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors

Ruoxuan Feng, Jiangyu Hu, Wenke Xia, Tianci Gao, Ao Shen, Yuhao Sun, Bin Fang, Di Hu

arXiv:2502.12191v3Feb 15, 2025cs.LGcs.CVcs.RO

TL;DR

Low standardization among visuo-tactile sensors makes unified tactile perception and cross-sensor transfer difficult. The paper introduces TacQuad and AnyTouch, combining aligned multimodal data with static-dynamic representation learning, and reports static, dynamic, and transferability capabilities across sensors and tasks. The authors conclude that this approach supports unified multisensor tactile representations, while the dataset remains limited in scale and sensor diversity.

Problem
Low-standardized visuo-tactile sensors produce discrepant data, hindering unified tactile representations and cross-sensor knowledge transfer.
Method
The paper combines TacQuad aligned multimodal data with AnyTouch, which learns static-dynamic multisensor representations through masked modeling, multimodal alignment, and cross-sensor matching.
Results
AnyTouch demonstrates static and dynamic perception abilities and cross-sensor transferability across tactile datasets and a real-world fine-grained pouring task.
Takeaways & Limitations
TacQuad and AnyTouch provide a unified multisensor tactile representation framework that explicitly integrates sensors and supports knowledge transfer between them.
Takeaways & Limitations
TacQuad currently has limited dataset scale and includes only four sensor types, without studying same-type sensor variation or gel damage.

Abstract

from arXiv · show

Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to precisely understand and manipulate objects. Over time, numerous meticulously designed visuo-tactile sensors have been integrated into robotic systems, aiding in completing various tasks. However, the distinct data characteristics of these low-standardized visuo-tactile sensors hinder the establishment of a powerful tactile perception system. We consider that the key to addressing this issue lies in learning unified multi-sensor representations, thereby integrating the sensors and promoting tactile knowledge transfer between them. To achieve unified representation of this nature, we introduce TacQuad, an aligned multi-modal multi-sensor tactile dataset from four different visuo-tactile sensors, which enables the explicit integration of various sensors. Recognizing that humans perceive the physical environment by acquiring diverse tactile information such as texture and pressure changes, we further propose to learn unified multi-sensor representations from both static and dynamic perspectives. By integrating tactile images and videos, we present AnyTouch, a unified static-dynamic multi-sensor representation learning framework with a multi-level structure, aimed at both enhancing comprehensive perceptual abilities and enabling effective cross-sensor transfer. This multi-level architecture captures pixel-level details from tactile data via masked modeling and enhances perception and transferability by learning semantic-level sensor-agnostic features through multi-modal alignment and cross-sensor matching. We provide a comprehensive analysis of multi-sensor transferability, and validate our method on various datasets and in the real-world pouring task. Experimental results show that our method outperforms existing methods, exhibits outstanding static and dynamic perception capabilities across various sensors.

1 INTRODUCTION

Visuo-tactile sensors support robotic tactile perception, but sensor-specific data discrepancies and limited alignment hinder unified representations. TacQuad and AnyTouch address these gaps through aligned multi-sensor data and static-dynamic representation learning.

Visuo-tactile sensors provide high-resolution tactile information for robotic tasks such as grasping and insertion.
Low standardization causes sensors to perceive the same tactile information differently, limiting single-sensor data scale, diversity, and perception capability.
TacQuad contains 72,606 contact frames from four diverse visuo-tactile sensors with fine-grained and coarse-grained alignment strategies.
AnyTouch learns unified representations from tactile images and videos to capture static properties, dynamic changes, pixel-level details, and semantic-level sensor-agnostic features.
Experiments evaluate multi-sensor transferability and static and dynamic perception across tactile datasets and a real-world fine-grained pouring task.

2 RELATED WORK

Prior work studies multi-source learning, visuo-tactile perception, and representation learning, but tactile sensors remain difficult to integrate because their data representations differ. AnyTouch builds on aligned multi-sensor and multimodal data to bridge this gap.

Multi-Source Learning: Multi-source learning seeks greater data scale and diversity but faces difficulty constructing unified representation spaces across discrepant data sources.
Multi-Source Learning: TacQuad spans four sensor types, including public, self-made, and force-field sensors, to provide aligned data despite noticeable differences between sensor outputs.
Multi-Source Learning: Existing tactile integration methods include joint training, multimodal alignment, and cross-sensor generation, but overlook jointly using multimodal and aligned multisensor data.
Visuo-tactile Perception: Visuo-tactile sensors capture contact deformations for manipulation tasks including dense packing, grasping, and insertion.
Representation Learning: Representation learning methods such as masked autoencoders and vision-language pre-training improve model performance across downstream applications and modalities.

3 ALIGNED MULTI-MODAL MULTI-SENSOR TACTILE DATASET

TacQuad addresses limited prior multisensor tactile datasets by combining diverse sensors, aligned tactile data, visual images, and tactile attribute descriptions. Its collection uses fine-grained calibration and faster coarse-grained handheld alignment to balance pairing accuracy and scale.

Prior paired-sensor data included 32,256 tactile-image pairs from two sensors but limited object variety, task scope, and tactile-property coverage.
TacQuad provides aligned multisensor data with text and images to learn semantic tactile attributes and sensor-agnostic features.
The dataset uses four sensor types spanning public, self-made, and force-field designs, with three sensors capturing tactile images and Tac3D capturing deformation force fields.
Fine-grained alignment: Fine-grained alignment sequentially presses the same object position with four sensors at shared speed and depth using a calibration platform.
Coarse-grained alignment: Coarse-grained handheld collection presses the same object location with twisting motions, producing larger-scale aligned data from 151 sets across 99 objects.
Each tactile frame is paired with a visual image and manually corrected tactile attribute descriptions to support multimodal alignment across sensors.

4 METHOD

AnyTouch unifies static tactile images and dynamic videos through a multi-level architecture that learns pixel-level details, semantic sensor-agnostic features, and transferable sensor information. Its training combines masked modeling, multi-modal alignment, and cross-sensor matching to support a unified representation space.

4 METHOD: The framework combines masked modeling with multi-modal alignment and cross-sensor matching to learn pixel-level details and semantic sensor-agnostic properties.Universal sensor tokens additionally integrate and transfer information across sensors.
4.1 UNIFIED INPUT FORMAT FOR STATIC AND DYNAMIC TACTILE PERCEPTION: AnyTouch processes tactile images as single-frame static videos, replicating images across time so images and videos share a unified 4-D input representation.The unified tensor is converted into spatio-temporal tokens through a shared patch projection layer.
4.2 MASKED MODELING: LEARNING PIXEL-LEVEL DETAILS: Masked modeling reconstructs masked tactile image tokens and video tokens, encouraging fine-grained representations of tactile deformations and temporal changes.The reconstruction losses use mean squared error over masked tokens, with video frames indexed by f.
4.2 MASKED MODELING: LEARNING PIXEL-LEVEL DETAILS: Next-frame prediction is added during dynamic-video training to strengthen the model’s understanding of continuous deformation changes.The prediction loss is mean squared error between the original next frame and its prediction.
4.3 MULTI-MODAL ALIGNING: UNDERSTANDING SEMANTIC-LEVEL PROPERTIES: Text serves as an anchor for aligning touch, vision, and language when datasets have missing modalities or differing collection scenarios.The modality-missing-aware contrastive method aligns the largest available modality intersections within each batch.
4.4 CROSS-SENSOR MATCHING: EXTRACTING SENSOR-AGNOSTIC FEATURES: Cross-sensor matching treats same-object, same-position samples from different sensors as positives and other-object or other-position samples as negatives.This explicitly clusters equivalent tactile information across sensors to form a unified sensor representation space.
4.5 UNIVERSAL SENSOR TOKEN: Universal sensor tokens integrate information from multiple sensors and are randomly substituted for sensor-specific tokens during training.The design targets greater utilization of seen-sensor information when generalizing to new sensors.

5 EXPERIMENTS

The experiments analyze sensor-data transfer, representation-space structure, and static and dynamic perception across seen and unseen sensors. AnyTouch is evaluated through multi-dataset comparisons, t-SNE visualization, and real-world pouring.

Experimental design: Experiments answer how sensor data contributes, what AnyTouch’s unified representation space looks like, and whether it benefits static and dynamic perception.The study uses nine training datasets, four downstream datasets, and qualitative and quantitative analyses.
Sensor transferability: Adding GelSight data substantially improves downstream performance over CLIP, while sequentially adding GelSlim, DIGIT, and GelSight Mini generally benefits unseen sensors more than seen sensors.These results indicate tactile pre-training is transferable across sensors.
Sensor transferability: GelSight-only training performs best on TAG material classification, whereas adding more sensors reduces performance because TAG appears in pre-training and receives a smaller data proportion.The authors relate this effect to greater overlap between downstream and pre-training data.
Static perception on seen sensors: AnyTouch is compared with single-sensor and multi-sensor baselines on seen-sensor datasets, outperforming UniTouch with matched data and achieving the best TAG hardness and roughness results with all data.It also achieves 99.0 when using frames after grasping, a setting noted as easier than frames during grasping.
Static perception on unseen sensors: On unseen-sensor ObjectFolder datasets, AnyTouch trained with matched data outperforms UniTouch, while full-data AnyTouch achieves the highest performance on OF 1.0.The comparison uses linear probing, and OF 2.0 is seen for the corresponding UniTouch and AnyTouch models.
Real-world dynamic perception: In fine-grained pouring, static-only AnyTouch performs comparably to T3, while dynamic-capability integration yields the best performance across 10 real-world test runs.The metric is mean error between poured and target mass; CLIP struggles when its tactile encoder is frozen.

6 CONCLUSION

The paper contributes TacQuad for explicit multi-sensor integration and AnyTouch for unified tactile representation learning across static and dynamic perception. It also examines the resulting representation space and sensor transferability.

TacQuad is an aligned multi-modal multi-sensor tactile dataset enabling explicit integration of various sensors.
AnyTouch learns unified multi-sensor tactile representations from both static and dynamic perception.
The paper explores multi-sensor representation space and sensor transferability.

A.1 TRAINING DATASET STATISTICS

The training phase uses nine datasets collected with five different tactile sensors, spanning multiple sensor types and modalities.

AnyTouch is trained on 9 datasets from 5 different sensors.
The training data include TAG, VisGel, and Cloth from GelSight, plus ObjectFolder Real from GelSlim.

A.2 MULTI-MODAL ALIGNING LOSS

AnyTouch performs multi-modal alignment within each batch using available touch, vision, and text representations. The alignment uses paired modality subsets and a temperature parameter.

The training process selects the largest available subset for each modality combination within a batch for multi-modal alignment.
Alignment operates on touch representations and optionally paired vision and text representations derived from uni-modal encoders.
The alignment formulation uses batch size B, modality-index sets ΩV and ΩL, and scalar temperature τ.

A.3 DOWNSTREAM DATASETS

The downstream evaluation compares AnyTouch with baselines on four datasets covering static tactile perception. The tasks include understanding material, hardness, and roughness.

Static perception is evaluated on four downstream datasets: TAG, Feel, ObjectFolder 1.0, and ObjectFolder 2.0.
TAG includes material, hardness, and roughness classification tasks for tactile-property understanding.
Feel is a robotic dataset from GelSight used in the static perception evaluation.

A.4 BASELINES

The evaluation compares AnyTouch against recent single-sensor and multi-sensor baselines for static perception, and against T3 for real-world dynamic perception. These comparisons include strong cross-sensor and manipulation-task baselines, while accounting for differences in training data and supervision.

Static perception: AnyTouch is compared with VIT-LENS-2, TLV-Link, and Omnibind as single-sensor baselines using GelSight data.
Static perception: UniTouch serves as the multi-sensor baseline because it demonstrates state-of-the-art cross-sensor performance.
Dynamic perception: T3 is the dynamic-perception comparator because it has demonstrated strong manipulation-task capabilities.
Dynamic perception: T3 uses approximately 3M training examples, more than AnyTouch, and incorporates downstream-task labels during pre-training.

A.5 DETAILS FOR FINE-GRAINED DATA COLLECTION

The supplementary experiments detail aligned data collection, GPT-4o tactile annotation, ablations, cross-sensor generation, and stated limitations. Together, these details describe the data pipeline, supporting analyses, and current scope boundaries of AnyTouch.

Fine-grained data collection: The calibration platform uses a movable end effector and a 3D-printed container holding four sensors fixed side by side.
Implementation: Training uses OpenCLIP-Large encoders, an 8-layer 512-dimensional ViT tactile decoder, and three frames per tactile video clip.The first and second stages are trained for 20 and 12 epochs, respectively, on four NVIDIA A800 GPUs.
GPT-4o annotation: GPT-4o generates paired tactile-property descriptions from visual and tactile inputs for Touch and Go, ObjectFolder Real, and TacQuad.For ObjectFolder Real, two visual views and one tactile image are jointly provided because touch details may be obscured.
GPT-4o annotation: GPT-4o extends phrase-level tactile descriptions in TVL and SSVTP using visual images and existing text descriptions.
GPT-4o annotation: A simple rough sampling check found annotations generally accurate when the object was correctly recognized, with errors arising under difficult categorization or occlusion.The authors report that such error cases are relatively uncommon.
Ablation study: Removing TacQuad text consistently reduces downstream performance, indicating that text acts as a bridge between sensors.
Ablation study: Removing cross-sensor matching and universal sensor tokens mainly harms unseen-sensor datasets, while removing stage 2 causes a significant overall decline.The ablation also reports that dynamic-perception joint training consistently improves static perception, while stage 1 remains meaningful for seen sensors.
Cross-sensor generation: Cross-sensor generation evaluates representations by generating DuraGel images and reconstructing Tac3D force fields from other sensor inputs.The comparison includes T3, which uses 3.08M pretraining examples versus AnyTouch’s 2.48M.