Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks

1 Introduction

• Different sensor designs and task-specific datasets make transferable tactile encoders difficult to build, often requiring new data collection and training from scratch.The authors argue that shared latent information across sensors and tasks makes a transferable representation technically viable and practically desirable.Source: §1 · p. 1
• T3 uses sensor-specific transformer encoders, a shared transformer trunk, and task-specific decoders to learn representations across sensor-task pairings.Decoder type varies by task: transformer for generative tasks, MLP for classification, and CNN + MLP for pose estimation.Source: Figure 1 · p. 2, §1 · p. 2
• Experiments report zero-shot transferability in some pairings, fine-tuning with small domain-specific datasets, and performance scaling with larger networks.The paper also demonstrates T3 as a tactile encoder for longer-horizon manipulation policies.Source: §1 · p. 2
• FoTa is a unified-format dataset of 3,083,452 tactile images collected with 13 sensors for 11 tasks, aggregated from several large open-source tactile datasets and additional in-house data.Source: §1 · p. 2

2 Related Works

• Camera-based tactile sensors vary in form, illumination, markers, and camera design, and a new sensor-task pairing often requires data recollection and network retraining.The paper gives examples including flat, dome-shaped, finger-shaped, compliant, multi-linked, and palm-shaped sensors.Source: §2 · pp. 2–3
• Existing tactile datasets cover varied tasks and objects, including multimodal reasoning, classification, grasping stability, and shape reconstruction.Several datasets contain well over a million images, so the cited shortcoming is diversity rather than quantity.Source: §2 · p. 3
• Most tactile datasets use a single sensor, limiting a pre-trained encoder’s transfer to other sensors despite the availability of large image collections.The paper argues that diverse sensor-task pairings are a prerequisite for general-purpose tactile representations.Source: §2 · p. 3
• Multimodal robot-learning methods combine modality-specific encoders or learn shared representations through aligned inputs, but tactile images from different hardware often lack meaningful alignment.This makes resemblance-based self-supervised learning assumptions difficult to apply to heterogeneous tactile data.Source: §2 · p. 3

3 The FoTa Dataset

• FoTa provides 3,083,452 data points from 13 tactile sensors, with labels for 11 tasks, in an I/O-efficient WebDataset format.It aggregates public datasets and new in-house data; preprocessing and dataset statistics are described in Appendix A.2.Source: §3 · p. 3
• New data from several recently emerged sensors were collected and used for final benchmarking and transferability tests.The named sensors include GelSight Finray, GelSight Svelte, GelSight Wedge, DenseTact 2.0, and GelSight 360.Source: §3 · pp. 3–4
• FoTa visualizations show constituent dataset, sensor, and task distributions alongside example sensor images; similar sensor variants can share encoders.Examples include GelSight17 variants 1–4 and GelSight variants 1–2 sharing encoders; not all FoTa tasks are used to train T3.Source: Figure 2 · p. 4
• The robotic platform records sensor-frame-relative poses while a 7-DoF arm explores an object fixed at a known location.It attaches two tactile sensors to a parallel-jaw gripper and records object IDs and names.Source: §3 · p. 4
• The gantry platform probes a sensor’s surface with six textured probes and records translational poses, forces, and probe IDs.The probes are attached to a force/torque sensor on a 3-axis CNC mill.Source: §3 · p. 4

• T3 pairs one encoder per sensor and one decoder per task with a shared trunk, attaching the components for each sensor-task pairing.The trunk and encoders use ViT in the described implementation; decoders vary with the task.Source: §4 · pp. 4–5
• For tasks needing multiple tactile images, T3 encodes each image separately, concatenates the representations, and passes them to the task decoder.Pose estimation is given as an example requiring two images to predict their relative pose.Source: §4 · p. 5
• Training has two pre-training phases for pixel-level and semantic understanding, followed by an optional task- and sensor-specific fine-tuning phase.Pre-training I uses all FoTa data, while Pre-training II requires suitable task labels available for only a subset.Source: §4 · p. 5
• Pre-training I uses masked autoencoding to reconstruct masked input portions, allowing self-supervision on unlabeled data and data without readily defined losses.A shared decoder reconstructs masked images using normalized patch-wise L2 loss.Source: §4 · p. 5
• Pre-training II supervises task-specific decoders with FoTa labels; classification uses MLP decoders and cross-entropy, while regression uses MSE losses.Regression examples include relative pose estimation and variance-of-Laplacian estimation.Source: §4 · p. 5
• Fine-tuning uses data for one target sensor-task pairing and may be omitted when that pairing is already represented in pre-training.Its architecture is configured as in Pre-training II, but with one target task and sensor.Source: §4 · p. 5
• Figure 3 covers performance across model sizes, training schemes, and fine-tuning-data amounts, plus masking-ratio and classification and pose-transfer tests.Panel (a) varies four sizes, three training schemes, and half or full fine-tuning data.Source: Figure 3 · p. 6

5 Experiments and Discussions

• Object classification and 3D pose-estimation data collected in-house on the gantry system were excluded from pre-training and reserved for evaluation and fine-tuning.Source: §5 · p. 6

5.1 The worthiness of pre-training

• Pre-training improved classification performance by a median 24%; large-model accuracy was 19% higher than tiny, while medium and large were not significantly different.With half the fine-tuning data, pre-trained models outperformed scratch-trained models; medium and large models performed nearly as well with half as with full data.Source: §5.1 · pp. 6–7

5.2 Zero-shot transferability of pre-trained T3 to new sensors or tasks

• Zero-shot transfer gave only minor gains over random guesses for classification but significantly improved pose estimation over dataset averages; fine-tuning raised classification accuracy to 17% and reduced DenseTact2.0 pose RMSE by 5.5mm.GelSight Svelte pose-estimation errors were nearly identical before and after fine-tuning and close to optimal.Source: §5.2 · p. 7

5.3 T3 in long-horizon manipulation tasks

• The insertion task tested three electronics parts with 0.4mm pin-to-hole clearance, using policies that shared robot-state and vision inputs but differed in tactile input.Each policy predicted 3-DoF actions, with success defined as insertion within 30 steps; results averaged 15 episodes.Source: §5.3 · pp. 7–8
• T3-encoded policies achieved higher success rates across all three insertion parts and reduced tactile exploration steps; vision-only policies failed on the two harder parts.Figure 4 reports the highest success rate and lowest average episode length for policies using T3.Source: §5.3 · p. 8, Figure 4 · p. 8

• The two most popular sensors contribute over 50% of FoTa, potentially biasing the trained policy toward those sensors.Source: §6 · p. 8
• T3 encodes individual images and uses explicit labels during semantic pre-training and fine-tuning; sequence learning with sparse or implicit labels remains future work.Source: §6 · p. 8
• T3 currently learns representations only for camera-based tactile sensors; extending it to non-camera-based sensors is future work.Source: §6 · p. 8

• T3 is a tactile representation framework for transfer across multiple sensors and tasks, paired with FoTa, described as the largest and most diverse tactile dataset.Source: §7 · p. 8
• Experiments found improved performance with FoTa-pretrained T3 across various tasks, including longer-horizon manipulation such as sub-millimeter electronics insertion.Source: §7 · p. 8
• T3 transferred to new sensor-task pairings with little fine-tuning.Source: §7 · p. 8

• Figure 5 contrasts original tactile images with encoder attention masks and trunk attention masks.Source: Figure 5 · p. 12
• The joint attention map multiplies the layerwise attention weights, normalizes the result, and applies it as a mask to the original tactile images.Source: §A.1 · p. 12
• Encoder attention highlights contact-object regions, whereas trunk attention spreads more broadly to include sensor-specific features such as markers.Source: §A.1 · p. 12

• FoTa preprocessing converts tactile images to .jpg and labels to .json without modifying the constituent datasets’ images or labels.Source: §A.2 · p. 12
• More than half of VisGel’s tactile images were flat, so the dataset was subsampled using a variance-of-Laplacian threshold of σt = 4.24.Source: §A.2 · p. 12
• Some flat images were retained because they contain information about sensor geometry and illumination design.Source: §A.2 · p. 12

A.3 Data format

• Each constituent dataset is split into train and validation sets and packaged as WebDataset shards of up to 10,000 image–label data points, with scripts for adding data or changing preprocessing.Source: §A.3 · p. 12

A.4.1 7-DoF robotic platform

• The 7-DoF platform uses a Franka arm with a parallel-jaw gripper and two tactile sensors to collect images and robot poses during random six-dimensional object exploration.Source: §A.4.1 · pp. 12–13

A.4.2 3-DoF gantry platform

• The 3-DoF gantry platform probes a tactile sensor with one of six interchangeable textured probes, recording images, probe positions, and force values.Source: §A.4.2 · p. 13

A.5 Network configurations

• T3’s components may use different architectures as long as they remain coherent; the paper specifies the configurations used in its experiments.Source: §A.5 · p. 13
• Sensor encoders use ViT blocks on 224 × 224 crops split into 196 patches, with color jittering and task-dependent cropping and flipping during training.Source: §A.5 · p. 13
• The shared trunk also uses ViT blocks, with its embedding dimension and number of heads matched to the sensor encoders.Source: §A.5 · p. 13
• The masked-autoencoding reconstruction decoder is a ViT, while σ_i regression and classification use three-hidden-layer MLP decoders.Source: §A.5 · p. 13
• Pose decoders remove the classification token, reshape the remaining trunk outputs into a 2D map, and apply ResNet blocks, average pooling, and an MLP.Source: §A.5 · p. 13