PaperlayerView exact revision on arXiv

Source-linked AI summary

Tactile-JEPA: Topology-Aware Self-Supervised Representation Learning for Distributed Tactile Sensors

Elizaveta Kovtun, Matvey Konovalov, Andrey Sakhovskiy, Semen Budennyy

arXiv:2609.24385v1cs.ROcs.AI

TL;DR

Distributed tactile sensors have sparse, irregular taxel layouts that existing vision-focused SSL methods do not explicitly represent. Tactile-JEPA predicts masked taxel embeddings using graph-based masks at local and global scales, and reports lower force and in-hand orientation errors than prior methods.

  • Problem

    Distributed e-skins have sparse, irregular taxel layouts, while existing approaches rely on task labels or privileged simulation information rather than self-supervised pre-training that represents topology.

    Source: §I · p. 1, B. Representation Learning for… · p. 3

  • Method

    Tactile-JEPA predicts embeddings of masked taxel signals from visible ones, sampling local and global target masks over the taxel connectivity graph.

    Source: §I · p. 2, B. Tactile-JEPA: Self-Supervised… · p. 3

  • Results

    Across task types and embodiments, Tactile-JEPA reduces in-hand orientation estimation error by 20.8% and force estimation error by 6.3% over the strongest baselines.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Tactile-JEPA brings effective wide-coverage tactile sensing within practical reach of robot learning.

    Source: §VI · p. 7

Abstract

from arXiv · show

Tactile sensing is an essential modality for robots performing contact-rich, dexterous manipulation, particularly under visual occlusion. While pre-trained image encoders are standard in robot learning pipelines, tactile encoders are still commonly trained from scratch from raw, noisy signals, which might limit their expressivity. Existing self-supervised learning (SSL) approaches focus predominantly on vision-based tactile sensors, leaving distributed electronic skins largely unaddressed. These sensors, however, have a distinctive property: their sensing elements are sparse and irregularly arranged over the surface they cover, which makes direct reuse of visual SSL methods suboptimal. We present Tactile-JEPA, an efficient self-supervised pre-training method that uses the spatial arrangement of tactile sensors to learn topology-aware representations. Specifically, it is trained to predict the embeddings of masked sensing elements from the unmasked remainder, using the sensor connectivity graph to guide spatial masking. Our analysis shows that effective tactile representations require capturing both local contact details and the global state of the tactile surface, which we achieve through dual-scale masking. Across three diverse datasets spanning magnetic and piezoresistive sensors, different robot embodiments, and single- and paired-sensor configurations, Tactile-JEPA reduces force estimation error by 6.3% and in-hand orientation error by 20.8% over the prior state-of-the-art, with consistent gains in other downstream applications, including policy learning. Overall, our results demonstrate that the benefit of tactile sensing depends critically on the quality of encoder pre-training, a problem which Tactile-JEPA addresses directly. Code is available at https://github.com/E-Kovtun/tactile.

I. INTRODUCTION

Tactile-JEPA addresses the lack of topology-aware self-supervised pre-training for distributed tactile sensors, whose sparse, irregular taxel layouts differ from image grids. It uses graph-based masks at local and global scales, with reported gains across tactile tasks and embodiments.

I. INTRODUCTION

  • Tactile sensing is critical for contact-rich manipulation under visual occlusion, where vision-only perception is limiting.The introduction also identifies fragile-object contact as a tactile-intensive scenario.Source: §I · p. 1
  • Distributed e-skins produce multivariate time series over taxels, but existing tactile encoders are commonly trained from scratch on raw, noisy signals.Unlike vision-based sensors, e-skins can be distributed across a hand or body rather than confined to fingertips.Source: §I · p. 1
  • Tactile-JEPA predicts masked taxel embeddings from visible ones, sampling masks over the sensor connectivity graph at local and global scales.Local masks cover compact regions, while global masks span distributed taxels to represent broader contact patterns.Source: §I · p. 2
  • Tactile-JEPA reports consistent gains over prior tactile representation methods across embodiments and tasks, including force and pose estimation, classification, and policy learning.Its mask study also finds that graph-sampled masks outperform image-style blocks, while mixed local and global masks transfer broadly.Source: §I · p. 2

II. RELATED WORK

Prior tactile representation work adapts vision SSL to image-like sensor outputs, while distributed-sensor methods leave topology underused or depend on labels or simulation information. The paper formulates pre-training on short tactile windows using a known sensor layout.

A. Representation Learning for Vision-based Tactile Sensors

  • Vision-based tactile sensors produce images that support vision SSL objectives; Sparsh spans several objectives, while Sparsh-X jointly encodes multiple sensory channels.The cited objectives include masked autoencoding, self-distillation, and joint-embedding prediction.Source: A. Representation Learning for… · p. 2

B. Representation Learning for Distributed Tactile Sensing

  • Distributed-sensor approaches either omit taxel connectivity, train end-to-end on labeled data, or require simulation geometry, motivating self-supervision using sensor signals and known layout.T-DEX reshapes signals as images; Sparsh-skin and STAT use per-taxel location features, while Tactile-GAT and TacGNN use adjacency graphs but train on labels.Source: B. Representation Learning for… · pp. 2–3

A. Problem Formulation

  • The formulation treats a tactile window as a short contact history and assumes a known sensor layout; the frozen encoder produces per-taxel embeddings for downstream task heads.The pre-training objective does not model temporal structure across windows, leaving temporal reasoning to the downstream decoder.Source: A. Problem Formulation · p. 3

B. Tactile-JEPA: Self-Supervised Pre-Training

Tactile-JEPA embeds each taxel’s windowed signal and uses the known sensor layout to predict hidden taxel representations from visible context. Graph-based local and global masks guide this prediction, while an exponential-moving-average target encoder supplies the prediction targets.

Input representation

  • Each taxel’s windowed response is mapped to a shared embedding, making each full window one semantic unit and keeping the representation size independent of frames per window.The shared affine projection is followed by layer normalization; embeddings are stacked across taxels.Source: B. Tactile-JEPA: Self-Supervised… · p. 3

Graph-based mask sampling

  • The fixed taxel graph connects adjacent sensors in the known layout, and graph-sampled masks combine connected local regions with scattered global regions.Local masks grow by graph expansion; global masks select taxels across the graph, and target taxels are excluded from context.Source: B. Tactile-JEPA: Self-Supervised… · pp. 3–4

Encoding and prediction

  • A context encoder embeds visible taxels, while a target encoder processes all taxels and supplies target-region embeddings for prediction.The predictor receives context embeddings and target positions, then predicts embeddings at the target positions.Source: B. Tactile-JEPA: Self-Supervised… · p. 4

SSL objective

  • The self-supervised objective averages mean squared error between predicted and target-encoder embeddings across context–target pairs; only the context encoder is updated by backpropagation.The target encoder is maintained as an exponential moving average of the context encoder weights.Source: B. Tactile-JEPA: Self-Supervised… · p. 4

A. Datasets

Evaluation uses three public tactile datasets spanning different transduction principles, embodiments, and downstream tasks. The DECO-50 evaluation is restricted to its Assembly-task subset.

Dataset selection

  • The evaluation spans Sparsh-skin, Tactile socks, and DECO-50, selected to cover distinct transduction principles, embodiments, and downstream tasks.They represent teleoperated robotic-hand play, human locomotion recorded with wearable socks, and bimanual contact-rich manipulation demonstrations, respectively.Source: A. Datasets · p. 4

DECO-50

  • DECO-50 evaluation uses only the Assembly-task subset, described as the most tactile-intensive subset.Source: A. Datasets · p. 4

Dataset coverage

  • The datasets cover teleoperated robotic-hand play, wearable-sock locomotion, and bimanual manipulation demonstrations.Source: A. Datasets · p. 4

B. Downstream evaluation

Downstream evaluation covers force, pose, and object or action classification on tactile datasets, plus visuo-tactile policy learning on DECO-50. Frozen pre-trained encoders feed lightweight task-specific heads, with temporal heads used where pose requires longer context.

Sparsh-skin

  • Sparsh-skin evaluation measures 3-axis force regression, planar in-hand pose estimation, and classification among 14 manipulated objects.Force and pose use RMSE metrics; pose accuracy counts predictions within 2 cm and 5° of ground truth, while object classification uses top-1 accuracy.Source: B. Downstream evaluation · p. 4

Tactile socks

  • Tactile socks evaluation predicts nine wearer activities and regresses 19 relative joint angles spanning the legs, torso, and arms.Activity classification uses top-1 accuracy; pose uses mean joint RMSE in radians.Source: B. Downstream evaluation · pp. 4–5

DECO-50 policy

  • DECO-50 policy learning predicts chunks of future joint-position actions for bimanual Assembly from binocular images and a preceding tactile-history window.The reported action RMSE is averaged over the 16-step horizon and 12 joints.Source: B. Downstream evaluation · p. 5

Task heads

  • A frozen tactile encoder feeds a separate task head; window-level attentive probes pool taxel embeddings, while pose tasks use a temporal transformer decoder over consecutive windows.The DECO-50 visuo-tactile policy combines pooled tactile embeddings, visual embeddings, and an action token in a bidirectional transformer.Source: B. Downstream evaluation · p. 5

C. Baselines

The baselines include image-style BYOL, masked reconstruction, and self-distillation approaches, as well as training from scratch. Comparisons use short windows or instantaneous frames rather than explicit long-horizon context.

Comparison regime

  • For temporally matched comparisons, baselines use short tactile windows or instantaneous frames rather than explicit long-horizon context.Source: C. Baselines · p. 5

T-DEX

  • T-DEX applies BYOL to an instantaneous tactile frame arranged as a three-channel image, using an ImageNet-initialized AlexNet encoder.The comparison setup supplies the middle frame of each time window.Source: C. Baselines · p. 5

Sparsh-skin and end-to-end baselines

  • Sparsh-skin baselines use masked reconstruction of taxel readings or self-distillation from a more heavily masked view, while end-to-end baselines train randomly initialized encoders with task labels.The MAE and DINO setups retain the tactile data view and transformer encoder used by Tactile-JEPA, apart from their stated input and training differences.Source: C. Baselines · p. 5

D. Experimental Setup

The setup standardizes tactile inputs and trains Tactile-JEPA on short windows with graph-sampled masks, task-specific data splits, and frozen-encoder evaluation. It also reports model training details, repeated-run evaluation, and compute costs.

Data preprocessing

  • Magnetic readings are baseline-corrected and standardized per axis, while piezoresistive signals use dataset-specific scaling procedures.Tactile socks use scalar statistics computed separately for each labeled subset; DECO-50 signals are divided by a fixed per-hand maximum, with statistics computed on training data only.Source: D. Experimental Setup · p. 5

Temporal windowing

  • Tactile windows span about 0.1 s for Sparsh-skin and DECO-50, while Tactile socks use five frames over about 0.36 s.Pose decoders consume 10 consecutive windows for Sparsh-skin and 12 for Tactile socks.Source: D. Experimental Setup · p. 5

Taxel grouping

  • For DECO-50, adjacent groups of four to five taxels are projected to one embedding, reducing the sequence length to approximately N/ng tokens.The two hands contain 2,124 taxels in total.Source: D. Experimental Setup · p. 5

Mask sampling

  • Tactile-JEPA uses one context mask and four targets—two local and two global—with each target covering 10–18% of its sampling domain.The context initially covers 50–90% of taxels; targets are sampled independently, may overlap, and their union is removed from the context.Source: D. Experimental Setup · p. 5

Data splits

  • Data splits are task-specific; DECO-50 is split by complete demonstrations, with training demonstrations shared by encoder pre-training and policy learning.Sparsh-skin force and pose data are disjoint from pre-training, while its object-classification data share trajectories with pre-training.Source: D. Experimental Setup · p. 5

Architecture and training

  • The encoder uses a 12-block, 192-dimensional ViT-Tiny configuration, while the four-block predictor keeps prediction-relevant information in encoder representations.Encoders train for 500 epochs on Sparsh-skin and Tactile socks, and 150 epochs on DECO-50; downstream heads use early stopping.Source: D. Experimental Setup · pp. 5–6

Evaluation protocol

  • Table II compares representation quality across two robotic embodiments and diverse downstream tasks, while Table III compares visuo-tactile policy representations on DECO-50.Evaluation uses three encoder seeds and three or four downstream heads per encoder, with means and sample standard deviations reported.Source: Table II · p. 6, Table III · p. 6, D. Experimental Setup · p. 6

Compute and runtime

  • Tactile-JEPA pre-training uses four A100 80 GB GPUs and takes 0.7–6.1 hours on the reported datasets and encoders.Downstream heads use a single GPU and take 0.2–4.0 hours, depending on the task.Source: D. Experimental Setup · p. 6

E. Results

Tactile-JEPA improves tactile representation results across tasks and embodiments, with reported gains in force and in-hand pose estimation. Policy comparisons and training dynamics show a more mixed picture across methods, alongside faster pre-training than DINO.

Tactile representation evaluation

  • 20.8% lower θ-RMSE and 6.3% lower force RMSE than the strongest baseline on each metric were reported for Tactile-JEPA.On Tactile socks, it also improves action-classification accuracy by 1.28% and reduces full-body pose RMSE by 2.6%.Source: E. Results · p. 6

Tactile-aware policy learning

  • Tactile-JEPA achieves the second-best DECO-50 policy results with stable training and no adaptation, while MAE has the lowest error there but not consistent superiority across datasets.DINO collapses in its standard configuration and becomes viable only after disabling iBOT loss.Source: E. Results · pp. 6–7

Training dynamics

  • Across three datasets, Tactile-JEPA pre-trains stably under the same masking configuration without per-dataset tuning; BYOL collapses on Tactile Socks and DECO-50, and MAE on Tactile Socks.Source: E. Results · p. 7

Computational efficiency

  • Tactile-JEPA pre-trains 65–75% faster than DINO across all datasets while matching or exceeding its downstream performance.Source: E. Results · p. 7

V. ABLATION STUDY

The ablation compares graph-based target-mask scales and context sampling, finding that the mixed-scale setup is a robust general-purpose choice while task performance varies with mask scale.

Masking strategy ablation

  • The 2L+2G setup achieves the best force RMSE and in-hand x-accuracy, and ranks second on DECO-50 close to 4G.4G gives the lowest DECO-50 policy error; the passage relates this to action prediction depending on whole-hand contact state.Source: §V · p. 7

Context-mask comparison

  • Replacing the global context with local context improves force estimation but substantially degrades pose and policy performance.The passage identifies global context as the better general-purpose choice.Source: §V · p. 7

Graph-based sampling

  • With local context, 4L targets outperform I-JEPA masking on force estimation and policy learning, isolating an advantage of graph-based sampling in that comparison.The comparison uses compact connected masks with the same budget, differing in whether masks are sampled over the taxel graph.Source: §V · p. 7

VI. CONCLUSION

Tactile-JEPA uses multi-scale masks sampled over the taxel graph to encode localized tactile cues and whole-hand contact state. Its reported results include transfer across tasks and embodiments, reduced estimation errors, and stable pre-training across sensor types.

VI. CONCLUSION

  • Multi-scale mask sampling over the taxel graph is the method’s key design, encoding localized tactile cues and the hand’s overall contact state.Source: §VI · p. 7
  • The taxel graph is used only during pre-training, so the encoder requires no layout information downstream.Source: §VI · p. 7
  • Tactile-JEPA reduces error over the strongest baselines by 20.8% in in-hand orientation estimation and 6.3% in force estimation, and pre-trains stably across sensor types.Table IV evaluates target-mask sampling strategies on Sparsh-skin force and in-hand pose estimation and DECO-50.Source: §VI · p. 7, Table IV · p. 7

Related papers

Loading 2609.24385v1…