Sparsh: Self-supervised touch representations for vision-based tactile sensing

1 Introduction

• Vision-based tactile images capture contact geometry, texture, and forces at the sensor-object-environment interface, supporting tasks including insertion, grasping, localization, and pose estimation.These contact properties can be difficult to access through vision alone.Source: §1 · pp. 1–2
• Custom labeled models are inefficient across sensors and tasks, while collecting ground truth for forces, slip, and other properties can be expensive or infeasible.GelSight marker-based features may not transfer to markerless sensors, and texture-focused encoders may not suit force or slip reasoning.Source: §1 · p. 2
• Sparsh is trained on 460k+ tactile images, and TacBench evaluates six tasks spanning tactile properties, physical perception, and robot manipulation.The training data combines a new Touch-Slide dataset with existing tactile-image datasets.Source: §1 · p. 2
• 95.1% average improvement over task- and sensor-specific end-to-end models is reported under a 33%-50% labeled-data budget for each task.Sparsh (DINO) and Sparsh (IJEPA) are the most competitive, outperforming Sparsh (MAE).Source: §1 · p. 2

2 Related work

• SSL research has developed general-purpose frameworks, with methods categorized by pretext task and learning algorithm.The paper explores masked image modeling, self-distillation, and JEPA.Source: §2 · p. 2
• Tactile representation work includes MAE for material classification and texture recognition, fine-tuned convolutional encoders for fabric decomposition, and pretrained-representation retrieval for XELA uskin.Other approaches use preprocessing tools or custom encoders tailored to particular sensors and tasks.Source: §2 · p. 2
• The paper adapts MAE [23], DINO/DINOv2 [48, 49], and Image/Video JEPA [50, 51] to curated tactile datasets for touch-representation training.The figure identifies these as the SSL approaches illustrated alongside the dataset curation.Source: Figure 2 · p. 3
• Multimodal visual-tactile methods have shown promise for material classification and grasp stability, but cited approaches primarily focus on texture and visual properties rather than forces, slippage, and poses.These contact properties are described as essential for dexterous manipulation.Source: §2 · p. 3
• T3 uses sensor-specific encoders with a shared trunk, while UniT learns representations for GelSight Mini; Sparsh covers three sensor families and adds standardized evaluation across touch-centric tasks.The benchmark spans tactile properties, physical perception, and manipulation planning.Source: §2 · p. 3

3 Touch representations via self-supervised learning

• The paper frames SSL as a way to pursue touch representations useful across tasks and sensors, using computation and diverse data without manual labels.The guiding principles also include performance across tasks, including real-time robot manipulation.Source: §3 · p. 3
• Sparsh adapts vision SSL to DIGIT, GelSight 2017, and GelSight Mini tactile sensors.GelSight 2017 uses markers, while GelSight Mini is markerless.Source: §3 · p. 3
• Tactile images can be ambiguous in isolation and vary with grasp force, material, and shape, motivating investigation of pixel reconstruction, latent reconstruction, and clustering objectives.The authors hypothesize latent reconstruction and clustering may be more efficient because they direct capacity away from fine reconstruction details.Source: §3 · p. 4
• Background subtraction for DIGIT and markerless GelSight Mini provides a no-contact reference and empirically helps generalization across the same sensor type.The reference conveys static shear information when perpendicular force is applied to the elastomer.Source: §3 · p. 4
• The curated datasets total approximately 661k images, with 462.7k used for SSL pretraining and the remainder held out for monitoring with online probes.The pretraining set is described as an order of magnitude larger than prior touch-representation work.Source: §3 · p. 4
• Image-based SSL inputs concatenate tactile images five samples apart, while video-based inputs use four frames spanning approximately 100 ms.The image-based pair corresponds to an approximately 80 ms inference window at 60 FPS.Source: §3 · p. 4

4 TacBench: Tactile sensing benchmark

• TacBench groups six tasks around tactile properties, perception, and manipulation planning, including force and slip estimation, pose and grasp-stability prediction, textile recognition, and bead-maze control.The benchmark compiles task data from various sensors to evaluate representation generalization.Source: §4 · p. 4
• The evaluation freezes pretrained Sparsh encoders and trains attentive decoders, comparing them with an equal-capacity end-to-end model trained from random initialization.Downstream decoders are also trained with different labeled-data amounts; force-field visualization and policy learning are exceptions to the stated decoder setup.Source: §4 · pp. 4–5
• The benchmark's additional analyses include unfrozen encoders, encoder sizes, and few-shot cross-sensor transfer.Further task and protocol details are provided in Appendix D, with these ablations in Appendix E.Source: §4 · p. 5

5.1 [T1] Force estimation

• Frozen Sparsh features estimate forces with low error across sensors, and Sparsh (DINO) remains robust when labeled data are sparse.With sufficient DIGIT labels, end-to-end training can achieve high accuracy, whereas the end-to-end model does not perform well on higher-resolution GelSight Mini images.Source: §5.1 · p. 5

5.2 [T1A] Force field visualization

• Sparsh (DINO) force-field visualizations provide directional information about relative motion of the contact patch, including sliding, torsional slip, and divergence upon contact.The markerless-sensor shear field is estimated without ground-truth field labels, using a CNN decoder for dense predictions.Source: §5.2 · p. 5

5.3 [T2] Slip detection

• Sparsh (VJEPA) achieves the highest F1 score for slip detection, with Sparsh features outperforming E2E when labeled data are significantly reduced.The JEPA-based frozen features are particularly effective on DIGIT, even with only 1% of the training dataset.Source: §5.3 · p. 5

6.1 [T3] Pose estimation

• Pre-trained representations maintain good pose-estimation performance with only a third of the data, while end-to-end approaches decline sharply as labels are reduced.The task predicts discretized relative translations and rotations from time-synchronized DIGIT observations and object poses.Source: §6.1 · p. 6

6.2 [T4] Grasp stability

• Sparsh (IJEPA) and Sparsh (VJEPA) reach approximately 80% grasp-success classification accuracy with the full dataset, surpassing the result from [8] that combined tactile and vision.The single-finger touch model remains competitive with 33% and 10% of the data, but performance drops significantly with 80 training samples.Source: §6.2 · p. 6

6.3 [T5] Textile recognition

• Sparsh improves textile-recognition performance even with 10% of the labeled data, and Sparsh (MAE) is particularly effective.With 10-shot adaptation, Sparsh also transfers the downstream task to DIGIT.Source: §6.3 · pp. 6–7, Figure 4 · p. 7

7.1 [T6] Bead maze

• ∼16% lower trajectory errors are reported for Sparsh (DINO) and Sparsh (IJEPA) than for E2E policy training.The comparison uses held-out maze trajectories evaluated in 3 cm chunks corresponding to 15 timesteps of action predictions.Source: §7.1 · p. 7
• ∼20–53% greater distance before failure is reported for Sparsh policies than E2E in real rollouts using all 50 demonstrations.The rollouts use 10 randomized novel starting locations.Source: §7.1 · p. 7
• None of the models completes the full maze in real robot rollouts, and the authors expect more diverse maze-pattern training data to highlight generalist capabilities.The paper also suggests temporal ensembling may improve policy smoothness.Source: §7.1 · p. 8

8 Discussion

• Sparsh (DINO) is better suited to physics-based force and pose tasks, while Sparsh (IJEPA) performs better on semantic tasks; DINO exceeds IJEPA by 5.6% on average across the benchmark.Both perform similarly on bead-maze test demonstrations, but that does not translate to real-robot performance amid force-control and system-level confounds.Source: §8 · p. 8
• 95.1% average improvement over end-to-end approaches is reported when models use 33%–50% of each task’s labeled data; latent-space learning is described as more advantageous than pixel-space learning.The paper attributes latent-space representations’ advantages to filtering and generalizing over noise or lighting differences.Source: §8 · p. 8
• Sparsh is proposed as a step toward a general pre-trained backbone, while TacBench is an initial benchmark to which further tasks can be added.The discussion names tactile policy learning and tracking changing object properties as examples for future exploration.Source: §8 · p. 8
• The evaluated open-source tactile datasets predominantly feature discrete contacts, and bead-maze policies complete only part of the real maze before compounding error drops the bead.The authors identify shear-rich data and further study of behavioral cloning as directions for improvement.Source: §8 · p. 8

B Broader related work

• Masked image modeling restores masked portions from corrupted samples, with supervision in pixels or an alternative representation space; masked autoencoders are comparatively sample inefficient during training.Self-distillation trains a student to predict teacher representations of augmented views, while JEPA predicts latent embeddings conditioned on masking.Source: B Broader related work · p. 16
• Prior tactile simulators are described as limited by marker-based sensors or narrow tasks, and as unable to model shadows and real-world per-instance sensor discrepancies.The paper links these gaps to hampered potential for representation learning.Source: B Broader related work · p. 17

C Touch representation and self-supervision details

• For fair evaluation, the SSL algorithms are largely adapted from the official MAE, I-JEPA, V-JEPA, DINO, and DINOv2 codebases.Source: C Touch representation and… · p. 17

C.1 Training details

• Online reconstruction probes help monitor convergence because joint-embedding predictive architectures’ training losses may not indicate convergence.All models are trained for 150 epochs with AdamW and a cosine weight-decay schedule; tuning momentum and weight decay was important for convergence without collapse.Source: §C.1 · p. 17, Table 1 · p. 17

C.2 Architecture details

• Sparsh uses ViT-B/14 encoder backbones and a small ViT predictor; DINO repurposes registers to predict classes without a [cls] token.The reported input concatenates tactile images five frames apart along the channel dimension before background removal and resizing to 224 × 224.Source: §C.2 · pp. 17–18
• Table 2 reports parameter counts and inference times for the Sparsh backbones.Figure 5 shows reconstructed tactile images used by online probes to monitor training.Source: §C.1 · p. 17, Table 2 · p. 18, Figure 5 · p. 18

C.3 Dataset splits

• YCB-Slide, Touch-and-Go, ObjectFolder, and the newly collected Touch-Slide supply tactile images for SSL training, with train-validation splits specified for the datasets.Touch-Slide contributes 180k frames from five trajectories on each of nine toy-kitchen objects; YCB-Slide contributes 180k frames, Touch-and-Go 220k, and ObjectFolder 81k.Source: §C.3 · p. 18
• Downstream task data use real sensors and hardware not seen during Sparsh SSL training, supporting evaluation of generalization to new sensor instances.The listed tasks include force estimation and slip detection on DIGIT and GelSight Mini, grasp stability on GelSight17, and bead manipulation on DIGIT; representation-learning data do not include these object-robot hand interactions.Source: §C.3 · p. 18, Figure 6 · p. 19

C.4 Short summary of SSL methods

• MAE learns contextual representations from masked images by reconstructing masked regions, while DINO uses self-distillation between a student and an EMA teacher on different image crops.The DINO branches produce class-probability vectors and use cross-entropy between student and teacher predictions.Source: §C.4 · p. 19
• IJEPA and VJEPA use context and target networks, with the context branch predicting target-network features for local crops of the same image.The approach combines masked-image modeling and self-distillation, and uses an L2 loss over features.Source: §C.4 · p. 19

D.1 Labeled datasets

• Table 3 is identified as the reference for labeled-data curation across TacBench tasks.Source: §D.1 · p. 19

D.2 Probe details

• For global tasks, frozen Sparsh features are evaluated using attentive pooling and a cross-attention module followed by a two-layer MLP probe.The benchmark lists downstream tasks and their labeled datasets in Table 3; Table 4 gives attentive-pooling hyperparameters.Source: §D.2 · pp. 19–20, Table 3 · p. 20, Table 4 · p. 20

D.3 [T1] Force estimation

• Tangential-force errors are around 500mN for E2E but below 100mN in general for Sparsh (DINOv2) on DIGIT across tangential and normal-force ranges.The cited friction-cone visualization shows error in mN; the accompanying table reports RMSE over 25k unseen DIGIT test samples.Source: §D.2 · p. 21, Figure 8 · p. 21, Table 5 · p. 21
• Sparsh (DINO) force-estimation errors remain below 150mN on GelSight Mini, while E2E has higher errors, particularly for normal forces.Most errors are localized around the dynamic shear region; the table reports RMSE over 25k test samples.Source: §D.2 · p. 21, Figure 9 · p. 22, Table 6 · p. 22

D.4 [T1A] Force field visualization

• Force-field visualization uses dense prediction rather than attentive probing, with DPT fusion modules upsampling intermediate Sparsh features into normal- and shear-field outputs.Source: Figure 7 · p. 20, §D.2 · pp. 21–22
• Normal-force fields are estimated from tactile images using a reprojection error, while shear fields are learned through photometric consistency and smoothness losses.The shear-flow method warps features between frames and combines Charbonnier and SSIM losses; Figure 10 visualizes predicted force fields during sliding and spherical-probe interactions.Source: §D.2 · p. 22, Figure 10 · p. 23

D.5 [T2] Slip detection

• Slip data come from a normal/shear load test with a hemispherical probe, and friction-cone regions label sticking versus slippage.The dataset contains 125k samples, 13% labeled as slip; the inference window is approximately 80 milliseconds.Source: §D.2 · p. 23, Figure 11 · p. 23
• Sparsh (VJEPA) outperforms all models on slip detection, including at low training-data budgets, and its 33%-data predictions are contrasted with E2E.Slip labels rely on an experimental coefficient of friction, which can make the ground-truth friction boundary inaccurate; Sparsh (VJEPA) nevertheless detects slip samples in the illustrated case.Source: §D.2 · pp. 23–24, Figure 12 · p. 24, Table 7 · p. 24

D.6 [T3] Pose estimation

• With 33% of labeled data, Sparsh distinguishes pose changes more accurately than the task-specific E2E approach, which struggles with adjacent bins and tends toward zero or maximum changes.The task discretizes each degree of freedom into 11 classes and reports accuracy over 630 test samples.Source: §D.2 · p. 25, Table 8 · p. 25

D.7 [T4] Grasp stability

• For grasp stability, frozen Sparsh with attentive probing and a 2-layer MLP classifies success from before/during tactile history, evaluated on 1.3k grasps.Source: §D.7 · pp. 25–26, Table 9 · p. 27

D.8 [T5] Textile recognition

• Textile recognition uses 4,467 clips across 20 classes; frozen Sparsh plus attentive probing evaluates a 2-layer classifier across label budgets.Source: §D.7 · p. 26

• ∼20 −53%: Sparsh-based bead-maze policies outperform E2E over 10 randomized novel starts, with no significant fine-tuning gains from 50 demonstrations.Failures were mostly due to the bead getting stuck in the maze or falling out of the robot hand.Source: §D.7 · p. 28, Table 11 · p. 28
• On the real robot, Sparsh policies completed almost 30% of a single-maze run versus almost 80% for single-maze E2E; unseen-maze trajectory fidelity did not ensure execution.The authors attribute the gap to compounded joint-angle errors that can lead to collisions and loss of the grasp.Source: §D.7 · p. 28, Table 12 · p. 29
• 98.75% average improvement over E2E across TacBench; Sparsh (DINO) and Sparsh (IJEPA) are generally strongest, while MAE remains competitive but was not evaluated on the policy task.Source: §D.7 · pp. 28–29, Table 13 · p. 29

E.1 TacBench evaluations via fine-tuning

• Full fine-tuning improves latent-space Sparsh models on regression and classification tasks, while partial fine-tuning brings only minor gains over frozen models.The reported regression gains include force estimation with only 33% and 10% labeled data; the authors hypothesize that MAE weights may be more brittle, and report smaller effects for MAE.Source: §E.1 · p. 29

E.2 Sparsh ViT-small and performance

• With 33% labeled data, Sparsh (DINO) ViT-small increases force-estimation error by 74% on DIGIT and 50.3% on GelSight Mini; pose accuracy falls 20% even with full labels [T3].Classification tasks show no major capacity-reduction effect except at the 1% training budget; for pose estimation, ViT-small performs better than an end-to-end ViT-base model.Source: §E.2 · p. 30

E.3 Sparsh and cross-sensory representation

• Zero-shot textile recognition on DIGIT is near chance, but with 10 samples Sparsh (DINO) quickly adapts and significantly outperforms E2E.The evaluation uses 110 test samples and covers 10 of the 20 textile classes.Source: §E.3 · p. 31