PaperlayerView exact revision on arXiv

Source-linked AI summary

SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection

Tong Jian, Aditya Thurvas Senthil Kumar, Xinyi Li, Ziling Chen, Tianyu Dai, Ali Sengul, Matteo Grimaldi, Wenjie Lu, Saleh Nabi, Tao Yu

arXiv:2609.15910v1cs.ROcs.AIcs.LG

TL;DR

Slip detection matters for grasp stability, yet detection latency and generalization across objects, sensor units, and robotic platforms remain incompletely characterized. SlipSense fuses piezoresistive pressure and accelerometer vibration signals with causal prediction. It reports strong detection results and zero-shot transfer from UMI to a Tesollo dexterous hand.

  • Problem

    Detection latency is rarely characterized alongside false-positive rates, and generalization across unseen objects, sensor units, and robotic platforms remains limited.

    Source: §1 · p. 2

  • Method

    SlipSense combines TacV5's spatial pressure array and vibration-sensing accelerometer using modality-specific encoders, fusion, and causal temporal classification.

    Source: §1 · p. 2

  • Results

    96.7% Macro F1 with false-positive rate below 1.6%; 76% of slip events are detected within 23.1 ms, and transfer to Tesollo is zero-shot without retraining.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Across the reported settings, SlipSense transfers across unseen objects, distinct sensor units, and robotic platforms without targeted fine-tuning.

    Source: §5.2 · p. 8

    Data collection uses a single linear slip axis; rotational and multidirectional slip lack quantitative latency characterization, and in-motion detection during active manipulation remains to be quantitatively validated.

    Source: §7 · p. 9

Abstract

from arXiv · show

Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a $32 \times 32$ piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal attention with causal temporal prediction at 240 Hz. Experiments on a dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining.

1 Introduction

SlipSense targets gaps in characterizing slip-detection latency and generalization across objects, sensors, and platforms. It combines complementary tactile modalities and reports low false-positive rates, rapid detection, and zero-shot transfer.

1 Introduction

  • The paper identifies poorly characterized detection latency and limited evaluation across unseen objects, sensor units, and robotic platforms as unresolved challenges.Latency is the interval between slip onset and the first correct detection, and is rarely reported alongside false-positive rates.Source: §1 · p. 2
  • TacV5 pairs a 32×32 piezoresistive array that captures spatial pressure with an 8 kHz accelerometer that captures friction-induced vibration.The modalities have complementary failure modes, motivating joint processing.Source: §1 · p. 2
  • 96.7% Macro F1 vs. 81–84% for either modality alone, with false-positive rate below 1.6%, is reported for fusion.The comparison is between the fused model and either single modality.Source: §1 · p. 2
  • 76% of slip events are detected within 23.1 ms by the 240 Hz system.Source: §1 · p. 2
  • On 1.4M frames across 37 objects, SlipSense transfers zero-shot from UMI to Tesollo across unseen objects, sensor instances, and platforms without retraining.Source: §1 · p. 2

2 Related Work

Related work spans optical, electrical, and vibration-based tactile sensing, each with characteristic strengths and limitations. The paper identifies latency measurement alongside cross-condition generalization as an under-evaluated combination.

2 Related Work

  • Optical sensors infer slip from high-resolution displacement fields, while electrical sensors capture shear-related signals and piezoresistive arrays measure normal force.The cited optical systems can degrade on unseen objects or smooth surfaces, while piezoelectric sensing is limited to dynamic transients.Source: §2 · p. 2
  • Prior work often collapses piezoresistive 2D structure into 1D signals, whereas TacV5's denser array enables joint modeling of spatial pressure and temporal evolution.Vibration-based methods capture friction-induced oscillations but can also respond to environmental disturbances.Source: §2 · p. 3
  • Prior studies report latency or generalization in limited settings, but the paper identifies their joint evaluation as missing.Examples include latency measurements on a single object or a few objects, and transfer studies without latency characterization.Source: §2 · p. 3

3 System and Data Acquisition

TacV5 combines spatial pressure sensing and high-bandwidth acceleration in a compact module used across two robotic platforms. The data protocol uses controlled object displacement to label slip independently of the tactile signals.

3.1 Sensor Specification

  • TacV5 combines a 32×32 piezoresistive array sampling at 240 Hz with a 3-axis accelerometer sampling at 8 kHz.The array measures normal pressure; the accelerometer captures high-frequency vibration from frictional dynamics.Source: §3.1 · p. 3

3.2 Data Collection and Labeling

  • The collection protocol varies grasp force before inducing slip, and non-slip actuation and grasp changes also produce strong XL vibrations.Spatial Piezo pressure cues help disambiguate those vibration responses from slip signatures.Source: Figure 3 · p. 4

3.1 Sensor Specification

  • TacV5 is integrated on a UMI parallel-jaw gripper and a Tesollo dexterous hand, with sensor units on two jaws and three fingers, respectively.Its compact design fits within a standard fingertip pad and remains mechanically independent of gripper actuation.Source: §3.1 · pp. 3–4

3.2 Data Collection and Labeling

  • The Mark-10 crosshead provides displacement-based slip labels at 0.02 mm resolution, independently of sensor output, and is absent at deployment.The non-stretchable-object constraint applies during data collection, not deployment.Source: §3.2 · p. 4
  • Slip is induced during crosshead ascent, with onset labeled after cumulative displacement exceeds 0.07 mm; contact frames outside that interval are labeled No slip.No contact is assigned when the mean taxel value falls below one ADC count, using a threshold shared across objects and sensors.Source: §3.2 · p. 4

4 Multimodal Learning

SlipSense predicts the final frame's contact and slip state from synchronized tactile observations. Its modality-specific encoders, sensor aggregation, cross-modal fusion, and causal temporal prediction are designed for variable sensor configurations and 240 Hz operation.

4.1 Problem Formulation

  • SlipSense formulates prediction as classifying the last frame of an observation sequence into no-contact, no-slip, or slip.Source: §4.1 · p. 5

4.2 SlipSense Framework

  • Piezo images use a compact convolutional encoder, while accelerometer signals are high-pass filtered and encoded from causal log-mel spectrograms.The accelerometer encoding uses a 10 Hz high-pass cutoff and a 16 ms analysis window aligned to each Piezo frame.Source: §4.2 · p. 5
  • Element-wise max pooling aggregates embeddings across sensor units, making each modality's representation independent of sensor count and discarding finger identity.Cross-attention then integrates Piezo and XL features.Source: §4.2 · pp. 5–6
  • The evaluation splits progressively test held-out objects, unseen contact regions, and new platforms with distinct sensor units.The platform evaluations include Tesollo two-finger and three-finger pinch configurations.Source: Figure 5 · p. 6
  • A causal self-attention layer accumulates temporal context, and the prediction head is trained on the final frame of the observation window.At inference, predictions can be produced per incoming frame for operation at 240 Hz when inference fits within a frame interval.Source: §4.2 · p. 6

5 Experiments

Experiments evaluate SlipSense on varied objects and progressively shifted test conditions, focusing on multimodal performance, generalization, latency, and real-world slip prevention. Fusion improves detection performance, and the model transfers across objects, sensors, and robotic settings without targeted fine-tuning.

5.1 Experimental Setup

  • Evaluation used 1.4M frames from 37 objects, with 28 training objects, nine held out, five test splits, and two UMI training units.Data included five slip speeds, varied grasp forces, and no-slip control scenarios; evaluations also spanned UMI and Tesollo sensor units over four months.Source: §5.1 · p. 6, §5 · p. 6

5.2 Analysis and Discussion

  • Table 1 reports that Piezo–XL fusion achieves the lowest latency and FPR while substantially improving Macro F1 over either modality alone; latency is mean per-seed median detection latency in frames across 3 seeds, against analytic baselines.Figure 6 shows fusion suppressing Piezo grasp-variation confusion and XL aftershock sensitivity during no contact.Source: Table 1 · p. 7, Figure 6 · p. 7
  • 76% of slip events are detected within 23.1 ms including model inference.Detection latency falls as slip speed increases; over 90% of trials at 18 mm/s are detected within 20.8 ms.Source: Figure 7 · p. 8, §5.2 · p. 8
  • 94.39% Macro F1 transfers to a 2-finger pinch and 87.24% to a 3-finger pinch, despite the latter's wider distribution gap from training.A single model trained on UMI is evaluated zero-shot across contact location, platform, and finger count; SlipSense generalizes across the tested settings without targeted fine-tuning.Source: §5.2 · pp. 7–8, Table 2 · p. 7
  • Macro F1 rises from 93.7% at T=6 fr to 96.7% at T=60 fr, then plateaus; longer windows increase variance across seeds.At higher slip speeds, T=60 fr (250 ms) captures sufficient contact evolution, while slower slip can require longer windows to produce one taxel of displacement.Source: §5.2 · p. 8
  • Fusion performance improves with Piezo resolution, and even an 8×8 array improves over XL-only.The lower-resolution inputs are spatially strided from native 32×32 measurements while keeping the fusion pipeline unchanged.Source: §5.2 · p. 8

5.3 Real-World Deployment and Slip Prevention Strategy

  • SlipSense detected every induced slip in 100/100 varied-direction trials and the re-grasp controller prevented object loss in 95/100 trials.Five prevention failures occurred when inward motion pushed an edge-held cable out despite correct detection; detection-to-peak-force latency was approximately 50 ms.Source: §5.3 · p. 8

6 Conclusions

SlipSense fuses piezoresistive arrays and accelerometers for tactile slip detection. It reports high accuracy, low-latency detection, zero-shot transfer across tested settings, and a closed-loop re-grasp proof of concept.

6 Conclusions

  • SlipSense fuses 32×32 piezoresistive arrays with 8 kHz accelerometers for tactile slip detection.Piezo grounds contact state while XL captures friction transients during slip.Source: §6 · p. 9
  • Fusion achieves 96.7% Macro F1, with 76% of slip events detected within 23.1 ms.The learned model transfers zero-shot from a UMI parallel-jaw gripper to a Tesollo dexterous hand across unseen objects, sensor units, and contact configurations.Source: §6 · p. 9
  • A closed-loop re-grasp controller provides a proof of concept for fast tactile feedback in dexterous manipulation.The reported results suggest a direction toward native integration of multimodal tactile sensing in dexterous robots.Source: §6 · p. 9

7 Limitations

SlipSense’s evaluation and deployment claims remain bounded by single-axis data collection, uncharacterized latency in some settings, and workstation inference. The appendix also documents the customized UMI and accelerometer-processing pipeline.

7 Limitations

  • Quantitative validation remains bounded by single-axis collection, uncharacterized latency for directional and in-motion slip, normal-pressure sensing, 37 objects, and workstation inference.Rotational and multidirectional slip were detected in 100/100 real-world trials, but without quantitative latency characterization.Source: §7 · p. 9

Appendix

  • The appendix covers nine supplementary topics, including setup, encoder details, ablations, latency characterization, real-world deployment, and direction-invariance analysis.Source: Appendix · p. 13

A Customized UMI

  • The customized UMI adds two sensor-equipped finger adapters and a top-mounted board to consolidate and synchronize their data streams.Source: Figure 8 · p. 13

B Accelerometer Transformation and Encoder Details

  • Per-axis causal 16 ms log-mel frames are log-sum-exp merged into a direction-invariant representation, then encoded by SSAST-Tiny.Each frame uses only past and current observations and aligns one-to-one with a Piezo frame.Source: B Accelerometer Transformation and… · pp. 13–14

C.1 Dataset

The dataset spans diverse objects, held-out materials, and a human-operated pressure range that covers test conditions. The section also specifies controls, training settings, and analytic baselines.

C.1 Dataset

  • 37 objects are split into 28 training objects and 9 held-out objects, including unseen Shore hardness blocks at 81 HA and 88 HA.Source: Figure 9 · p. 14
  • Training pressure coverage spans the test distribution, with both sets concentrated below 200 kPa·taxels and a long tail beyond 400 kPa·taxels.Slip data use five crosshead speeds from 2–18 mm/s and naturally varying human grasp forces.Source: §C.1 · p. 14, Figure 10 · p. 14
  • Three control scenarios, each with 5 trials, cover no-contact motion, holding an unlinked object, and free-grasping or pick-and-place without induced slip.Source: §C.1 · p. 14

C.2 Implementation Details

  • Training uses 60-frame windows, modality-specific augmentation, 10 epochs, learning rate 1 × 10^-5, batch size 32, and exponential moving average.Source: §C.2 · p. 15

C.3 Baseline Details

  • Three analytic baselines are Piezo-only pressure-difference entropy, XL-only MFCC features with QDA, and their logical-AND combination.Thresholds are fitted on training data and applied directly to test data.Source: §C.3 · p. 15

D Ablation Study on Encoder Architectures

The encoder ablation varies one modality’s encoder while holding the other encoder and fusion pipeline fixed. Results favor pretrained spatial encoders for Piezo and Tiny-SSAST for XL, while a custom CNN offers a lighter Piezo alternative.

D Ablation Study on Encoder Architectures

  • The ablation varies one modality’s encoder while fixing the other encoder and fusion pipeline to isolate each encoder’s contribution in the fused deployment setting.Source: D Ablation Study on Encoder… · p. 15
  • The Piezo candidates include ResNet-18 and a ViT-MAE pretrained on about 2 million simulated and real contact frames, using masked-patch reconstruction.Source: D Ablation Study on Encoder… · p. 15
  • The XL candidates include a compact MLP and a causal 1D CNN that preserves per-frame temporal structure.Source: D Ablation Study on Encoder… · p. 16
  • On UMI OOD, all Piezo ViT-MAE variants outperform ResNet-18, the frozen encoder slightly exceeds finetuning, and Custom CNN matches ViT-MAE across metrics while lighter.For XL, Tiny-SSAST leads across metrics; a flattened-spectrogram MLP collapses to 29-frame median latency, while the 1D CNN recovers most performance.Source: D Ablation Study on Encoder… · p. 16, Table 3 · p. 16

E Ablation on XL Augmentation

XL augmentation improves transfer to Tesollo while leaving UMI performance largely unchanged. The result suggests a possible role in suppressing platform-specific vibration signatures while preserving slip cues.

E Ablation on XL Augmentation

  • Removing XL augmentation substantially reduces Tesollo transfer performance while leaving UMI performance largely unchanged.The authors suggest frame-level standardization and spectral masking may suppress platform-specific vibration signatures while preserving slip-relevant features.Source: E Ablation on XL Augmentation · p. 16, Table 4 · p. 16

F Ablation on Fusion Strategy

Cross-attention outperforms simpler fusion strategies; detection latency varies with slip speed, and real-world trials test deployment on UMI and Tesollo. The appendix shows how summed-axis vibration features yield a direction-invariant log-mel representation.

F Ablation on Fusion Strategy

G Detection Latency by Slip Speed

H Real-World Deployment

I Proof of Direction Invariance of Merged Log Mel Spectrogram

  • Summing accelerometer-axis power cancels direction-dependent coefficients, yielding the same log-mel spectrogram as the underlying vibration independent of direction; LogSumExp provides an efficient, stable realization.The direction-invariance proof assumes a single vibration source and a direction that is constant within each STFT window.Source: I Proof of Direction Invariance of… · pp. 19–20

Related papers

Loading 2609.15910v1…