Source-linked AI summary
Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor
Ziyang Ma, Yuhao Sun, Zichen Ai, Xiangyang Ji, Bin Fang
TL;DR
Existing slip datasets largely lack fine-grained directional labels, motivating a markerless visuo-tactile sensor and unified perception framework. The system combines uniform RGB illumination, an RGB-D dataset, and dual-head temporal modeling, achieving strong unseen-object slip performance and material recognition across 15 categories.
Problem
Existing slip datasets are predominantly binary and lack fine-grained directional perception, limiting the available slip-classification detail.
Method
The paper develops a markerless visuo-tactile sensor with uniform RGB illumination, a synchronized RGB-D dataset, and unified TimeSformer- and ResNet-50-based perception.
Results
Across unseen objects, contact-state and eight-direction slip classification reached 95.5% and 91.5% accuracy, while 15-category material recognition reached 98.8%.
Takeaways & Limitations
The hardware-software framework provides high-fidelity feedback and a multi-modal perception baseline for complex robotic manipulation.
Abstract
from arXiv · showhide
Tactile sensing is central to robotic manipulation, among which slip detection stands out as a quintessential and critical task. However, existing slip datasets are predominantly limited to binary classification, lacking fine-grained directional perception. To address this limitation, we propose a visuo-tactile sensor featuring customized uniform RGB illumination, alongside a unified perception framework. At the hardware level, the sensor achieves high-precision, sub-millimeter depth reconstruction. Based on this capability, we collect a multi-task visuo-tactile dataset encompassing 15 objects, synchronously generating depth information for each data sample. Algorithmically, we design a dual-head TimeSformer network to process dynamic spatiotemporal slip. On unseen objects, this network achieves robust accuracies of 95.5% and 91.5% for 3-class contact state prediction and fine-grained 8-class slip direction classification, respectively. Furthermore, static tactile-based object class recognition utilizing a ResNet-50 backbone yields an outstanding accuracy of 98.8% across 15 categories. The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
I. INTRODUCTION
The paper targets markerless, multi-task visuo-tactile perception by combining uniform RGB illumination, an RGB-D tactile dataset, and unified models for slip and material recognition.
- The collected data include 7,925 slip samples and approximately 19,336 material-classification images from 15 common objects.
- Existing visuo-tactile sensors often use physical markers that obscure fine surface textures and limit high-definition contact geometry.
- The proposed markerless sensor uses customized RGB illumination to avoid specular highlights, capture micro-textures, and enable sub-millimeter depth reconstruction.
- The multi-task RGB-D dataset covers 15 object categories, synchronously providing precise depth information for spatiotemporal slip and material classification.
- The unified pipeline combines a dual-head TimeSformer for contact states and slip directions with a ResNet-50 for static material classification.
II. RELATED WORK
Related work frames slip detection as data-driven spatiotemporal perception and highlights the need for directional feedback, while visuo-tactile sensing supports material classification through rich spatial features.
- A. Data-Driven Slip Detection: Slip detection supports reactive grasping and stable manipulation, motivating data-driven models that process consecutive tactile or visuo-tactile frames.
- A. Data-Driven Slip Detection: Modern approaches increasingly use self-attention and Vision Transformers to model visuo-tactile sequence data during dynamic interactions.
- A. Data-Driven Slip Detection: Specific slip-direction prediction provides richer feedback for adjusting grasping force than simplistic binary slip classification.
- B. Tactile-based Object Class Recognition and Benchmark Datasets: Visuo-tactile sensors offer rich spatial features that support material classification, while large contact-rich datasets facilitate fine-grained robotic manipulation.
- B. Tactile-based Object Class Recognition and Benchmark Datasets: The sensor design is presented as a compact visuo-tactile platform integrating elastomeric contact, RGB illumination, and miniature-camera reception modules.
III. HARDWARE DESIGN AND SENSING PRINCIPLE
The sensor uses a compact modular structure and engineered uniform RGB illumination to improve optical consistency, suppress reflections, and support markerless 3D tactile reconstruction.
- The sensor measures 37 mm × 34 mm × 34 mm and separates the detachable elastomeric contact module from the integrated illumination-camera base.
- A PCB with distinct illumination zones and a frosted diffuser distributes light uniformly across the contact layer.
- Direct illumination beneath the elastomer can create acrylic-plate reflections, so the optical path is engineered to suppress resulting ghost images.
- Color-channel intensity measurements on a flat surface are used to quantify illumination variation and assess consistency with the LED layout.
- The customized uniform RGB illumination suppresses specular highlights and supports photometric-stereo reconstruction of 3D topography and high-frequency micro-textures.
B. Sensor Fabrication
The sensor combines a compact replaceable elastomer contact module with empirical photometric-stereo calibration for dense depth reconstruction from RGB tactile images.
- Sensor Fabrication: The contact module is assembled from cast optically clear silicone and a sprayed matte white diffuse-reflection coating.The silicone is molded and cured before the coating is applied and the components are integrated.
- Depth Reconstruction: The sensor uses a reference sphere and precision three-axis stage to calibrate photometric stereo.The calibration sphere has a known radius of 3 mm, corresponding to a 6 mm diameter.
- Depth Reconstruction: Known spherical geometry provides ground-truth horizontal and vertical surface gradients for calibration.The contact center and each pixel’s projected radial distance define the spherical impression geometry.
- Depth Reconstruction: A continuous mapping converts RGB intensities and pixel coordinates into dense gradient fields for unknown tactile contacts.The mapping is fitted to reference gradients during calibration and applied to captured RGB images during sensing.
- Depth Reconstruction: Poisson integration reconstructs the continuous depth map z(x,y) from the estimated gradient fields.This completes the RGB-to-depth reconstruction pipeline after calibrated gradient estimation.
IV. DATA COLLECTION AND DATASET
The benchmark contains static material and dynamic slip subsets, with synchronized dense depth maps for all visuo-tactile samples and uniform illumination supporting consistent Transformer inputs.
- IV. DATA COLLECTION AND DATASET: The tactile benchmark combines static material classification and dynamic spatiotemporal slip subsets.It is designed to evaluate micro-texture extraction and dynamic physical interactions.
- IV. DATA COLLECTION AND DATASET: All raw visuo-tactile samples are synchronously converted into one-to-one corresponding dense depth maps, forming an RGB-D tactile dataset.The conversion spans both benchmark subsets.
- IV. DATA COLLECTION AND DATASET: Uniform illumination eliminates specular highlights and maintains spatial and temporal micro-texture consistency for downstream spatiotemporal attention.This consistency is described as preventing the Transformer from overfitting to lighting noise.
A. Material and Object Classification Dataset
The material-classification dataset uses 15 diverse objects and approximately 19,336 high-resolution tactile images collected across varied contact conditions.
- A. Material and Object Classification Dataset: The dataset includes 15 representative daily objects spanning geometric indenters, hardware tools, electronic connectors, and natural textures.The object set is intended to test the sensor’s spatial resolution across complex topographies.
- A. Material and Object Classification Dataset: Approximately 19336 high-resolution tactile images were captured across diverse contact locations, spatial poses, and pressing depths.These variations broaden the static material examples collected from the object set.
- A. Material and Object Classification Dataset: Uniform RGB illumination stably captures high-frequency details without specular interference, providing a data foundation for learning material features.The passage links the illumination condition to stable detail capture during collection.
B. Spatiotemporal Slip Dataset
The slip dataset models dynamic interactions as 8-frame sequences with three contact states and eight slip directions, using a dual-head TimeSformer and masked joint training.
- B. Spatiotemporal Slip Dataset: The omnidirectional slip dataset contains 7925 high-fidelity sequences totaling around 63,400 images from 15 objects.Each sample is organized as an 8-frame continuous image sequence.
- B. Spatiotemporal Slip Dataset: Slip-state labeling includes Pressing, Static, and Slip, with Slip further refined into fine-grained directional classes.The dataset is designed to support directional perception beyond binary slip classification.
- B. Spatiotemporal Slip Dataset: A dual-head TimeSformer alternates temporal and spatial self-attention to track micro-texture displacement across tactile frames.Uniform illumination supports this tracking by reducing distraction from lighting artifacts.
- B. Spatiotemporal Slip Dataset: The State Head predicts 3 contact classes while the Direction Head predicts 8 slip-direction classes simultaneously.Both outputs are produced from the network’s global token through fully connected heads.
- B. Spatiotemporal Slip Dataset: The masked joint loss propagates direction loss only when the ground-truth state is Slip, preventing interference from static or pressing states.The indicator function implements this conditional optimization behavior.
- B. Spatiotemporal Slip Dataset: Static material classification uses a pretrained ImageNet ResNet-50 backbone with a fully connected output for 15 material categories.The method processes a single tactile image and applies color and geometric augmentation during training.
V. EXPERIMENTS AND RESULTS
The experiments evaluate the uniformly illuminated visuo-tactile sensor across depth reconstruction, material classification, and spatiotemporal slip detection.
- Experiments cover 3D depth reconstruction, material classification, and spatiotemporal slip detection.
A. Experimental Setup
The experimental setup combines calibrated depth reconstruction, representative tactile objects and sequences, and robotic hardware for evaluating visuo-tactile perception.
- Ground-truth depth is analytically derived from a calibrated standard sphere, while predicted morphology comes from RGB-derived gradients integrated by solving Poisson’s equation.
- The evaluated objects include representative daily items whose micro-textures and geometric contours are clarified by uniform illumination without specular highlight interference.
- Dynamic tactile data comprise continuous raw RGB image sequences and temporal difference maps captured during real-world grasping.
- The reconstruction pipeline extracts a binary contact mask, predicts high-frequency planar gradients, and applies masked Poisson integration with a zero-potential non-contact baseline.
- 0.072 mm Plane-aligned RMSE demonstrates sub-millimeter depth reconstruction accuracy after isolating topological error from mechanical misalignment.
C. Dynamic Spatiotemporal Slip Detection
Dynamic slip perception uses a dual-head TimeSformer to classify contact states and fine-grained slip directions from tactile sequences, with evaluations on RGB and reconstructed-depth inputs.
- 99.69% and 98.86% accuracy are achieved on RGB sequences for 3-class contact state prediction and 8-class slip direction classification, respectively.
- The dual-head TimeSformer uses patch embedding, divided space-time attention, a global token, dual fully connected heads, and masked joint loss.
- 95.5% contact-state accuracy is maintained on five unseen objects evaluated with 1,200 independent sequences.
- 91.5% accuracy is maintained across eight slip directions for unseen-object sequences classified as slip.
- Static recognition uses single-frame RGB images and reconstructed depth maps with a ResNet-50 backbone; RGB reaches 98.8% and depth 96.06% accuracy across 15 categories.
VI. CONCLUSIONS
The paper presents a uniformly illuminated visuo-tactile sensor and unified framework combining high-fidelity depth reconstruction, dynamic slip detection, and static material classification.
- 0.072 mm RMSE is reported for high-fidelity 3D topological reconstruction using optimized optical design to mitigate specular-reflection artifacts.
- The dataset contains 7,925 slip sequences, 63,400 sequence images, and 19,336 static images across 15 diverse objects.
- 95.5% state-classification and 91.5% fine-grained direction-detection accuracies are achieved on unseen objects, while static recognition reaches 98.8% across 15 material categories.
- Future work targets integration into a multi-fingered dexterous robotic hand and fusion with global vision and proprioceptive data.