PaperlayerView exact revision on arXiv

Source-linked AI summary

An Interpretable Deep Learning Framework for Material Perception and Classification from Multisensory Tactile Data

Li Zou, Dave Hogendoorn, Yasemin Vardar

arXiv:2608.21894v1cs.ROcs.HC

TL;DR

The paper addresses the poorly understood relationship between multisensory tactile signals, perceptual representations, and material recognition. It develops three interconnected deep-learning models with Integrated Gradients, finding that direct classification is strongest while thermal cues are especially informative.

  • Problem

    The mapping from tactile information to perceptual attributes and material recognition remains poorly understood, limiting faithful digital touch interfaces and human-like robotic tactile perception.

  • Method

    The framework uses deep learning without hand-crafted features to map multisensory interaction signals to adjective-rating distributions, material classes through those distributions, or material classes directly, with Integrated Gradients for interpretation.

  • Results

    Path 2 outperforms Path 1, Model 3 achieves strong overall F1 performance with several classes identified nearly perfectly, and thermal cues are particularly informative across perceptual modeling and classification.

  • Takeaways & Limitations

    Interpretable deep learning can approach near-perfect material classification while identifying sensory cues that may inform tactile sensing systems and haptic interfaces.

Abstract

from arXiv · show

Human tactile perception relies on complex multisensory cues. Yet the relationship between tactile signals and perceptual representations remains poorly understood, limiting the integration of touch in digital environments and human-like robotic perception. To address this gap, we developed a computational framework comprising three interconnected deep learning models that map multisensory touch data to material perception, without relying on hand-crafted features. The models represent progressively different routes from tactile signals to material class: from low-level interaction signals to perceptual attribute distributions (Model 1), from predicted attribute distributions to material classification (Model 2), and directly from tactile signals to material categories, bypassing intermediate representations (Model 3). By combining deep learning with Integrated Gradients, the framework achieved high accuracy while offering interpretability, revealing which sensory modalities most strongly drive its decisions. Our results show that deep learning can approach near-perfect material classification when unconstrained by intermediate perceptual stages, but matching human-like performance is harder once those stages are modeled explicitly. Notably, thermal cues emerged as particularly informative across all models, providing robust signals for material differentiation. The results offer a computational account of how tactile signals lead to material perception and show how interpretable deep learning can both approach human-level performance and reveal cues that robotic and haptic systems need to incorporate.

I. INTRODUCTION

The framework addresses poorly understood links between multisensory tactile signals, perceptual attributes, and material recognition. It combines deep learning and interpretability across pathways that either model or bypass intermediate human-aligned perception.

  • Motivation: The framework targets open questions about which tactile cues inform perception, how cues combine, and how they produce distinct tactile sensations.These gaps limit faithful digital touch interfaces and robotic systems with human-like tactile perception.
  • Motivation: Deep learning can learn useful features directly from raw, high-dimensional signals, avoiding manually encoded rules and hand-engineered features.Prior tactile recognition work reported accuracies as high as 98.8%.
  • Contribution: The framework replaces hand-crafted features and classical machine learning with deep learning throughout a computational model of human material perception and recognition.The approach is intended to connect multisensory tactile measurements with perceptual representations and material categories.
  • Framework: The framework uses multisensory force, acceleration, indentation, temperature, and heat-flux signals collected during sliding, pressing, and static contact.Its analysis considers both algorithmic interpretability and structural pathways from signals to material classes.
  • Framework: Path 1 maps interaction signals to psychophysical attributes and then material classes, whereas Path 2 maps signals directly to material classes through Model 3.This design compares recognition with and without alignment to human perception.

A. Dataset

The study uses the SENS3 multisensory tactile dataset together with psychophysical ratings to construct material-classification and perceptual-distribution targets. Signals are organized by exploratory action, while ratings are adjusted and aggregated into smoothed probability distributions.

  • Data collection: SENS3 records multisensory tactile signals from 50 surfaces using accelerometer, force-torque, thermistor, heat-flux, and infrared position sensors.Three participants explored surfaces through static contact, pressing, and sliding.
  • Data collection: Pressing captures force and indentation, static contact captures thermal signals, and sliding captures lateral force and acceleration related to friction and roughness.For interpretability, actions are divided into temporal phases or force–speed bins.
  • Dataset composition: The classification dataset contains 45 materials across seven classes after excluding classes with two or fewer material instances.The retained classes are fabric, foam, metal, paper, sandpaper, vinyl, and wood.
  • Psychophysical ratings: Twenty participants rated materials on hot–cold, hard–soft, rough–smooth, and sticky–slippery adjective pairs using a discrete 1–15 scale.These pairs represent primary psychophysical dimensions of tactile surface perception.
  • Rating processing: Participant ratings are centered on the global scale mean to preserve relative ordering while removing systematic rater bias.The adjusted ratings are aggregated into interval counts, with smoothing constant 0.1 added to avoid empty bins.
  • Rating processing: The processed ratings form one empirical probability distribution per material and adjective pair, serving as Model 1 training targets.The distributions lie on the 15-point psychophysical scale.

B. 1D-Convolutional Neural Networks (1D-CNNs)

The 1D-CNN architecture processes multichannel tactile time series with convolutional kernels that capture local patterns and longer temporal structure. Its weight sharing, parallel training, and receptive-field control suit mixed transient and slow thermotactile signals.

  • Architecture: A 1D-CNN processes sequential data by sliding learnable kernels along one dimension of a multichannel input.The input is represented as a time series with channels, and each kernel produces a feature map.
  • Architecture: Kernel size, stride, and dilation provide precise control over the receptive field while convolution enables weight sharing and stable parallel training.These properties distinguish 1D-CNNs from recurrent models and hand-crafted features.
  • Signal processing: Early convolutional layers capture brief transients, while deeper layers integrate slower trends such as thermal equilibration and force ramps.The signals also contain heat-flux peaks and vibration bursts.
  • Model design: Shallow 1D-CNNs with small kernels serve as convolutional encoders in Models 1 and 3.This choice reflects the temporal structure of thermotactile signals.

C. Integrated Gradients (IG)

Integrated Gradients attributes a model output to input features by integrating gradients along a path from a baseline to the input. Temporal Saliency Rescaling further concentrates attribution on relevant time steps in multivariate tactile signals.

  • Attribution method: Integrated Gradients quantifies each input feature’s contribution to a scalar model output relative to a zero baseline or same-shaped baseline matrix.Importance is computed along the linear path from baseline to input.
  • Attribution method: The integral is approximated with a Riemann sum over N steps.This provides a practical numerical estimate of the gradient-based attribution.
  • Attribution output: For multivariate time series, Integrated Gradients produces a two-dimensional attribution map over time steps and channels.The resulting maps can be summarized across temporal phases or channel groups.
  • Attribution refinement: Temporal Saliency Rescaling identifies relevant time steps before evaluating feature importance within them, reducing attribution dilution.This refinement is designed for multivariate time-series inputs.

A. Model 1: Signal to Adjective Rating Distributions

Model 1 maps thermal, sliding, and pressing signals to discrete psychophysical rating distributions for four adjective pairs. Its multi-head late-fusion architecture specializes action-specific encoders before jointly predicting each perceptual attribute.

  • Model 1 maps three exploratory actions—thermal/static contact, sliding, and pressing—to 15-bin rating distributions for each adjective pair.
  • Each action branch encodes its sequential tactile input with convolutional layers, pooling, normalization, dropout, flattening, and a dense layer.Encoder depth varies by action, with sliding and pressing using more convolutional blocks than thermal.
  • The resulting action embeddings are concatenated and passed through a shared fusion block before separate adjective-pair output heads generate predictions.Each head uses a Dense(64) ReLU layer, dropout, and a final Dense(15) softmax layer.
  • The late-fusion design lets each encoder specialize to its signal characteristics while the shared block combines them to predict perceived attributes.Examples include thermal transients, vibration/friction structure, and indentation dynamics.
  • Kullback–Leibler divergence serves as both the training loss and evaluation criterion for matching predicted and target rating distributions.

B. Model 2: Adjective-to-Material Classifier

Model 2 uses Model 1’s predicted psychophysical attribute distributions as features for material classification. It concatenates four 15-bin distributions and classifies samples with a compact multilayer perceptron.

  • Model 2 concatenates Model 1’s 15-bin distributions for four adjective pairs into a single 60-dimensional feature vector.
  • The feature vector is classified by an MLP with one dense hidden layer, batch normalization, dropout, and a seven-class softmax output.The default hidden layer contains 32 units.

C. Model 3: Signal-Based Material Classifier

Model 3 classifies materials directly from fused tactile signals, replacing Model 1’s adjective-specific heads with one material-classification head. The supplied interpretability and performance passages emphasize action- and channel-level contributions and stronger direct-classification results than the intermediate route.

  • C. Model 3: Signal-Based Material Classifier: Model 3 uses action-specific encoders and late fusion, then predicts one of seven material classes without an intermediate adjective-distribution stage.Its classification head comprises dense processing, batch normalization, dropout, and a seven-class softmax output.
  • A. Model 1: Signal to Adjective Rating Distributions: Model 1’s adjective-pair predictability varies, with hot–cold most predictable and hard–soft least predictable under the available data.The pattern suggests stronger learned thermal than compliance-related perceptual structure.
  • A. Performance of Model 1: Table I reports KLD for Model 1 and F1 scores for Models 2 and 3, with HC, HS, RS, and SS denoting the four adjective pairs.
  • A. Performance of Model 1: Model 1 interpretability results compare action-level attributions in the outer ring with channel-level attributions in the inner ring for each adjective pair.Static contact is labeled Thermal because it contains only thermal measurements.
  • A. Performance of Model 1: Thermal cues account for approximately half of Model 1’s attribution mass, exceeding sliding’s one-third and pressing’s one-fifth.Rough–smooth relies disproportionately on thermal data, whereas sticky–slippery depends most strongly on pressing.
  • C. Model 3: Signal-Based Material Classifier: The confusion-matrix comparison covers Model 2 and Model 3 using row-normalized means aggregated across folds.

B. Performance of Model 2

Model 2 shows limited and uneven material classification, while its interpretability analysis identifies hard–soft ratings as the strongest attribute-level contributor. Model 3 instead classifies directly from tactile signals, with thermal cues and transient heat-transfer phases especially informative.

  • Model 2 classification: Model 2 has limited and uneven classification capability, with Fabric and Metal recognized better than Paper, Sandpaper, and Vinyl.The cited passage summarizes overall and per-class F1 performance without providing the numerical scores.
  • Model 2 interpretability: 34.0% of Model 2’s positive IG share comes from hard–soft, exceeding rough–smooth at 23.4%, sticky–slippery at 24.0%, and hot–cold at 18.6%.Hard–soft contributes especially strongly to predictions of Metal, Wood, Foam, and Fabric; sticky–slippery and hot–cold are prominent for Sandpaper and Paper, respectively.
  • Model 3 classification: Model 3 classifies materials directly from tactile signals and achieves strong overall F1 performance, with Fabric and Metal identified nearly perfectly.Foam, Sandpaper, Vinyl, and Wood also yield high F1 scores, while Paper has a 20% probability of confusion with Foam.
  • Model 3 interpretability: Model 3 assigns 39.9% attribution to thermal, 34.0% to pressing, and 26.1% to sliding actions.Within modalities, heat flux, normal force, friction coefficient, and band-pass IMU axes are the strongest cited channels.
  • Model 3 interpretability: Thermal attribution concentrates in transient peak and half-equilibration phases rather than steady-state contact, while sliding attribution is dominated by low-force bins.Pressing attribution spans loading, plateau, and lift-off phases.
  • Model 3 interpretability: Model 3’s per-class patterns show Metal as strongly thermal-driven, Fabric as more pressing-driven, and Foam as reliant on sliding and thermal contributions.Paper, Wood, Vinyl, and Sandpaper use more mixed action profiles, which the passage connects to their confusability.

D. A Comparison Between Path 1 and Path 2

The two paths both classify materials from interaction signals, but Path 1 inserts an attribute-prediction stage whereas Path 2 classifies directly. Path 2 outperforms Path 1, and class behavior varies across the two routes.

  • Path definitions: Path 1 maps interaction signals through psychophysical attributes before classification, whereas Path 2 maps signals directly to material classes.The additional intermediate stage is the defining structural difference between the paths.
  • Performance comparison: Path 2 outperforms Path 1, with the authors suggesting that prediction errors may accumulate across Path 1’s intermediate stage.The passage frames this as a likely explanation rather than a demonstrated mechanism.
  • Class-specific behavior: Fabric and Metal are easily classified regardless of path, whereas Vinyl is correctly classified only via Path 2.The remaining materials are reasonably classified by both paths.
  • Sensory cues: Thermal cues are particularly informative for both attribute-rating distributions in Path 1 and material classification in Path 2.The passage characterizes thermal information as a salient and robust signal for differentiating materials in the computational models.

V. CONCLUSION

The framework links multisensory tactile signals to material perception through three progressively different deep-learning routes and uses Integrated Gradients to interpret their decisions. It indicates that direct classification can approach near-perfect performance, while human-like modeling remains challenging; thermal cues are especially informative.

  • Three interconnected models map tactile signals to perceptual attributes and material categories, including a direct route that bypasses intermediate perception.Model 1 predicts psychophysical attributes, Model 2 classifies materials from those predicted distributions, and Model 3 classifies directly from tactile signals.
  • Integrated Gradients combines with deep learning to provide strong classification accuracy and identify which sensory modalities and aspects drive decisions.
  • Direct material classification can approach near-perfect performance when intermediate perceptual rules do not constrain the model, whereas human-like behavior remains difficult.The framework examines representational contributions without claiming to model the biological or neural mechanisms of human perception.
  • Thermal cues are particularly informative for both perceptual modeling and material classification, offering a salient and robust signal for differentiating materials.The authors suggest that thermal sensing could improve robotic material perception and that thermal feedback could enrich haptic interfaces.
Loading 2608.21894v1…