PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Tactile Understanding From Visual and Haptic Data

Yang Gao, Lisa Anne Hendricks, Katherine J. Kuchenbecker, Trevor Darrell

arXiv:1511.06065v2cs.ROcs.CVcs.LG

TL;DR

The paper addresses how robots can infer fine-grained haptic properties before or during physical interaction. It trains deep models on visual and haptic data, including transferred visual features, and finds that combining modalities yields the strongest classification performance.

  • Problem

    Robots need fine-grained tactile understanding of objects and surfaces, including haptic properties that may be inferred before touching.

  • Method

    The paper uses deep neural networks to learn haptic features from physical-interaction signals and visual features transferred from material classification, then combines modalities.

  • Results

    AUC 85.9 is achieved by combining activations from models J and L, while multimodal models perform best by well over 2.7 points.

  • Takeaways & Limitations

    Visual and haptic signals are complementary, so multimodal deep models can provide haptic information for robotic tasks without relying only on physical interaction.

Abstract

from arXiv · show

Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by this cognitive pattern, we propose and explore a purely visual haptic prediction model. Purely visual models enable a robot to "feel" without physical interaction. Furthermore, we demonstrate that using both visual and physical interaction signals together yields more accurate haptic classification. Our models take advantage of recent advances in deep neural networks by employing a unified approach to learning features for physical interaction and visual observations. Even though we employ little domain specific knowledge, our model still achieves better results than methods based on hand-designed features.

I. INTRODUCTION

The paper develops deep-learning models that classify haptic properties from visual and physical-interaction data. Combining modalities reflects multisensory human understanding and outperforms using either input alone.

  • Tactile understanding could help robots adjust grips, avoid wet or slippery surfaces, and describe unfamiliar objects.
  • Humans combine visual and haptic signals when identifying objects and processing their environment.
  • Combining learned features across modalities achieves higher performance than models trained on haptic data or visual data alone.
  • The proposed models use deep neural networks to learn features from visual and haptic data with little domain-specific knowledge.
  • Visual data from material classification transfers to haptic classification, while haptic and visual signals provide complementary information.

II. RELATED WORK

Related work spans tactile measurement, hand-designed haptic features, material classification, and neural architectures. The paper positions deep networks as flexible, transferable alternatives for haptic classification.

  • Prior studies classify haptic adjectives using rich measurements of temperature, compliance, roughness, and friction or hand-designed features.
  • Haptic classification extends beyond material recognition because the same material can exhibit substantially different roughness and hardness properties.
  • B. Neural Networks: Neural networks learn compositional representations by stacking layers whose outputs feed subsequent layers and training weights with backpropagation.
  • B. Neural Networks: CNNs use convolutions and reduce parameter counts, supporting large-scale recognition and learning from one-dimensional signals.
  • B. Neural Networks: Recurrent networks update hidden units over time, with LSTMs and GRUs using gates to retain or forget previous states.
  • B. Neural Networks: Neural networks require little domain knowledge and can transfer learned weights to related datasets through fine-tuning.
  • Prior multimodal systems train models for individual modalities before using their activations in a multimodal classifier.

III. DEEP HAPTIC CLASSIFICATION MODELS

The paper applies deep learning as a unified framework for haptic classification using haptic, visual, and multimodal inputs. It explores both convolutional and recurrent models.

  • Deep learning provides a unified framework for learning classification models from multiple signal types.
  • The proposed systems target haptic classification with haptic data, visual data, and combined multimodal data.
  • The study explores both CNN and LSTM architectures for haptic classification.
  • Figure 2 presents the structure of the haptic CNN.

1) Haptic CNN Model:

The models include temporal CNNs and LSTMs for haptic signals, and transfer learning from a material-recognition CNN for visual haptic classification. The visual pipeline uses intermediate network representations.

  • 1) Haptic CNN Model:: The haptic CNN applies temporal convolutions to one-dimensional signals, reducing parameters for training with limited data.
  • 1) Haptic CNN Model:: Its input concatenates 32 haptic measurements along the channel axis, and ReLU nonlinearities follow each convolution.
  • 1) Haptic CNN Model:: Grouping separates channels into groups of 32, delaying cross-channel interactions until the fully connected layer.
  • 1) Haptic CNN Model:: The models begin with logistic loss, then use hinge-loss fine-tuning for the reported results.
  • 1) Haptic CNN Model:: LSTM models use recurrent units for haptic time series, but stacking LSTMs degraded haptic classification performance.
  • B. Visual CNN Model: Visual haptic classification transfers weights from a CNN fine-tuned on the 23-class Materials in Context Database.
  • B. Visual CNN Model: The visual pipeline transfers layers below inception (5a), then applies average pooling and L2 normalization before classification.
  • B. Visual CNN Model: Figure 3 summarizes the visual CNN structure.

C. Multimodal Learning

The multimodal model combines learned visual and haptic representations before classification. It transfers pretrained modality-specific weights and trains only the final classification layer with hinge loss.

  • C. Multimodal Learning: Visual and haptic CNN activations are concatenated to form the multimodal representation.The model uses haptic conv3 activations and L2-normalized visual inception (5a) activations.
  • C. Multimodal Learning: The final multimodal classifier is trained with hinge loss.
  • C. Multimodal Learning: The multimodal network directly transfers weights from previously trained visual and haptic networks.Only the classification layer is learned after the modality-specific weights are transferred.
  • C. Multimodal Learning: Training uses standard backpropagation with SGD, a 0.01 learning rate, 0.9 momentum, Xavier initialization, 200 epochs, and batch size 1000.

IV. EXPERIMENTAL SETUP

The experiments use PHAC-2, which pairs BioTac exploratory signals and multiview object images with 24 binary haptic-adjective labels. Signal preprocessing normalizes, resamples, and compresses haptic measurements before training.

  • A. PHAC-2 Dataset: PHAC-2 contains haptic signals and images of 53 household objects explored by paired BioTac sensors on a PR2 gripper.Objects undergo four exploratory procedures, including Squeeze, Hold, Slow Slide, and Fast Slide.
  • A. PHAC-2 Dataset: Each object has high-resolution 3000 × 2000 images from eight viewpoints.Objects are placed at the center of an aluminum plate, with uncontrolled lighting variations described as insignificant.
  • A. PHAC-2 Dataset: Objects receive 24 binary haptic-adjective labels determined by a majority vote of approximately twelve annotators.The labels represent the presence or absence of adjectives such as slippery or fuzzy.
  • 1) Haptic Signal:: Haptic signals are standardized per channel and downsampled to fixed length 150 for each exploratory procedure.PAC is downsampled to 100 Hz, while fixed-length resampling addresses variation in Squeeze signal duration.
  • 1) Haptic Signal:: Four principal components capture 95% of electrode-impedance variation before the impedance signals are incorporated into the haptic representation.
  • 1) Haptic Signal:: Data augmentation increases the 530 original training instances to 5,300 by using both sensors and multiple signal starting points.

2) Visual Signal:

Visual inputs are standardized and cropped around the object, while multiple viewpoints and haptic explorations can be combined by concatenating intermediate activations and retraining the loss layer.

  • 2) Visual Signal:: Images are mean-subtracted, resized to 224×224, and centrally cropped to include the object.The crop is derived by detecting the circular aluminum plate and estimating its center and radius.
  • 2) Visual Signal:: The visual preprocessing uses a central crop whose dimensions are 2R × R and whose center is shifted above the plate center by R.
  • C. Combining Data Instances: Combining separate instances from one sample is used to boost classification results in related recognition settings.The paper contrasts activation concatenation and retraining with averaging model outputs across image crops or mirrors.
  • C. Combining Data Instances: The method combines ten haptic explorations and eight visual viewpoints for each object.Rather than averaging test-time outputs, it concatenates intermediate activations from the separate instances and retrains the network loss layer.

D. Train/Test Splits

The evaluation uses adjective-specific object splits and AUC to compare shallow, deep, visual, haptic, and multimodal classifiers. Deep haptic and multimodal models outperform prior features, while visual performance depends strongly on multiple viewpoints.

  • Evaluation setup: 24 separate 90/10 train/test splits prevent any object from appearing in both training and testing for an adjective.The 53 objects are partitioned separately for each of 24 adjectives because no single split covers every adjective in both sets.
  • Evaluation setup: AUC replaces the prior F1 metric because it considers both true-positive and false-positive rates.The evaluation reports average AUC across adjectives for reproduced baselines and proposed models.
  • Baseline comparison: Combining static and dynamic hand-designed features increases AUC by 0.9.Static features slightly outperform dynamic features before combination.
  • Model comparison: The best deep haptic CNN improves AUC by 5.0 over hand-designed features.The evaluation also compares data augmentation and multiple-instance configurations for haptic models.
  • Model comparison: Visual performance is 6.0 points below the best haptic model, while multiple viewpoints improve visual AUC from 71.5 to 77.2.The result suggests varied viewpoints are important for visual haptic classification in robotic applications.
  • Model comparison: The multimodal model using models J and L achieves an AUC of 85.9, exceeding the corresponding single-modality result despite model J being slightly weaker alone.Multimodal models perform best by well over 2.7 points.

VI. DISCUSSION

The discussion introduces qualitative analysis to understand what the model learns and how its predictions compare across modalities.

  • VI. DISCUSSION: The authors analyze model predictions to gain a more intuitive understanding of the learned capabilities.The analysis is intended to illustrate what the models can learn.

A. Adjective Prediction Using Different Models

Haptic, visual, and multimodal classifiers produce different prediction profiles, while modality usefulness varies by adjective. Activation analysis further links different sensor channels to different haptic properties.

  • A. Adjective Prediction Using Different Models: For three objects, the haptic classifier has high recall, whereas the visual classifier is more conservative and predicts no false positives.The visual classifier appears to have higher precision in these examples.
  • A. Adjective Prediction Using Different Models: The multimodal classifier combines information rather than taking a simple union, generally achieving higher overall AUC than either single-modality model.For some objects, such as furry eraser, it performs worse than the haptic classifier.
  • B. Haptic and Visual Data are Complementary: Size adjectives such as thin and thick are better classified visually, while squishy and absorbent favor haptic data.Cool is better classified visually, whereas metallic is better classified haptically in the reported comparisons.
  • C. Which Haptic Signals Matter?: The haptic CNN processes each signal independently before the fully connected layer, enabling inspection of final convolutional activations.The analysis uses conv3 activations to identify channels that may be important for classification.
  • C. Which Haptic Signals Matter?: Core Temperature Change activations appear important for metallic classification, and similar activations across trials suggest robustness to trial variation.The temperature interpretation is consistent with the presumed thermal conductivity of metallic objects.
  • C. Which Haptic Signals Matter?: Electrode activations appear important for compressible classification, indicating that different adjectives rely on different signal channels.The authors suggest most BioTac signals may contribute to fine-grained haptic classification.

VII. CONCLUSION

The paper concludes that deep architectures can learn from visual and haptic signals, and that combining the modalities improves classification. It identifies larger, more diverse paired data as a route for future improvement.

  • VII. CONCLUSION: Deep architectures learn features for haptic classification from both haptic and visual signals with little domain knowledge.The conclusion presents deep learning as a paradigm applicable to a variety of signals.
  • VII. CONCLUSION: Combining visual and haptic data improves performance and may provide useful haptic information for robotic pipelines.The authors connect this integration to performance across many robot tasks.
  • VII. CONCLUSION: A larger, more diverse dataset could improve the model by supporting larger networks, end-to-end multimodal fine-tuning, and less aggressive haptic downsampling.The authors identify this dataset expansion as future work.
Loading 1511.06065v2…