PaperlayerView exact revision on arXiv

Source-linked AI summary

Robotic Tactile Perception of Object Properties: A Review

Shan Luo, Joao Bimbo, Ravinder Dahiya, Hongbin Liu

arXiv:1711.03810v1cs.ROcs.CV

TL;DR

Robotic tactile perception lacks a comprehensive synthesis of methods for interpreting touch data about object properties. This survey reviews tactile sensing technologies and perception methods for shape, material, and pose, concluding that these capabilities remain important for grasping and manipulation while localization and computational scalability remain open challenges.

  • Problem

    A comprehensive review of interpreting tactile data for object-property perception had not yet been reported, despite growing research attention.

  • Method

    The paper surveys tactile sensing technologies and methods for recognizing shape, surface material, and object pose, including tactile-visual sensor fusion.

  • Results

    The survey identifies current approaches spanning tactile feature descriptors, deep and unsupervised learning, global shape recognition, pose estimation, and sensor fusion.

  • Takeaways & Limitations

    Shape, surface material, and pose information are described as important for successful robotic grasping and manipulation, with touch complementing vision through local information.

  • Takeaways & Limitations

    Object localization by touch remains an open problem, while single-point sensors provide limited contact information and may require multiple contacts that can move the object unpredictably.

Abstract

from arXiv · show

Touch sensing can help robots understand their sur- rounding environment, and in particular the objects they interact with. To this end, roboticists have, in the last few decades, developed several tactile sensing solutions, extensively reported in the literature. Research into interpreting the conveyed tactile information has also started to attract increasing attention in recent years. However, a comprehensive study on this topic is yet to be reported. In an effort to collect and summarize the major scientific achievements in the area, this survey extensively reviews current trends in robot tactile perception of object properties. Available tactile sensing technologies are briefly presented before an extensive review on tactile recognition of object properties. The object properties that are targeted by this review are shape, surface material and object pose. The role of touch sensing in combination with other sensing sources is also discussed. In this review, open issues are identified and future directions for applying tactile sensing in different tasks are suggested.

I. INTRODUCTION

Robotic tactile systems convert touch stimuli into task-oriented information about object properties and actions. This review frames tactile perception within hierarchical sensing architectures and categorizes sensors by spatial resolution and biological analogy.

  • Tactile sensing systems: Tactile systems transduce pressure, vibration, and thermal stimuli into processed information for perceiving object properties and guiding actions.Their functional hierarchy separates sensing, perception, and action, while hardware blocks implement these stages.
  • Review scope: The review addresses tactile perception of material, shape, object pose, recognition, and localization, including combinations with vision and other modalities.It emphasizes that techniques adapted from computer vision may not always suit tactile data because the modalities operate differently.
  • Tactile sensing modalities: Tactile sensors are categorized by analogy to biological body parts and by spatial resolution into single-point, fingertip-like arrays, and large-area skin-like sensors.Single-point sensors detect contact forces or vibrations; higher-resolution and large-area sensors support broader contact measurement.
  • Tactile perception: Tactile perception interprets and represents sensing information to observe object properties and provide task-oriented information for planning and control.Its representation depends on the hardware and the robot’s intended task.

B. Tactile Perception

Tactile data representations support robot applications by converting sensor readings into task-relevant features. Common approaches draw on vision, biological models, or dimensionality reduction, while learned representations are increasingly explored.

  • Representation approaches: Tactile representations commonly treat sensor elements as image pixels, use biologically inspired descriptors, or apply dimensionality reduction.The extracted information can include object shape, material properties, and pose.
  • Applications: Tactile perception has been applied to grasp control, slip detection and prevention, grasp stability assessment, surgery, interactive games, medical training, and underwater robotics.These applications use tactile information for control, recognition, or task assistance.

III. MATERIAL RECOGNITION BY TACTILE SENSING

Surface material is a key cue for robotic interaction, but vision alone cannot estimate physical material parameters independently of prior visual knowledge. Touch is therefore used to identify surface properties.

  • Motivation: Vision alone can recognize only previously known surface materials and cannot independently estimate their physical parameters.The review motivates tactile sensing as necessary for identifying material properties through contact.

A. Surface texture based tactile material recognition

Tactile material recognition uses contact-derived texture cues, including friction, roughness, microstructure, and acoustic signatures. These cues are measured through sliding, tactile images, or dynamic sensing during contact.

  • Texture cues: Friction coefficients and roughness can be obtained by sliding a force or tactile sensor across an object surface, while microstructure can be captured in tactile images.These measurements provide complementary representations of surface texture.
  • Acoustic sensing: Dynamic tactile sensing can map frictional sounds into the frequency domain with FFT to detect different surface textures.An artificial finger equipped with a microphone collects the frictional sound data during contact.
  • Terrain classification: Tactile arrays can classify terrain materials such as wood, carpet, clay, and grass.The cited approach estimates texture types using probability densities derived from tactile-array means and variances.

B. Object stiffness based tactile material recognition

Tactile sensing supports recognition of object stiffness-related material properties through compliance, contact geometry, electrode data, and active impact sensing.

  • BioTac sensors estimate object compliance, defined as the reciprocal of stiffness, from fingertip contact angle or electrode data.
  • An accelerometer-equipped robot leg can reveal hardness, elasticity, and stiffness by actively knocking on object surfaces.
  • Tactile image sequences from GelSight sensors have also been used to estimate object hardness.

IV. TACTILE OBJECT SHAPE PERCEPTION

Tactile shape perception identifies or reconstructs object geometry at local and global scales, supporting grasping and manipulation when vision is occluded or illumination is poor.

  • Shape perception ranges from reconstructing exact geometry, such as point clouds, to classifying shape elements or overall profiles.
  • Complete shape information supports planning and executing grasping trajectories and manipulation strategies.
  • Tactile shape perception is unaffected by visual occlusion or poor illumination and can capture detailed shapes through sensor-object interactions.
  • Golf and tennis balls share the global category of ball but differ locally through small pits versus curvilinear surface shapes.
  • Proprioceptors provide finger and end-effector position and movement information that can be integrated with local tactile features for object recognition.

A. Local shape recognition

Local shape recognition increasingly treats tactile pressure patterns as images, while tactile and visual sensing differ in field of view, data collection, and variation factors.

  • Tactile pressure patterns can be treated as images, with features extracted from pressure distributions to identify local shapes.
  • Vision typically offers a large global field of view, whereas tactile sensing provides a small local field of view through direct sensor-object interaction.
  • Cameras collect data relatively easily and provide many features per image, but visual processing requires high computational resources.
  • Visual feature performance can fluctuate with scaling, rotation, translation, and illumination, while tactile sensing directly probes local contact regions.

1) Shape descriptors for tactile object recognition:

Tactile object recognition uses handcrafted, statistical, vision-derived, PCA-based, and self-organizing descriptors, each trading implementation simplicity, representation, computation, or physical interpretability.

  • Raw tactile readings as features: Raw tactile readings avoid feature extraction and are easy to implement but are sensitive to position, orientation, and size variations.
  • Statistical features: Statistical features are effortless to obtain, yet their usefulness is not guaranteed and redundant information may remain.
  • Descriptors adapted from computer vision: Vision-derived descriptors represent tactile arrays as images, including image moments, regional descriptors, Hough transforms, and high-level descriptors for high-resolution sensors.
  • PCA-based features: PCA reduces tactile-data redundancy and dimensionality, but its principal components lack physical meaning.
  • Self-organizing features: Handcrafted features fed to shallow classifiers are easy to implement but can restrict representation capability and capture insignificant characteristics.
  • Self-organizing features: Denoising autoencoders improved classification of 20 objects by around 20% over shallow neural networks and supervised learning, while RTCN reached 100% recognition on most tested datasets.

2) Discussions of local shape descriptors:

Tactile feature extraction increasingly treats tactile readings as images, while unsupervised learning and deep architectures are emerging as alternatives for learning features directly from raw data.

  • Vision-based descriptors that treat tactile readings as images are widely employed for tactile object recognition and other applications.
  • Unsupervised learning and deep architectures are increasingly used to learn self-organizing features from raw tactile readings.
  • The reviewed descriptors support tasks including grasp stability assessment, touch-modality identification, pose estimation, slip detection, texture recognition, and tactile servoing.

B. Global shape perception

Global tactile shape recognition is organized by whether methods use contact locations, tactile-array pressure distributions, or both. Approaches range from geometric reconstruction and state estimation to codebook-based classification and multimodal fusion.

  • Global-shape input categories: Global-shape methods use contact-point distributions, tactile-array pressure distributions, or combinations of tactile patterns and contact locations.Global shape concerns overall object contours extending beyond fingertip scale.
  • Points based recognition: Contact-point methods fit sensed points to geometric models, including superquadric surfaces or polyhedral models, to reconstruct object contours.These approaches were widely used early because tactile sensors had low resolution and single-point force sensors were prevalent.
  • Tactile patterns based recognition: Pressure-distribution methods recognize contact shapes from tactile arrays, but relatively few approaches analyze tactile images collected across multiple global contact locations.
  • Tactile patterns based recognition: Bag-of-Features represents an object with a fixed-length vector formed by assigning local tactile features to codewords learned by clustering training features.The codewords are cluster centroids in a learned tactile-feature dictionary.
  • Object recognition based on both sensing modalities: Combining spatial information with tactile features can benefit object recognition, although existing fusion methods may concatenate or decision-fuse modalities without fundamentally linking them.

V. POSE ESTIMATION VIA TOUCH SENSING

Touch-based pose estimation supports manipulation by localizing objects or features using single-point contacts, tactile arrays, vision, or combinations of sensing inputs. The review identifies high accuracy and real-time localization with practical contact requirements as unresolved challenges.

  • Accurate object pose estimates are important because location errors can cause incorrect finger placement, mistaken grasp-stability assumptions, and unsuccessful manipulation.Pose comprises object position and orientation relative to the end-effector or a global coordinate frame.
  • Single-point contact based: Early single-point contact methods used force, dynamic tactile, angle, or joint-torque sensing to localize objects and features.
  • Tactile-based: Tactile-array methods can localize in-hand objects or features by registering tactile images against a height map.A height map is built from tactile readings before keypoints are localized in the map and incoming tactile data.
  • Multimodal pose estimation: Contact-visual and tactile-visual methods combine tactile local information with visual geometry or maps for object or sensor localization.Vision may provide an approximate pose that tactile sensing subsequently refines through local or global optimization.
  • Open issues: Object localization during manipulation remains open because high accuracy and real-time performance are unmet, while many approaches require impractically numerous contacts.The review foresees combining different tactile sensors with vision for localization.

VI. TACTILE SENSING IN SENSOR FUSION

Tactile sensing is fused with vision, kinaesthetic cues, force-torque, and range sensing to form richer representations for physical interaction. The review describes fusion for contact verification, grasping assistance, local pose adjustment, and transferring visual knowledge to tactile classification.

  • Combining tactile sensing with other modalities can increase precision and robustness in interaction tasks.Typical combinations include vision, kinaesthetic cues, force-torque sensing, and range sensing.
  • Verifying contacts: Contact-verification methods use vision to build rough object models and tactile sensing to refine contact descriptions or generate probabilistic contact maps.Contact hypotheses from tactile, force-torque, vision, and range sensors can be fused probabilistically.
  • Extracting features to assist vision: Tactile features can be combined with vision and action or proprioceptive features to assist goal-oriented grasping and estimate in-hand object location.
  • Providing local and detailed information: During door opening, vision detects the handle while tactile moment analysis measures gripper orientation for pose adjustment after contact.
  • Transferring knowledge with vision: Paired visual and tactile samples can train a mapping that transfers visual material representations to tactile-domain classification.At test time, materials can be classified using only tactile observations.

VII. DISCUSSION AND CONCLUSION

The survey identifies sensor, representation, localization, fusion, and learning challenges that shape robotic tactile perception of object properties. It summarizes open issues and future directions for recognition and manipulation applications.

  • Sensor selection: Tactile sensor selection requires balancing spatial and magnitude resolution against sensor size and cost.Higher resolution can provide more detailed object information but generally increases development and fabrication costs; GelSight offers high resolution but is bulky.
  • Tactile object recognition: Shape recognition methods trade off robustness, computational cost, and geometric coverage.Contact-point distributions can recover arbitrary global shapes but are time-consuming and motion-sensitive, whereas Bag-of-Features captures local patterns without three-dimensional distribution information.
  • Tactile object recognition: Future shape perception may integrate tactile patterns with kinaesthetic cues because cutaneous sensing and proprioception are correlated in object recognition.
  • Object localization: Touch-based localization remains open on mapping sensor-pad contacts into 3D space, combining vision and touch, and resolving conflicting sensor information.
  • Tactile sensing in sensor fusion: Tactile sensing in fusion has expanded from contact verification to feature extraction, local-detail support for vision, and knowledge transfer with vision.The survey identifies sensor hierarchy as a direction for establishing preferences among sensor data.
  • Deep learning in robotic tactile perception: Deep learning can learn self-organizing features, but tactile applications face computational, interpretability, data-collection, and dataset-standardization challenges.The same algorithm may perform differently across data collected from different tactile sensors, and commonly used evaluation datasets remain unavailable.
Loading 1711.03810v1…