PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review

Nathan F. Lepora

arXiv:2105.14455v2cs.RO

TL;DR

Human-like robot dexterity requires machines to reproduce important capabilities of human touch. This review synthesizes SoftBOT sensing through the BRL TacTip, covering its biomimetic morphology, optical transduction, fabrication, integration, and learning-based control. The reviewed advances include accurate tactile perception and manipulation capabilities, with deep learning enabling real-time interaction with complex 3D objects.

  • Problem

    Robots need human-like tactile capabilities to achieve the dexterity of human hands.

  • Method

    The article reviews the BRL TacTip as a prototypical SoftBOT sensor combining soft robotics, biomimetics, optical tactile sensing, and AI.

  • Results

    The review reports TacTip-based tactile capabilities including 93% object classification, 95% grasp-success prediction, and real-time interaction with complex 3D objects after deep learning.

  • Takeaways & Limitations

    SoftBOT sensors offer a route to artificially recreate key aspects of human touch and manual intelligence in robots.

Abstract

from arXiv · show

Reproducing the capabilities of the human sense of touch in machines is an important step in enabling robot manipulation to have the ease of human dexterity. A combination of robotic technologies will be needed, including soft robotics, biomimetics and the high-resolution sensing offered by optical tactile sensors. This combination is considered here as a SoftBOT (Soft Biomimetic Optical Tactile) sensor. This article reviews the BRL TacTip as a prototypical example of such a sensor. Topics include the relation between artificial skin morphology and the transduction principles of human touch, the nature and benefits of tactile shear sensing, 3D printing for fabrication and integration into robot hands, the application of AI to tactile perception and control, and the recent step-change in capabilities due to deep learning. This review consolidates those advances from the past decade to indicate a path for robots to reach human-like dexterity.

I. INTRODUCTION

Human-like robotic dexterity depends on reproducing tactile capabilities through a combination of soft robotics, biomimetics, optical tactile sensing, and learning. The review uses the TacTip as a prototypical SoftBOT sensor to consolidate progress toward that goal.

  • Human tactile perception is presented as central to the dexterous use of hands and to the development of technology.
  • Reproducing human tactile capabilities is identified as an important step toward human-level dexterity in robotic hands.
  • Soft robotics, biomimetics, optical tactile sensing, and deep learning are combined to provide adaptable bodies, biological principles, rich contact information, and tactile perception.
  • The review defines this combination as SoftBOT sensing and treats the BRL TacTip as its prototypical example.
  • The article reviews TacTip-based tactile sensing to consolidate advances from the past decade and indicate a path toward human-like robot dexterity.

II. SOFT, BIOMIMETIC AND OPTICAL TACTILE SENSING

Soft, biomimetic, and optical tactile sensing are defined by their material compliance, biological inspiration, and use of light to observe deformation. The TacTip combines these principles through a biomimetic skin structure and optical marker motion.

  • A tactile sensor transduces deformation of a sensing surface into a signal containing information about physical contact.
  • Soft tactile sensors use soft materials and flexible or compliant structures to obtain contact information and conform to three-dimensional objects.
  • Soft biomimetic tactile sensors apply principles distilled from biological systems rather than merely copying biological shapes.
  • Soft optical tactile sensors use light, often from inside the sensor, to view deformation of the sensing surface.
  • Camera-based optical tactile sensors include marker-based designs that typically measure lateral shear and reflection-based designs that typically measure normal indentation.
  • The BRL TacTip combines optical imaging with biological principles and was proposed as a tactile sensor based on biologically inspired edge encoding.

III. SOFT BIOMIMETIC OPTICAL TACTILE SENSING: THE TACTIP

The TacTip reproduces key structural and functional principles of glabrous human skin through layered soft materials, biomimetic features, and optical marker imaging. This design supports shape, texture, slip, and fine spatial perception, while some sensory modalities remain only partially mimicked.

  • Biomimetic morphology: The TacTip mimics glabrous skin with an outer elastomeric epidermis, inner gel dermis, and interdigitating ridges and nodular pins.These structures amplify surface deformation into lateral marker movement, paralleling the mechanical role of dermal papillae and epidermal ridges.
  • Biomimetic transduction: Marker displacements and velocities correspond respectively to slowly adapting and rapidly adapting mechanoreceptor responses for sustained contact and changing contact.The biomimetic mapping links marker position to shape and edge recognition and marker motion to events such as slip.
  • Open modalities: High-frame-rate imaging only partially mimics vibration sensing, and the review suggests embedded gel pressure sensing as a more biomimetic counterpart.The TacTip can also incorporate temperature sensing through a thermoactive outer skin.
  • Perceptual capabilities: The TacTip achieves sub-millimetre spatial discrimination despite millimetre-scale pin spacing through overlapping receptive fields and spatial interpolation.The review relates this tactile hyperacuity to super-resolution in optical imaging.
  • Biomimetic surface features: Biomimetic fingerprints can increase texture sensitivity and spatial localisation, while ringed patterns can reveal incipient slip before grip failure.Fingerprints are implemented as raised bumps or concentric rings over the papillae.
  • Emerging directions: Event-based cameras offer a route toward neuromorphic TacTip transduction using temporally precise trains of spike events.This direction adopts event-based signalling principles from biological nervous systems.

A. Tactile sensing of normal strain and shear strain

The TacTip measures contact through biomimetically generated shear rather than directly measuring only normal strain. Its stiff-pin structure produces marker-displacement patterns that distinguish indentation from sliding and differentiates it from other tactile sensors.

  • Shear transduction: Skin indentation tilts nodular pins and converts normal surface strain into measurable shear strain through lateral marker displacement.The stiff pins interdigitate with soft elastomeric gel to perform this mechanical transduction.
  • Contact signatures: Sliding produces a relatively uniform shear-strain field, whereas normal indentation produces characteristic dipole or multipole marker patterns.Real contacts commonly combine normal and shear strain, complicating contact-shape inference.
  • Comparison with other sensors: The TacTip differs from taxel-based sensors such as BioTac and the iCub fingertip, which measure normal strain, and from GelSight, which reconstructs normal strain from shading.This distinction concerns the transduction principle rather than a claim that one design is universally superior.
  • Comparison with other sensors: Unlike floating-marker sensors such as GelForce and ChromaTouch, the TacTip uses stiff nodular pins to connect surface deformation to optical markers.The comparison highlights the TacTip's biomimetic mechanical pathway through the pin structure.

B. The shear-sensing hypothesis

The shear-sensing hypothesis proposes recovering shape-related tactile information from internal shear strain, which reflects gradients of surface pressure or indentation. For the TacTip, the pin-based model preserves this gradient computation while avoiding the blurring associated with a uniform elastomer.

  • Hypothesis: The shear-sensing hypothesis argues that shape-related tactile information is more suitably recovered from shear strain than normal strain.The motivation is that surface pressure must be inferred from the strain field induced inside the sensor medium.
  • Uniform-elastomer model: In a uniform elastomer, shear strain follows the gradient of a depth-blurred pressure profile and increases linearly with depth.The model represents mechanical blurring through convolution with a depth-dependent Gaussian.
  • Model scope: The uniform-elastomer model is better suited to floating-marker sensors than to the TacTip's inner structure of stiff pins and soft gel.The review therefore treats it as an imperfect model of TacTip mechanics.
  • Pin model: For the TacTip, normal indentation levers nodular pins so their tips move horizontally in proportion to the surface indentation gradient.The linear approximation applies when the surface gradient is small, dδ/dx ≪ 1.
  • Model comparison: Both models predict shear strain proportional to depth and a surface gradient, but the pin structure avoids the mechanical blurring term present in the uniform elastomer.This shared mathematical form supports the shear-sensing interpretation of the TacTip.
  • Signal processing: TacTip processing converts tactile images into two-dimensional shear fields and extracts shear magnitude and Voronoi-area change for edge-contact analysis.The figure presents these quantities across contact and release over time.

V. PERIOD I (2009-2014): INITIAL DEVELOPMENT OF THE TACTIP

From 2009 to 2014, TacTip development established a biomimetic optical tactile sensor inspired by the tactile contact lens and human skin, then refined its sensing, form factor, and integration.

  • Initial design: The TacTip emerged in 2009 as an optical tactile sensor based on biologically inspired encoding.The name TACTIP was coined as a contraction of “tactile fingertip” and later standardized as TacTip.
  • Biomimetic inspiration: The tactile contact lens inspired TacTip’s mechanism for magnifying surface indentation into lateral marker movement and shear strain.An array of pins acts as levers, paralleling the proposed mechanical structure of human tactile skin.
  • Tactile image processing: Initial TacTip studies converted camera images into marker velocity fields, inspired by motion-sensitive Meissner corpuscles.The resulting fields made surface edge features, such as coin contours, visible for biologically inspired edge encoding.
  • Early refinements: Early refinements miniaturized the tactile tip to a 20 mm-diameter dome and introduced a high-frame-rate camera for texture studies.In both cases, the camera remained separate from the sensor design.

VI. PERIOD II (2015-): THE TACTIP FAMILY

From 2015 onward, TacTip research adopted multi-material 3D printing, modular design, and marker tracking to create a coherent family of tactile sensors, hands, and robotic systems.

  • Period II: The second development period introduced multi-material 3D printing and modular design as common approaches for expanding TacTip research.These approaches supported diversification into a family of tactile sensors, hands, and robotic systems.

A. 3D-printed TacTip and integration into robot hands

Multi-material 3D printing transformed TacTip manufacture into a modular platform spanning customized sensors, robot hands, and integrated robotic systems, while biomimetic tip designs improved sensing and grasping.

  • Fabrication and platform development: Multi-material 3D printing enabled rapidly prototyped TacTip probes, grippers, and manipulators.The approach supported a family of sensors and robotic systems with varied bodies, tips, cameras, and integrations.
  • Sensor family: The TacTip family expanded from the 2009 TACTIP and Open TacTip to specialized variants including TacCylinder, TacWhisker, TacFoot, NeuroTac, and miniaturized fingertip sensors.Table II records differences in design, dimensions, pin counts, cameras, resolution, and frame or event rates.
  • Application-specific designs: 3D printing enabled customization for applications ranging from tactile walking feet to rodent-inspired whiskers.The TacTip family therefore extended beyond fingertip sensing into different robot morphologies and tasks.
  • Biomimetic tip design: A raised-bump fingerprint improved spatial acuity, while a ringed fingerprint encouraged outer contact motion before global slip to provide reaction time.The latter design supported incipient-slip detection before grasp loss.
  • Robot-hand integration: TacTip integration progressed from single-finger and two-finger grippers to the three-fingertipped Tactile Model O.The Tactile Model O replaced all three fingertips with miniaturized TacTips for multipurpose soft optical tactile sensing.
  • Integrated capabilities: The Tactile Model O achieved 93% object classification on 26 objects and 95% grasp-success prediction on the same objects.Slip detection enabled rapid re-grasping of 11 slipping objects, including 6 novel and 1 compliant object.

B. Progress in tactile capabilities

TacTip tactile capabilities progressed from interpretable marker-based representations and model-based control toward more continuous learning methods, while accuracy improved substantially.

  • Tactile capabilities: Deepened tactile capabilities included servo control over unknown 3D objects, tactile-only pushing, single-fingered braille typing, and object-related manipulation skills.These capabilities combine tactile perception with robot control in tasks requiring interaction with unknown or changing objects.
  • Representations: From 2015–19, marker-deflection time series provided an efficient tactile-image representation with a biomimetic analogue in mechanoreceptor activity.Pin displacements were also readily visualized for interpreting tactile sensing.
  • Perception and control: Perception and control methods included histogram likelihoods, feature-position prediction, Gaussian-process regression, polynomial regression, online latent-variable learning, and support-vector-machine slip detection.These methods addressed rolling, contour following, multi-fingered grasping, online learning, and dynamic object slippage.
  • Accuracy and sensitivity: 0.1–0.5 mm accuracy improved on initial ∼1 mm performance, while the best reported sensitivity detected ≲10 microns indentation using signal averaging.The review describes a progression from hyperacuity finer than pin spacing to sensitivity at the pixel level of the tactile image.

VII. PERIOD III (2019-): DEEP LEARNING WITH THE TACTIP

Deep learning produced a step-change in TacTip capabilities, enabling real-time tactile interaction with complex 3D objects and diverse manipulation tasks. Convolutional neural networks supported direct prediction from tactile images, robustness to variation, scalability, and generalization, while deep reinforcement learning remains constrained by physical-robot training times.

  • Deep learning enabled real-time TacTip interaction with complex 3D objects, advancing beyond earlier 1D rolling and 2D shape-exploration demonstrations.The review identifies this as a step-change in capability.
  • ConvNets predicted quantities directly from tactile images and were relatively robust to lighting changes, skin wear, dust, and accidental sensor damage.The review also describes straightforward scaling from 2D to 3D servo control and accurate predictions under varied stimulus or task conditions.
  • TacTip systems used deep learning for pose-based servoing, tactile pushing, Braille typing, item recognition, grasp-success prediction, and in-hand manipulation.These examples span single tactile sensors on robot arms and tactile robot hands.
  • Convolutional neural networks were critical across the diverse tactile capabilities shown, including robot-arm tasks and tactile-hand manipulation.Figure 5 covers both single-sensor robot arms and the Tactile Model O, Tactile Modular Grasper, and Pisa/IIT SoftHand.
  • Deep reinforcement learning learned Braille-keyboard navigation, but long physical-robot training times make simulated tactile environments more practical for broader skill acquisition.Most Figure 5 tasks used supervised learning, whereas tactile deep reinforcement learning required many hours of training for the Braille task.

VIII. CONCLUSION

The review frames human-like robot dexterity as a fundamental challenge in intelligent interaction with complex environments. It presents SoftBOT sensing and TacTip research as a route toward recreating aspects of human touch and manual intelligence in robots.

  • A major gap remains between laboratory robot manipulation and human dexterity and touch during intelligent interaction with complex environments.The review identifies bridging this gap as a central problem for engineering and robotics.
  • SoftBOT sensors combine Soft, Biomimetic, Optical, and Tactile sensing to recreate key aspects of human touch and manual intelligence.The TacTip is presented as an example of this approach.
  • This agenda aims both to advance knowledge of haptic intelligence and to improve robot dexterity with accessible hardware and software.The review describes these as two interconnected goals of SoftBOT sensing.
Loading 2105.14455v2…