PaperlayerView exact revision on arXiv

Source-linked AI summary

Taxim: An Example-based Simulation Model for GelSight Tactile Sensors

Zilin Si, Wenzhen Yuan

arXiv:2109.04027v2cs.RO

TL;DR

Tactile sensing remains difficult to integrate into robotics simulation, especially for GelSight sensors that combine elastomer deformation with optical measurement. Taxim uses example-based optical and marker-motion models, achieving realistic, computationally efficient simulation with calibration from fewer than 100 real-sensor examples.

  • Problem

    Robotics simulators generally lack integrated tactile sensing, while GelSight simulation must model both elastomer deformation and optical response.

  • Method

    Taxim maps contact geometry to pixel intensities with a polynomial table and models elastomer marker motion using linear deformation and superposition.

  • Results

    Taxim’s optical simulation outperforms other state-of-the-art tactile simulations, while its marker-motion simulation achieves high accuracy on a self-designed dataset.

  • Takeaways & Limitations

    The example-based framework is lightweight, easy to apply across sensors, and incorporates illumination features and sensor noise through real-sensor calibration.

Abstract

from arXiv · show

Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile sensors, has been a long-standing challenge. In this paper, we propose Taxim, a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight. A GelSight sensor uses a piece of soft elastomer as the medium of contact and embeds optical structures to capture the deformation of the elastomer, which infers the geometry and forces applied at the contact surface. We propose an example-based method for simulating GelSight: we simulate the optical response to the deformation with a polynomial look-up table. This table maps the deformed geometries to pixel intensity sampled by the embedded camera. In order to simulate the surface markers' motion that is caused by the surface stretch of the elastomer, we apply the linear elastic deformation theory and the superposition principle. The simulation model is calibrated with less than 100 data points from a real sensor. The example-based approach enables the model to easily migrate to other GelSight sensors or its variations. To the best of our knowledge, our simulation framework is the first to incorporate marker motion field simulation that derives from elastomer deformation together with the optical simulation, creating a comprehensive and computationally efficient tactile simulation framework. Experiments reveal that our optical simulation has the lowest pixel-wise intensity errors compared to prior work and can run online with CPU computing. Our code and supplementary materials are open-sourced at https://github.com/CMURoboTouch/Taxim.

I. INTRODUCTION

Taxim addresses the lack of integrated tactile sensing in robotics simulation by combining GelSight optical response and marker motion simulation in a lightweight, example-based model.

  • Existing robotics simulators model rigid bodies, soft bodies, vision, and laser sensors, but do not integrate tactile sensing.This leaves tactile simulation outside widely used frameworks despite tactile sensing being important to robotic systems.
  • GelSight uses an elastomer, printed markers, LEDs, and a camera to capture deformation and infer contact information.Marker motion indicates contact forces and torques, while optical changes arise from altered light reflection at the gelpad surface.
  • With ground-truth geometry, Taxim generates tactile images that are very similar to real GelSight outputs for richly textured objects.
  • Taxim combines a polynomial geometry-to-pixel-intensity mapping with a marker-motion model based on linear elastic deformation and superposition.The model also accumulates shadow effects from illumination and simulates the elastomer’s marker motion.
  • Calibrated with fewer than 100 contact examples, Taxim is designed to migrate across vision-based tactile sensors with similar designs.The authors present it as a comprehensive simulator covering optical geometry measurement and marker-based force/torque measurement.

A. Tactile Sensing Simulation

Prior tactile simulators separately approximate elastomer deformation, optical behavior, or sensor-specific dynamics, motivating an integrated and efficient approach for vision-based tactile sensors.

  • Soft-medium tactile simulation commonly models elastomer deformation with FEM, mass-spring, particle, or learning methods.These approaches approximate the soft body that mediates contact information.
  • GelSight simulation additionally requires optical modeling because reflected light is used to measure contacted-object geometry.Prior work includes Phong reflection models and ray tracing for sensor illumination and light transport.
  • TACTO links vision-based tactile rendering with PyBullet, while Taxim is data-driven and targets computational efficiency and intrinsic sensor noise.
  • Marker-array motion is driven by planar elastomer stretch and supports manipulation tasks such as slip detection and grasping-stability prediction.Prior work simulated TacTip membrane dynamics to extract marker motion, highlighting the role of marker behavior in tactile sensing.

A. Overview

Taxim combines example-based optical rendering and elastomer marker-motion simulation, calibrating both components from real-sensor examples to reproduce tactile contact.

  • Taxim maps contact geometry to tactile-image intensities with a polynomial table and attaches shadow masks collected from real sensor examples.The optical component models the sensor’s illumination response from contact geometry.
  • It simulates marker motion by applying the superposition principle to loading displacements of finite elastomer units.
  • Both simulation components are calibrated with examples from a real sensor and combined to replicate object contact on GelSight.The overall construction and application pipeline is illustrated in Fig. 2.

B. Optical Simulation

Taxim simulates GelSight optical responses by mapping contact geometry to pixel intensity with example-based photometric stereo. It combines polynomial mapping, calibration, deformation approximation, and shadow synthesis to generate tactile images.

  • Optical response: A polynomial table maps surface geometry to tactile-image intensity, replacing explicit lighting knowledge with example-based photometric stereo.The model uses surface normals and image-plane location as inputs to approximate the optical response.
  • Polynomial mapping: A second-order polynomial compensates for the GelSight LEDs’ nearby, nonparallel, and spatially nonuniform illumination.The polynomial parameters are calibrated per sensor from real contact data.
  • Calibration: The data-collection setup calibrates the polynomial table with fewer than 100 spherical-indenter points and collects shadow masks with 10 pin-indenter points.An XYR stage controls location and rotation while a vertical stage controls indentation.
  • Calibration: Calibration fits the polynomial mapping from a known-radius spherical indenter and discretizes surface normals into a 125 × 125 table.The spherical indenter provides known geometry for locating contact regions and calculating surface normals.
  • Simulation pipeline: The simulation constructs a height map from contact geometry, approximates gelpad deformation, extracts surface normals, and maps them to intensities.Contact-area shape is preserved while contact boundaries are smoothed before optical synthesis.

C. Shadow Simulation

Taxim models GelSight shadows by decomposing general contact geometries into unit pin-shadow cases. It accumulates direction-specific shadow masks around the contact area to synthesize the three color channels.

  • Shadow synthesis: Shadow effects are simulated by collecting unit shadow masks at different pin-indentation depths and linearly accumulating their contributions.The accumulation follows from treating the three light beams as independent and ignoring inter-reflection.
  • Shadow synthesis: General objects are approximated as side-by-side unit pin shadows, whose masks are attached where neighboring contact points are lower.The method applies masks for the three dominant light directions and all color channels.

D. Marker Motion Field Simulation

Taxim models GelSight marker motion by combining linear displacement relationships with superposition over elastic deformation under normal and shear loads. It calibrates deformation influence using FEM data, then computes active-node virtual displacements and resultant displacements across dense surface nodes.

  • D. Marker Motion Field Simulation: Taxim composes gelpad deformation under normal and shear loads using linear displacement relationships and the superposition principle.Each node’s displacement aggregates the influence of active contact nodes on the gelpad surface.
  • D. Marker Motion Field Simulation: Active nodes contact the object and receive external forces, while passive nodes lie outside contact and are constrained only by internal elastic forces.
  • D. Marker Motion Field Simulation: Virtual displacements correct active-node initial displacements for mutual interactions before superposition produces final displacements for all nodes.The correction is needed because active nodes are constrained by both external loads and their interactions with one another.
  • D. Marker Motion Field Simulation: Taxim calibrates the deformation tensor T using dense gelpad deformation measurements from ANSYS under a unit-node load.The FEM calibration supplies dense deformation information because the real gelpad’s printed markers are sparsely distributed.
  • D. Marker Motion Field Simulation: The simulation applies initial active-node displacements, solves active-node virtual displacements, and calculates resultant displacements at every node.This three-step process covers both active nodes in contact and passive nodes in non-contact areas.

IV. EXPERIMENTS

The experiments evaluate how closely simulated tactile data matches data from real GelSight sensors. The evaluation is framed as a similarity assessment between simulated and real tactile data.

  • IV. EXPERIMENTS: The experiments assess similarity between simulated tactile data and data collected from real GelSight sensors.

A. Experiment Setup and Data Collection

The setup collects controlled GelSight contacts by positioning an indenter over a sensor mounted on a precision XYR and vertical-stage platform. Objects vary in shape and are represented both as simulation meshes and 3D-printed physical samples.

  • A. Experiment Setup and Data Collection: The GelSight is mounted on an XYR stage, while a vertical linear stage controls indenter position and contact depth with 0.01 mm precision.The XYR stage enables horizontal movement, and the vertical stage adjusts indenting depth.
  • A. Experiment Setup and Data Collection: Contact experiments use objects with different shapes, designed in Solidworks as simulation meshes and 3D printed for real-sensor data collection.
  • A. Experiment Setup and Data Collection: The object dataset uses base sizes of either 10mm × 10mm or 15mm × 15mm.

B. Optical Simulation

Taxim calibrates an example-based optical simulation model from fewer than 100 contact examples and evaluates it across depths, locations, sensors, and objects. It outperforms the compared methods on image similarity and runs fastest on CPU.

  • The optical model is calibrated with 50 surface-location samples and 10 shadow samples, requiring fewer than 100 contact examples overall.Calibration can be conducted manually within 1 hour without precise control of contact locations.
  • Taxim outperforms the physics-based model, TACTO, and Phong’s model on L1, MSE, SSIM, and PSNR image-similarity metrics.The comparison uses manually aligned 400 × 400 tactile-image crops against real data.
  • MSE errors increase as indentation becomes deeper and farther from the gelpad center, although the model works across indentation depths and locations.The tested depths are 0.5 mm, 1.0 mm, and 1.5 mm.
  • The model simulates fine-textured objects and transfers across four GelSight sensors, a DIGIT sensor, and objects from the Google Scan dataset.The paper attributes differing tactile-image appearances to sensor-manufacturer differences while reporting that the model works across them.
  • Taxim runs at the fastest CPU speed among the compared optical simulators and achieves real-time data-transfer speed from real sensors.The comparison uses 480 × 640 height maps on an AMD Ryzen Threadripper 2950X CPU.

C. Marker Motion Field Simulation

Taxim simulates GelSight marker motion using elastomer-deformation principles and compares its displacement fields with FEM and real-sensor data. Its results match FEM closely, but a gap remains relative to the real gelpad, while CPU simulation is substantially faster than FEM.

  • Comparison with FEM simulation: Taxim’s marker-motion field matches FEM with average interpolated pixel-wise L1 errors of 3.58 × 10^-3 mm in X, 3.32 × 10^-3 mm in Y, 5.43 × 10^-3 mm in Z, and 5.40 × 10^-3 mm in XY.The comparison covers dense-mesh displacements under combinations of normal and shear loads.
  • Comparison with both real data and FEM simulation: Against real marker motion, Taxim’s magnitude L1 error is 1.02 × 10^-2 mm and weighted angular L1 error is 14.57°, versus 3.96 × 10^-3 mm and 4.89° against FEM.The corresponding real–FEM errors are 1.00 × 10^-2 mm and 12.94°.
  • Comparison with both real data and FEM simulation: The remaining gap to the real gelpad reflects hand-manufacturing mismatch, marker-tracking noise, and unmodeled partial slip under shear loads.Partial slip is common in real contact cases but is not modeled by Taxim.
  • Speed Testing: Dense marker-motion simulation takes 9.22 seconds on CPU on average, compared with 2–4 hours for FEM simulation in ANSYS.The FEM comparison is also CPU-only.
  • The combined optical and marker-motion simulations produce final tactile results, including marker fields evaluated against dense FEM and sparse real-sensor references.The displacement references span load displacements from 0.3 mm to 0.8 mm.

V. CONCLUSION

Taxim integrates optical response and marker motion simulation for GelSight sensors in a lightweight, adaptable example-based model. The authors report strong optical and marker-motion performance, while identifying dynamic contact and GPU acceleration as future extensions.

  • Taxim combines optical simulation with marker motion field simulation in an example-based GelSight model that is computationally lightweight and adaptable to different sensors.The model uses real-sensor examples to incorporate illumination features and system noise during calibration.
  • The optical simulation outperforms other state-of-the-art tactile simulations, while the marker motion field simulation achieves high accuracy on a self-designed dataset.
  • The current pipeline models quasi-static contact; future work will address dynamic phenomena such as slip and improve computation with GPU acceleration.
Loading 2109.04027v2…