Source-linked AI summary
TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, Roberto Calandra
TL;DR
Vision-based tactile sensing remains difficult to simulate because realistic touch readings require modeling contact and optical phenomena while remaining fast and flexible. TACTO addresses this gap with an open-source simulator that renders configurable tactile outputs and demonstrates its use across perception, control, and Sim2Real tasks.
Problem
Realistic simulation of vision-based tactile sensing requires modeling contact dynamics, optical properties, and illumination while remaining flexible and fast enough for practical use.
Method
TACTO uses a modular simulator architecture that renders vision-based tactile outputs, interfaces with physics engines, and supports configurable sensor designs.
Results
TACTO generates realistic high-resolution tactile readings at hundreds of frames per second and is demonstrated on grasp-stability learning, marble manipulation, and Sim2Real applications.
Takeaways & Limitations
TACTO provides an open-source tool for studying touch in robotic perception, control, hardware design, and machine learning without requiring all experiments to use real-world tactile hardware.
Abstract
from arXiv · showhide
Simulators perform an important role in prototyping, debugging, and benchmarking new advances in robotics and learning for control. Although many physics engines exist, some aspects of the real world are harder than others to simulate. One of the aspects that have so far eluded accurate simulation is touch sensing. To address this gap, we present TACTO - a fast, flexible, and open-source simulator for vision-based tactile sensors. This simulator allows to render realistic high-resolution touch readings at hundreds of frames per second, and can be easily configured to simulate different vision-based tactile sensors, including DIGIT and OmniTact. In this paper, we detail the principles that drove the implementation of TACTO and how they are reflected in its architecture. We demonstrate TACTO on a perceptual task, by learning to predict grasp stability using touch from 1 million grasps, and on a marble manipulation control task. Moreover, we provide a proof-of-concept that TACTO can be successfully used for Sim2Real applications. We believe that TACTO is a step towards the widespread adoption of touch sensing in robotic applications, and to enable machine learning practitioners interested in multi-modal learning and control. TACTO is open-source at https://github.com/facebookresearch/tacto.
I. INTRODUCTION
TACTO addresses the difficulty of simulating vision-based tactile sensing by providing an open-source simulator designed to be fast, flexible, and capable of realistic perceptual outputs. The paper demonstrates it on grasp-stability learning, marble manipulation, and a Sim2Real proof of concept.
- Vision-based tactile sensors are difficult to simulate because realistic outputs require modeling contact dynamics, optical properties, and illumination simultaneously.
- TACTO is an open-source simulator explicitly designed to render realistic perceptual outputs from vision-based tactile sensors quickly and flexibly.
- TACTO interfaces natively with PyBullet and can also connect to other physics engines, while its modularity supports sensors with different form factors and lighting properties.
- The experiments use touch to learn grasp-stability models from a simulated dataset of 1 million grasps and to learn in-hand marble manipulation with Bayesian optimization.
- A Sim2Real pose-estimation proof of concept compares simulated and real signals and explores augmentation strategies for improving performance.
- TACTO is intended to support hardware design, robotic control, and machine-learning research by enabling simulation and generation of touch-based or multimodal inputs without real-world hardware.
II. RELATED WORK
Prior tactile simulators cover low-dimensional sensors or use rendering models, but often sacrifice speed, physical integration, or active interaction scenarios. TACTO positions itself as an open-source simulator for fast tactile rendering integrated with physics-based manipulation.
- Although many physics engines exist, tactile-sensor simulation remains limited because accurately and efficiently simulating touch is difficult.
- Earlier work simulated low-dimensional sensors using tactels, but independently modeling many sensing elements could severely reduce simulation speed.
- Vision-based tactile simulators have used Phong rendering, Mitsuba2, Unity, optical flow, and finite-element models.
- TACTO bridges physics simulation and rendering through a modular architecture and supports workflows that synchronize simulated scenes for tactile interaction.
- Concurrent work using Mitsuba2 focused on realistic tactile imprints without efficient physical-simulator integration, while Phong-and-Gazebo work mostly studied sensors pressing fixed objects.
III. A FAST AND FLEXIBLE SIMULATOR OF VISION-BASED TACTILE SENSORS
TACTO is designed around high throughput, flexibility, realistic rendering, and ease of use. Its examples show configurable simulation of DIGIT and the more complex OmniTact sensor designs.
- Design Desiderata: High throughput is a primary design goal because tactile simulation has traditionally been computationally intensive and barely faster than real time.
- Design Desiderata: Flexibility requires support for varied sensor geometries, mirrors, transparent cases, optical components, and mechanical designs.
- Sensor Examples: TACTO can generate simulated DIGIT color and depth imprints simultaneously, with local-geometry detail at high speed.
- Design Desiderata: Realistic simulation should reproduce illumination, global lighting, shadows, and deformation details at the contact boundary.
- Design Desiderata: Ease of use is treated as important for installation, modification, setup, and deployment across perception and control tasks.
B. Architectural Choices
TACTO compares rendering architectures and selects a synchronized-scene OpenGL design that avoids repeated mesh-transfer bottlenecks while retaining advanced rendering capabilities. The design is fast, but has scope boundaries for deformable objects and contact-boundary deformation.
- Architectural Choices: Rendering tactile output requires computing local contact geometry as depth and producing corresponding RGB images that satisfy the simulator’s design goals.
- Architectural Choices: Phong rendering from depth is simple but limited because single-bounce lighting makes reflection, refraction, and shadows difficult to support.
- Architectural Choices: OpenGL rendering from repeatedly generated depth meshes supports advanced effects, but mesh-loading I/O limited the method to 20 frames per second even on GPU.The reported bottleneck was loading meshes generated from depth maps into OpenGL.
- Architectural Choices: The proposed design preloads gel and object meshes, synchronizes poses from the physics simulator, and directly renders depth and RGB in OpenGL.Changing poses and rerendering is much faster than repeatedly loading meshes.
- Architectural Choices: TACTO can render at up to 200 frames per second by overlapping gel and contact-object geometries with synchronized poses.
- Limitations: The fast synchronized-scene method targets rigid objects with negligible deformation, while very deformable objects use slower RGB-from-depth rendering.
- Limitations: Simultaneously calculating RGB and depth makes contact-boundary gel deformation difficult to model, though smoothing, augmentation, or generative refinement can approximate greater realism.
- Architectural Choices: The authors selected the third architectural option after experiments and analysis because it best matched their goals of speed, flexibility, and power.
C. Overview of the Software Architecture
TACTO bridges a physics simulator and rendering engine to produce tactile signals, using synchronized scene information and configurable sensor components. Its architecture supports real-data fine-tuning and can interface with PyBullet or other physics engines.
- TACTO bridges the physics simulator and rendering engine to generate high-quality tactile signals efficiently.
- Real sensor readings can fine-tune rendering by combining a simulated touch difference with a reference real-world image.
- Sensor configurations specify cameras, lights, and gel-surface parameters, enabling different sensor renderings without changing the core architecture.
- The workflow initializes sensor configuration, creates a synchronized rendering scene from object meshes, and advances rendering after physics simulation.
- TACTO can be used with PyBullet and supports integration with other physics engines through scene synchronization or depth-based rendering.
D. Salient Features
TACTO is designed for fast, configurable tactile rendering with force-dependent gel deformation, realistic illumination, calibration from real sensors, and compatibility with multiple physics engines. It supports both DIGIT and more complex OmniTact sensor structures, while retaining limitations in boundary deformation and transfer-function realism.
- Fast: 200 frames per second is achieved for one DIGIT sensor at 160x120 resolution on GPU with a 12K-face object mesh.Four DIGIT sensors can be rendered at 50 frames per second under the same resolution and GPU setting.
- Flexible: TACTO adapts to different sensor designs through configuration files, including DIGIT and the more complex OmniTact structure.OmniTact includes a round surface, 5 cameras, and 11 light sources.
- Calibration from real sensors: TACTO captures non-uniform illumination across the sensor, and real readings can be used to fine-tune simulated renderings.The comparison uses a DIGIT sensor touching a 5.3 mm ball and overlays simulated signals on a real background image.
- Force dependent: Contact forces from the physics engine are mapped to gel deformation using a piece-wise linear function with lower and upper force thresholds.The lower threshold represents undetectable deformation, while the upper threshold represents saturated deformation.
- Limitations: More realistic sensor-specific transfer functions remain future work, beyond the current deformation mapping.
- Rendering from depth: TACTO can render tactile imprints from depth images, although input/output bottlenecks limit speed in this mode.
IV. SIMULATED EXPERIMENTS
TACTO is evaluated on simulated grasp-stability perception and touch-based marble manipulation, using PyBullet and tasks selected for comparison with prior real-robot studies. The simulations achieve results similar to those obtained with real robots.
- TACTO evaluates touch-based grasp-stability learning and marble manipulation between two fingers in simulation.
- The experiments use TACTO with the PyBullet physics engine and mirror tasks from prior real-robot work.
- The simulated experiments achieve results similar to those with real robots.
A. Learning Grasp Stability in Simulation
The grasp-stability study trains classifiers from simulated vision and touch readings using a self-supervised dataset of 1 million grasps. Touch learns effectively with far less data than vision, while combining vision and touch performs best in low-data settings.
- Task: The classifier predicts whether a grasped object will be successfully lifted from touch and vision readings recorded before lifting.
- Dataset and setup: 1 million grasps were collected in one day by randomizing gripper position, orientation, and force, with labels based on whether objects remained between the fingers after lifting.
- Results: Touch learned grasp stability with little data, whereas vision required 3 or 4 orders of magnitude more data to catch up.
- Results: Two tactile sensors performed significantly better than one because a grasp can appear stable from one side while contacting the other side unstably.
- Results: Combining vision and touch worked best in most low-data cases, consistent with previous real-world experiments.
- Results: In simulation, the models could be evaluated with 2 orders of magnitude more data than in the previous real-world study, while all models continued improving in the high-data regime.
B. Learning In-hand Marble Manipulation
TACTO and PyBullet are used to learn tactile marble manipulation with a parameterized controller optimized by Bayesian optimization. The system reaches target locations efficiently while rendering high-resolution tactile data at practical speed.
- Task setup: The setup uses two DIGIT sensors, with the lower sensor fixed and the upper sensor rolling a marble toward target locations.The task involves rich frictional contacts and evaluates simulation stability during active manipulation.
- Controller: The controller sets horizontal velocity from the marble’s tactile-space error using a learnable matrix K.The control law is u = Kx̄, where x̄ is the error between the current and goal marble locations.
- Optimization: Bayesian optimization with an upper confidence bound learns K by minimizing average cumulative tactile-space error across eight target locations.Averaging over eight directions improves robustness of the optimization objective.
- Results: 8 minutes sufficed for Bayesian optimization to learn marble manipulation with 50 iterations, including 2 minutes of simulation.The run rendered 20,000 tactile imprints at 160 × 120 resolution across eight rolling directions.
- Results: More iterations enable faster marble rolling into different target locations, while PyBullet and TACTO run without abnormal situations.The results provide both qualitative and quantitative validation of the simulated manipulation system.
V. SIM2REAL EXPERIMENTS
The Sim2Real experiment estimates a pen’s contact pose using simulated and real tactile imprints. Data augmentation narrows the simulation-to-reality gap, and mixing simulated data with limited real data outperforms Real2Real training with the same real-data amount.
- Qualitative results: Figure 12 shows simulated and real tactile readings alongside the corresponding estimated pose represented by a red line.The figure provides qualitative visual comparison for the pose-estimation experiment.
- Task and data: The task estimates the contact center and angle of a pen relative to the touched sensor.TACTO generated simulated tactile imprints with corresponding poses, while real imprints received manual pose annotations.
- Experimental design: 10,000 simulated tactile imprints and 200 real tactile imprints support Sim2Sim, Real2Real, Sim2Real, and Sim+Real training evaluations.The same convolutional neural networks are trained and validated across these dataset combinations.
- Data augmentation: Color jittering improves Sim2Real robustness by varying RGB-channel brightness and contrast across illumination conditions.The paper compares Sim2Real with and without augmentation to assess the effect of this intervention.
- Results: Sim2Real with augmentation achieves comparable results to Real2Real without using real training data.The comparison is reported for the pose-estimation errors summarized in Table II.
- Results: Sim2Real consistently outperforms Real2Real when trained with the same amount of mixed real data.The authors present this as evidence for potential data-efficiency gains from simulated data.
VI. CONCLUSION
The paper concludes that TACTO is an easy-to-use, fast, flexible simulator for realistic high-resolution vision-based tactile readings, validated across perception, control, and Sim2Real tasks. It is open-source, while future work targets more realistic elastomer deformation and additional transfer experiments.
- VI. CONCLUSION: TACTO generates realistic high-resolution tactile readings and is validated for grasp-stability perception, marble manipulation, and Sim2Real pose estimation.The simulator is demonstrated with PyBullet-based perception and control experiments.
- VI. CONCLUSION: TACTO is open-source to support broader use of touch simulation by robotics and machine-learning researchers.The release is provided through the project’s GitHub repository.
- Future work: Future work will improve elastomer force-deformation modeling to generate more realistic tactile readings.The paper also proposes evaluating deformation models, image filtering, sensor geometries, and transfer of grasp policies.