Source-linked AI summary
HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning
An Dang, Jayjun Lee, Mustafa Mukadam, X. Alice Wu, Bernadette Bucher, Manikantan Nambi, Nima Fazeli
TL;DR
Tactile sim-to-real transfer remains difficult because simulated contact dynamics do not match high-fidelity tactile feedback, while visual fidelity has outpaced dynamic fidelity. HydroShear addresses this with a path-dependent hydroelastic model, achieving more faithful shear simulation and robust zero-shot transfer across the evaluated tasks.
Problem
Tactile reinforcement-learning policy transfer remains challenging because simulated contact dynamics differ from high-fidelity tactile feedback, while visual fidelity has outpaced dynamic fidelity.
Method
HydroShear is a non-holonomic hydroelastic tactile simulator using SDFs and path-dependent contact tracking to model realistic shear from full SE(3) interactions.
Results
HydroShear achieves consistently lower shear-field error and higher directional alignment than the compared method, and robust zero-shot transfer across all evaluated tasks.
Takeaways & Limitations
Tracking shear accumulation, dissipation, and stick–slip transitions supports one policy architecture and training pipeline across evolving contact configurations and slipsensitive force modulation.
Takeaways & Limitations
The method depends on a physics engine permitting penetrating contact simulation, while deformable objects may require alternative representations that affect point tracking and force computation.
Abstract
from arXiv · showhide
In this paper, we address the problem of tactile sim-to-real policy transfer for contact-rich tasks. Existing methods primarily focus on vision-based sensors and emphasize image rendering quality while providing overly simplistic models of force and shear. Consequently, these models exhibit a large sim-to-real gap for many dexterous tasks. Here, we present HydroShear, a non-holonomic hydroelastic tactile simulator that advances the state-of-the-art by modeling: a) stick-slip transitions, b) path-dependent force and shear build up, and c) full SE(3) object-sensor interactions. HydroShear extends hydroelastic contact models using Signed Distance Functions (SDFs) to track the displacements of the on-surface points of an indenter during physical interaction with the sensor membrane. Our approach generates physics-based, computationally efficient force fields from arbitrary watertight geometries while remaining agnostic to the underlying physics engine. In experiments with GelSight Minis, HydroShear more faithfully reproduces real tactile shear compared to existing methods. This fidelity enables zero-shot sim-to-real transfer of reinforcement learning policies across four tasks: peg insertion, bin packing, book shelving for insertion, and drawer pulling for fine gripper control under slip. Our method achieves a 93% average success rate, outperforming policies trained on tactile images (34%) and alternative shear simulation methods (58%-61%).
I. INTRODUCTION
Tactile sensing remains a bottleneck for contact-rich manipulation because simulated contact dynamics often diverge from real tactile feedback. Existing approaches improve visual rendering but inadequately model shear, while accurate FEM simulation is too expensive for scalable RL.
- Tactile sim-to-real transfer remains challenging because simulated contact dynamics differ from high-fidelity tactile feedback during real contact events.
- Vision-based tactile images primarily encode contact geometry and do not adequately model tactile shear, which captures sensor-object force interactions.
- Existing shear simulations often struggle to bridge the reality gap, whereas FEM is accurate but computationally too expensive for scalable reinforcement learning.
- HydroShear models path-dependent contact, full SE(3) motion, and stick-slip behavior using SDFs to generate efficient, physics-based shear fields.The method tracks object motion history across the sensor membrane and is GPU-parallelizable and physics-engine agnostic.
II. RELATED WORKS
Tactile simulation research seeks a compromise between FEM’s fidelity and faster physical approximations. HydroShear addresses this trade-off with GPU-parallelizable hydroelastic modeling and SDF-based tracking for complex geometries.
- FEM-based tactile simulation can accurately model shear but remains computationally expensive, limiting scalable policy training.
- Physical contact approximations run faster than FEM but generally sacrifice fidelity, motivating efforts to improve accuracy without losing speed.
- Rigid-body, soft-contact, penalty-based, and hydroelastic approximations struggle with sim-to-real transfer because they do not model tactile shadowing effects.
- HydroShear tracks full SE(3) motion of SDF-based complex geometries relative to the elastomer and supports GPU-parallelized large-scale policy training.The authors also provide a GPU-parallelized implementation of FOTS.
III. METHODOLOGY
HydroShear represents tactile shear fields generated by sensor-object interaction and maps tactile query points plus indenter-pose histories to marker displacements. The RL pipeline trains privileged teachers and tactile students in simulation for real-world deployment.
- III. METHODOLOGY: HydroShear simulates tactile shear fields induced by physical interaction between a robot’s soft tactile-sensor elastomer and indenter objects.The resulting shear field serves as a proxy for forces transmitted between the robot and object.
- III. METHODOLOGY: The RL procedure trains a privileged-state teacher before training a tactile student with an expert critic, then deploys the student actor in the real world.The student uses end-effector pose, relative goal pose, and left and right tactile shear; PPO and encoder-LSTM-MLP networks are used.
- III. METHODOLOGY: The model maps N tactile grid query points and a sequence of indenter poses in the elastomer frame to a marker displacement field.Each query point has a 2D shear vector representing marker displacement.
- III. METHODOLOGY: The total marker displacement field is decomposed into dilation and shear fields.
- III. METHODOLOGY: Shear depends on tangential translation or SO(3) rotation during contact and on the indenter-pose history.
A. Dilation
HydroShear computes dilation from the influence of indenter-contacting tactile grid points, with signed-distance queries determining contact and distance-based effects modeling elastomer deformation.
- A. Dilation: The dilation vector field aggregates the influence of tactile grid query points that are in contact with the indenter.
- A. Dilation: A tactile grid point is considered in contact when the indenter’s signed distance function is negative.
- A. Dilation: The contact-point index set contains tactile grid points whose signed distance to the indenter is below zero.
- A. Dilation: A distance-dependent influence term dissipates each contact point’s effect across queried 2D grid coordinates to model tactile shadowing from elastomer deformation.
B. Shear
HydroShear models tactile shear by recursively tracking in-contact indenter motion, converting it into friction-limited forces, and projecting those forces onto the elastomer surface and tactile grid.
- Force tracking: HydroShear tracks contact forces on indenter surface points to represent their 3D displacement relative to the compliant elastomer.Surface-point coordinates are obtained by transforming local indenter points with the indenter pose in the elastomer frame.
- Force tracking: The tracker retains only the portion of each surface-point displacement occurring inside the elastomer, using signed-distance changes between consecutive poses.The in-contact fraction is 1 for continuous penetration, 0 for continuous separation, and between 0 and 1 when the point crosses the elastomer boundary.
- Contact forces: Tracked displacement is decomposed into normal and tangential components, converted to forces using elastomer stiffness, and constrained by nonnegative normal force and Coulomb friction.The friction coefficient determines the tangential force threshold before sliding, and forces reset when contact breaks.
- Force projection: HydroShear projects each contact force onto the corresponding elastomer surface point and recursively tracks projected displacement to represent slippage and viscoelastic effects.The projected tracker uses a slippage parameter, while contact-force tracking follows the interaction history across the membrane.
- Force projection: Under E = K, µ = ˆµ, and uniform point area A, one tracker can recover the other by scaling projected displacement by KA.This reduces the computation needed for the shear field while preserving the specified parameter relationship.
- Shear field: The shear vector field aggregates projected contact effects over tactile grid coordinates, weighting penetration depth and exponentially dissipating influence with distance.The model uses four parameters: λd, λs, K, and µ.
C. Calibration
HydroShear calibrates its four parameters separately by isolating normal penetration, shear, stiffness, and slipping effects in real-world tactile data.
- Parameter calibration: The model calibrates λd from real tactile data collected while pressing the indenter at multiple elastomer positions.Calibration uses a least-squares optimization problem for the dilation-related parameter.
- Parameter calibration: The friction coefficient µ is calibrated from data capturing indenter slipping on the elastomer.The calibration is performed after the preceding parameters have been estimated.
- Parameter calibration: The shear parameter λs is calibrated from data collected while pressing and translating the indenter in contact with the elastomer.Previously calibrated parameters are used to estimate the full vector field during this stage.
- Parameter calibration: The stiffness K is calibrated using real shear data without the earlier rescaling trick.The calibration follows estimation of the rescaled real shear vector field.
- Calibration strategy: Separate single-variable optimizations simplify calibration because the original joint problem is heavily nonlinear and good optima are not guaranteed.The procedure isolates each parameter’s effect in the real-world data.
D. Sim-to-Real Reinforcement Learning
The authors train tactile-feedback policies in parallelized Isaac Gym simulation and deploy them zero-shot in the real world using asymmetric teacher–student distillation.
- Policy training: HydroShear policies are trained entirely in Isaac Gym with parallelized environments using PPO, then deployed zero-shot without modification or fine-tuning.The simulated and real settings differ in access to the state space.
- Teacher training: A privileged-state teacher is first trained with a curriculum that adds contact penalties only after initial task success.This avoids the suboptimal behavior observed when agents are trained naively with contact penalties.
- Teacher–student distillation: Asymmetric actor-critic distillation uses the privileged teacher critic to train each student policy actor from scratch.All task students use the same teacher critic checkpoint trained with the contact-penalty curriculum.
IV. EXPERIMENTS AND RESULTS
The experiments evaluate HydroShear’s shear modeling under diverse full-SE(3) motions and its ability to transfer reinforcement-learning policies from simulation to reality without task-specific modification.
- Shear simulation: The evaluation measures HydroShear’s ability to model diverse real-world shear modes induced by full SE(3) indenter motion relative to the elastomer.This axis tests the simulator’s shear fidelity under varied contact configurations.
- Validation: A separate validation dataset with unseen contact configurations is used to assess shear simulation accuracy.The experiments first describe a real-world calibration setup for system identification before validation.
- Sim-to-real transfer: The second evaluation axis tests whether reinforcement-learning policies trained in simulation transfer zero-shot to real-world tasks.This assesses policy transfer separately from the simulator’s shear-field accuracy.
A. Calibration and Real-world Shear Evaluation
HydroShear is calibrated in a digital-twin setup that reproduces controlled indenter motions on a GelSight Mini, then evaluated against real marker-displacement shear fields across four shear types.
- Calibration setup: The digital twin replicates real-world sphere-indenter motions to calibrate baseline and HydroShear tactile simulation models.The setup applies dilation and shear to the sensor elastomer in both real and simulated environments.
- Calibration setup: Calibration uses a 7-DoF KUKA robot with a 35-mm spherical indenter and a GelSight Mini rigidly mounted to the table.The procedure assumes access to the full poses of the tactile sensor and sphere indenter.
- Parameter fitting: Calibration collects vertical indentation samples for λd and combined normal-tangential motions for jointly fitting λs and K to measured shear fields.The samples use marker displacement fields recorded from the real tactile sensor.
- Shear evaluation: Evaluation reports per-taxel shear RMSE [px] and cosine similarity (CS) against ground-truth shear vectors for dilation, translational, torsional, and rolling shear.The calibration results use 10 samples per shear type, totaling 40 test samples.
- Shear evaluation: HydroShear achieves consistently lower error and higher directional alignment across all evaluated shear types than the compared methods.TacSL lacks tactile shadowing, while FOTS degrades under tilt motions involving out-of-plane SE(3) components.
B. Zero-shot Sim-to-Real RL
The paper evaluates five tactile-policy variants across four contact-rich tasks, contrasting image-based and shear-based feedback with HydroShear. HydroShear supports robust zero-shot transfer as contact configurations evolve, while baseline performance depends on contact stability, shear representation, and reference-frame alignment.
- Task suite: The four tasks target pose-uncertain peg insertion, multi-object bin packing, book shelving insertion, and fine gripper control during drawer pulling.Peg testing spans 3 goals and 10 in-hand orientations per goal.
- Policy evaluation: Five policy variants compare TacSL grayscale images, TacSL normalized shear, two FOTS implementations, and HydroShear tactile shear.The evaluation covers peg insertion, bin packing, book shelving, and drawer pulling.
- Baseline behavior: TacSL Gray transfers moderately on Peg Insertion and Bin Packing but performs substantially worse on Book Shelving and Drawer Pulling.Full fingertip contact hinders textures, while image feedback lacks explicit shear direction, accumulation, and slip cues.
- Baseline behavior: TacSL Shear performs strongly on most tasks but degrades notably on Bin Packing because per-taxel normalization discards shear magnitude and spatial contact structure.Large shear and torsional interactions create broader contact patches and global elastomer deformations than the policy expects.
- Baseline behavior: Original FOTS performs poorly on Peg Insertion and Bin Packing when its object-frame center is far from the contact patch center.The resulting tactile signals fail to reflect local contact dynamics.
- HydroShear transfer: HydroShear achieves robust zero-shot transfer across all tasks by tracking path-dependent SE(3) contact interactions and evolving stick-slip shear.This supports one policy architecture and training pipeline across insertion, multi-object interaction, full-contact manipulation, and slip-sensitive force modulation.
- Training ablation: Contact-penalty curriculum improves downstream student-policy behavior during sim-to-real transfer.Without it, peg policies make aggressive repeated socket contact; with it, policies make controlled contact and show emergent goal-searching behavior.
V. DISCUSSION AND LIMITATIONS
HydroShear’s applicability depends on simulation and sensor assumptions, while its training pipeline combines staged teacher learning, student distillation, and task randomization. The method also has representation and scope boundaries involving SDF resolution, object deformability, elastomer shape, and sensor type.
- Limitations: HydroShear depends on a physics engine that permits penetrating contact simulation, implemented here with Isaac Gym TacSL’s Kelvin-Voigt compliant-contact model.
- Limitations: Higher SDF resolution can improve shear accuracy but slows simulation by increasing the number of points tracked during penetration.
- Limitations: SDFs are particularly effective for rigid bodies, whereas deformable objects may require alternative representations and tracking algorithms.
- Limitations: HydroShear assumes a flat sensor elastomer, with curved elastomers on dexterous fingertips left for future work.
- Limitations: The evaluation targets deformation-based vision tactile sensors, while extension to other tactile sensors remains outside the current evaluation scope.
- Training Pipeline: The training procedure first learns an efficient teacher, then adds contact penalties, and finally distills a student actor using the finetuned teacher’s critic.
I. Speed Comparison between FOTS, FOTS (Reimpl.), and HydroShear
The speed comparison evaluates FOTS, a reimplemented FOTS, and HydroShear across increasing rollout batch sizes. Batched operations improve both reimplemented FOTS and HydroShear over the original, but HydroShear remains limited by SDF computation.
- Speed Comparison: Both reimplemented FOTS and HydroShear outperform original FOTS because their algorithms are more parallelizable.
- Speed Comparison: Reimplemented FOTS has the most consistent performance, with mean compute time varying by only 1ms as the number of environments increases.
- Speed Comparison: Original FOTS is slowest because it does not use batched operations, whereas the reimplementation incurs little overhead as environments increase.
- Speed Comparison: HydroShear is much faster than FOTS, but its mean compute time increases with environment count because SDF calculation remains a bottleneck.
- Limitations: Improved batched SDF algorithms are expected to increase HydroShear’s computational speed and parallelizability.