PaperlayerView exact revision on arXiv

Source-linked AI summary

Factorized Tactile Representation and Control for Sim-to-Real Manipulation

Siqi Shang, Bianca Aumann, Tye Brady, Joshua Migdal, Taskin Padir

arXiv:2610.10510v1cs.RO

TL;DR

Tactile sim-to-real learning must bridge simulated contact and device-specific sensor responses while preserving control-relevant information. The paper maps force and contact patch to a recoverable effective response, factorizes tactile cues, and keeps them separately selectable in one policy. Different tactile representations benefit different interactions, and the complete representation achieves 19/30 real-world adversarial peg-insertion success versus 14/30 for Response r.

  • Problem

    Tactile sim-to-real policies must transfer contact feedback shaped by contact mechanics and sensor-specific responses, while compact capacitive arrays’ distributed signals carry geometry, force, and temporal information.

    Source: §I · p. 1

  • Method

    The framework maps normal force and contact patch to an effective response recoverable from sensor readings, then separately encodes geometry, force distribution, and temporal change in a Tactile Gated Policy trained across all mask configurations.

    Source: §I · p. 2

  • Results

    Different tactile representations benefit different interactions, and GFC achieves 19/30 (63.3%) real-world adversarial peg-insertion success versus 14/30 (46.7%) for Response r.

    Source: 3) Adversarial Peg Insertion · pp. 7–8

  • Takeaways & Limitations

    Keeping tactile representations separately accessible allows their task-specific utility and transfer behavior to be assessed, with active representations changed at deployment without retraining.

    Source: 3) Adversarial Peg Insertion · p. 8, §VI · p. 8

Abstract

from arXiv · show

Tactile sim-to-real learning must bridge simulated contact and device-specific sensor responses while preserving information needed for control. We propose a factorized tactile representation and control framework that maps normal force and contact patch to an effective contact response recoverable from sensor readings. The response is separated into contact geometry, force distribution, and temporal contact change, with representation-specific encoding and randomization. A Tactile Gated Policy preserves these representations separately through control and operates over all mask configurations without retraining. We evaluate the approach through response reconstruction, spatial alignment, force regulation, and contact-rich adversarial peg insertion in simulation and the real world, enabling the utility and transfer reliability of different tactile representations to be assessed independently. The approach achieves <1 mm contact localization, 1.69 N force-tracking error on unseen geometries, and a 35% improvement in real-world adversarial peg insertion over the unfactorized response, with different tactile representations benefiting different interactions.

I. INTRODUCTION

Sim-to-real tactile learning must bridge simulated contact and device-specific sensor responses while preserving the information needed for control. The paper proposes a shared, factorized tactile interface and a policy that keeps its components independently selectable.

I. INTRODUCTION

  • Sim-to-real reinforcement learning offers scalable autonomous interaction, but tactile policies must transfer contact feedback shaped by both contact mechanics and sensor-specific responses.Acquiring diverse physical interactions for demonstration datasets requires substantial hardware resources and human effort [1].Source: §I · p. 1
  • The approach targets compact capacitive arrays, whose distributed responses carry geometry, force, and temporal information that binary contact or compact force-location summaries reduce.The framework keeps those forms separately accessible because their useful information and tolerated uncertainty differ across behaviors.Source: §I · pp. 1–2
  • One policy is trained across all eight tactile-mask configurations, allowing active representations to change at deployment without retraining.Representation-specific encoding and randomization preserve the components separately through control.Source: §I · p. 2
  • Sub-millimeter contact localization and 1.69 N real-world force-tracking MAE on unseen geometries are reported for the representation and best force-aware mask, respectively.Evaluation spans held-out response reconstruction, grasp-force regulation, tactile alignment, and adversarial peg insertion on a gripper with two 28-taxel pads.Source: §I · p. 2
  • The framework maps normal force and contact patch to a shared effective contact response recoverable from capacitive sensor readings.It preserves contact geometry, force distribution, and temporal contact change as separate representations through a Tactile Gated Policy.Source: §I · p. 2

A. Tactile Simulation and Sim-to-Real Transfer

The related work contrasts tactile simulation and transferable representations with this paper’s goal of preserving several tactile components separately. The interface then represents contact using normal force and contact patch, an abstraction motivated by the studied pads’ limited indentation.

A. Tactile Simulation and Sim-to-Real Transfer

  • Tactile simulation for vision-based sensors can render measurements as images, whereas taxel arrays require mapping simulated contact to distributed sensor responses.The paper positions its interface as a response mapping that avoids reconstructing deformation.Source: A. Tactile Simulation and Sim-to-Real… · p. 2
  • Unlike approaches that reconstruct a deformation field, the paper defines an effective contact response from normal force and contact patch that is recoverable from sensor readings and retains distributed information.Prior taxel-array approaches model local penetration and spread across taxels, while a TSF-85 digital twin simulates a deformable sensing field,.Source: A. Tactile Simulation and Sim-to-Real… · p. 2

B. Tactile Representations for Sim-to-Real Manipulation

  • The paper preserves geometry, force distribution, and temporal contact change separately, so their utility and sim-to-real behavior can be assessed independently and policy access changed without retraining.This differs from learning one transferable tactile embedding or compressing touch into a single representation.Source: B. Tactile Representations for… · p. 2

A. Shared Contact State

  • Contact is abstracted as total normal force and contact patch, measured from the simulator contact solver or controlled real-world contacts.The abstraction is described as appropriate for the relatively stiff pads studied; indentation is less than 1 mm at grasp forces up to approximately 60 N, and pressure is assumed uniform over the patch.Source: A. Shared Contact State · p. 2

B. Effective Contact Response

The effective contact response maps a spread pressure field to a per-taxel value, separating a shared contact response from taxel-specific sensor gain. The model uses shared parameters across both pads and is identified and evaluated within the range where the calibrated response-to-count mapping remains approximately linear.

B. Effective Contact Response

  • The response r_j is formed by spatially spreading pressure and nonlinearly aggregating it over each taxel region.Fig. 2 depicts force and contact patch producing the pressure field and the per-taxel response; the anisotropic Gaussian kernel models spreading, η controls pointwise nonlinearity, and γ controls overall force scaling.Source: B. Effective Contact Response · p. 2, Figure 2 · p. 3
  • The response is phenomenological, not an estimate of pad deformation, pressure, stress, or strain, and does not use simulator-specific penetration once force and patch are specified.Thus simulated and physical pads need not share the same deformation field.Source: B. Effective Contact Response · p. 3

C. Response Identification and Operating Range

  • Sensor counts follow c_j = β_j r_j + ε_j, and calibration estimates r_j by dividing counts by the taxel-specific gain β_j.A single set of response parameters is fit across both pads and all taxels, with gain estimated for each taxel; the selected values are σ = (1.5, 1.25) mm, η = 0.7, and γ = 0.5.Source: B. Effective Contact Response · p. 3, C. Response Identification and… · p. 3
  • The calibrated mapping remains approximately linear up to 20 N per taxel, compresses above that range, and is identified and evaluated within the approximately linear range.The same shared response model is used for both pads throughout the paper.Source: C. Response Identification and… · p. 3

D. Tactile Representations

The response is factorized into geometry, force distribution, and temporal contact change so each representation exposes different contact information to the policy.

a) Contact geometry (G)

  • G represents all 56 taxels as a point set, preserving pad-local contact location and each pad’s global pose.Each point combines normalized pad-local coordinates and pad identity with the response and pad pose features.Source: D. Tactile Representations · p. 3

b) Force distribution (F)

c) Temporal contact change (C)

  • C captures temporal contact change through per-taxel normalized contrast of successive responses and is invariant to static per-taxel gain in the ideal noise-free case.Responses below the noise floor and the first observation of each episode are set to zero.Source: D. Tactile Representations · p. 3

IV. TACTILE GATED POLICY

The Tactile Gated Policy preserves geometry, force, and temporal-change representations through separate encoding, then selects among them with a mask supplied to the recurrent actor.

IV. TACTILE GATED POLICY

  • Separate geometry, force, and temporal-change encoders produce latents that are selected by a tactile mask before entering the recurrent policy.Source: §IV · p. 3, Figure 3 · p. 4

A. Representation-Specific Encoders

  • Geometry uses a hierarchical point-set encoder, while force and change use pad-grid encoders with pad-specific adapters and a shared CNN architecture.Each representation is encoded as a 64-D latent; the force and change encoders use separate weights.Source: A. Representation-Specific Encoders · p. 3

B. Tactile Mask

  • The three-bit mask selects available latents, which are concatenated with encoded robot state and the mask before the LSTM actor.Source: B. Tactile Mask · p. 4, Figure 3 · p. 4
  • All eight mask configurations are sampled during training, so one policy per task supports changing active representations at deployment without retraining.The all-zero mask uses proprioception only.Source: B. Tactile Mask · p. 4

C. Representation-Specific Randomization

Representation-specific randomization perturbs tactile uncertainty along dimensions intended to preserve each representation’s information; the experiments assess how those representations affect policy performance separately and jointly.

C. Representation-Specific Randomization

  • Factorization enables tactile uncertainty to be randomized selectively along dimensions that preserve each representation’s intended information.Source: C. Representation-Specific Randomization · p. 4
  • For G, independently scaling normal force on each pad perturbs response magnitude while leaving spatial contact structure unchanged.The scale is sampled once per episode from U(0.7, 1.3).Source: C. Representation-Specific Randomization · p. 4
  • For F, independently rotating and translating each pad’s contact patch perturbs spatial distribution while preserving absolute normal force.The transformed patch is used to recompute the pressure field with the original force.Source: C. Representation-Specific Randomization · p. 4
  • C receives no episode-level tactile randomization because normalized temporal contrast is invariant to static per-taxel gain; sensor noise is applied upstream at every policy step.The count-space noise has σc = 4.2 ct and is mapped to response space through βj.Source: C. Representation-Specific Randomization · p. 4

V. EXPERIMENTS

  • The experiments ask whether the response bridges simulation and real measurements, preserves geometry, force, and temporal change, and affects sim-to-real policy performance separately and jointly.Source: §V · p. 4

A. Experimental Setup

The evaluation combines tactile measurements and simulation on a UR5e and capacitive gripper pads with held-out contact tests and response-model ablations. The shared response model generalizes to an unseen contact family, while validation and ablations test the pad-contact model and its assumptions.

A. Experimental Setup

  • Experiments use a UR5e, Robotiq 2F-85 gripper, and two TSF-85 capacitive pads, with localized indentation and grasp-based loading measured against force references.Each pad has a 7 × 4 array over a measured 25×40 mm active area.Source: A. Experimental Setup · p. 4
  • The policy-mask comparisons change only available tactile information, not policy weights, within a checkpoint.Source: A. Experimental Setup · p. 5

B. Effective Contact Response

  • The setup collects synchronized tactile and force measurements from localized pokes, edge grasps, and whole-pad grasps, with spatial holdout and unseen-family evaluation settings.Family holdout excludes all edge contacts from identification and model selection.Source: B. Effective Contact Response · p. 5
  • The compliant pad-contact twin achieves approximately 30-count RMSE and median response-level deviation below 3%, remaining within 10% RMSE across a 20× range of stiffness k.Validation covers 92 grasps on both pads with two flat punches and two spheres, over 1.5 N to 20 N per taxel.Source: B. Effective Contact Response · p. 5
  • 31.4/30.8 counts on the left/right pads versus 92.4/73.8 for the MLP and 114.8/85.7 for the CNN under family holdout, where all edge data are excluded from fitting and model selection.The reported advantage is generalization to unseen contact geometries, not minimizing interpolation error on training pokes.Source: B. Effective Contact Response · p. 5, Table I · p. 5
  • Removing spatial spreading or nonlinear response substantially increases edge reconstruction error, while a Hertz-like pressure profile provides no improvement over uniform pressure.Source: B. Effective Contact Response · p. 5

C. Information Preserved by the Factorization

The factorized response preserves recoverable spatial and force information in held-out physical contacts, while its temporal component is assessed under constant-force conditions.

Evaluation scope

  • The probes use grouped five-fold cross-validation and measure recoverable information, not statistical independence among geometry, force, and temporal components.Per-taxel calibration is fitted within each fold, while whole-pad data remain in training.Source: C. Information Preserved by the… · p. 5

Contact geometry

  • Held-out contacts are localized with 0.48 ± 0.20 mm matched-filter error and 0.54 ± 0.08 mm response-centroid error, both below taxel spacing.These are response-level tests; controller use of spatial information is evaluated separately.Source: C. Information Preserved by the… · p. 5

Force distribution

  • Force decoding achieves 1.7–1.9 N RMSE over 0.5–15 N, while the wider 0.5–80 N evaluation yields 3.03 ± 0.57 N RMSE.The wider-range result includes high-load compression beyond the declared operating regime and is supervised recovery, not zero-shot transfer.Source: C. Information Preserved by the… · p. 5

Temporal contact change

  • Temporal contact change is evaluated over 144 constant-force plateaus after static gain cancellation.The supplied passage does not report the observed contrast value.Source: C. Information Preserved by the… · p. 5

Information probes

  • Fig. 5 evaluates modeled versus measured count patterns, force error by true-force bin, and held-out contact localization finer than taxel pitch.Force error rises as the sensor approaches compression.Source: Figure 5 · p. 6

Temporal contact change

  • Under static contact, measured variation is 0.85–0.93 times that predicted by the additive-noise model across a 100× range of contact levels.The reported agreement indicates residual static-contact variation is well explained by sensor noise.Source: Figure 6 · p. 6

D. Factorized Tactile Control

The three sim-to-real tasks isolate spatial, force, and integrated tactile demands. Their results show that useful tactile representations depend on the interaction.

D. Factorized Tactile Control

  • The evaluation uses Tactile Alignment for spatial information, Grasp-Force Regulation for force information, and Adversarial Peg Insertion for integrated contact constraints.All tasks use the Tactile Gated Policy with task-specific actions and low-level robot control.Source: D. Factorized Tactile Control · p. 6

1) Tactile Alignment:

  • G and GC provide the strongest alignment, with GC reaching 1 mm clearance in 9/10 and 8/10 trials on r10 and unseen r18.F gives only coarse alignment, while C alone is comparable to proprioception; the taxel-line reference performs substantially worse.Source: 1) Tactile Alignment · p. 6, Table II · p. 6

2) Grasp-Force Regulation:

  • F-containing gates achieve below 2 N mean real-world force-regulation error, while removing F raises hardware error to 3.80–11.78 N.GFC reaches 2.28 ± 1.15 N, so using every representation is not optimal for this force-dominated task.Source: 2) Grasp-Force Regulation · p. 7, Table III · p. 7

3) Adversarial Peg Insertion:

Adversarial peg insertion combines uncertain grasp and fixture poses with changing contact constraints. GFC achieves the highest observed hardware success, although the unfactorized response leads in simulation.

3) Adversarial Peg Insertion:

  • The square peg must pass through a tilted 1 mm-clearance bore and two internal protrusions; grasp and fixture poses vary and are hidden from the actor.Hardware testing uses six fixture poses, and success requires full seating.Source: 3) Adversarial Peg Insertion · p. 7, Figure 7 · p. 7
  • GFC succeeds in 19/30 hardware trials (63.3%), versus 14/30 (46.7%) for Response r, while Response r leads simulation at 70.1%.The representation favored in simulation is not the one with the best physical transfer result.Source: 3) Adversarial Peg Insertion · p. 7, Table IV · p. 7
  • Removing any one representation from GFC reduces hardware success: GF reaches 12/30, while GC and FC each reach 14/30.C is weak alone at 8/30, but adding it does not reduce success in matched comparisons and improves three, consistent with complementary information.Source: 3) Adversarial Peg Insertion · p. 7, Table IV · p. 7
  • CoP reaches 64.3% simulation success but 6.7% on hardware, where simulated centroid shifts are absent and distributed GFC cues outperform compact summaries.Proprioception-only retains indirect contact cues through operational-space-control tracking deviations.Source: 3) Adversarial Peg Insertion · pp. 7–8, Table IV · p. 7
  • Across tasks, G-containing masks support alignment, F-containing masks support force regulation, and the complete representation achieves the highest insertion success.Full tactile input is therefore not uniformly optimal across tasks.Source: 3) Adversarial Peg Insertion · p. 8

VI. DISCUSSION

The discussion frames the effective response as an operational interface rather than a physical-state estimate, with a bounded validated range and replaceable forward model.

VI. DISCUSSION

  • The effective response is phenomenological, not a measurement of pad deformation, pressure, stress, strain, or mechanical state.Its justification is operational: reconstructing held-out physical measurements while preserving usable information.Source: §VI · p. 8
  • Recoverability and usable information are certified only below approximately 20 N per taxel.This is the stated operating boundary for the response's validation.Source: §VI · p. 8
  • Because downstream components consume r, a more faithful forward model could replace the compact response without changing them.The response's net-force and contact-patch basis is compatible with GPU-parallel policy training.Source: §VI · p. 8
Loading 2610.10510v1…