PaperlayerView exact revision on arXiv

Source-linked AI summary

TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering

Yitao Zhang, Hong Ying, Haoran Guo, Xiaoying Zhou, Guanyu Chen, Chenxi Xiao

arXiv:2610.01943v1cs.RO

TL;DR

Conventional object assets often omit microscale tactile structure and transient temperature dynamics needed for tactile and thermal interaction. TouchTherm reconstructs registered tactile micro-height fields and dynamic thermal fields from real-object observations; experiments report tactile, thermal, recognition, and VR feedback results.

  • Problem

    Existing visual assets primarily represent object-scale geometry and appearance, while contact-scale tactile relief and object-level dynamic thermal fields are missing.

    Source: §I · p. 1

  • Method

    TouchTherm combines a coarse collision mesh and registered tangent-space tactile micro-height fields with a physics-regularized dynamic thermal field reconstructed from infrared cooling observations.

    Source: §I · p. 2, 3) Contact-conditioned Tactile Rendering · p. 3, D. Dynamic Thermal Field Reconstruction · p. 4

  • Results

    0.465 °C and 0.592 °C held-out surface-temperature MAEs at 30 s and 45 s; tactile rendering improves over coarse-geometry and image-space baselines, and recognition outperforms coarse geometry across Top-1, Top-3, and Macro-F1.

    Source: B. Tactile Appearance Evaluation and… · p. 5, §V · p. 8

  • Takeaways & Limitations

    The reconstructed assets support synthetic-to-real tactile object recognition, while a wearable-glove VR application demonstrates spatially and temporally varying thermal feedback.

    Source: §V · p. 8

    The framework uses microgeometry only for tactile rendering and models natural cooling, not contact-dependent bidirectional heat transfer.

    Source: §V · p. 8

Abstract

from arXiv · show

Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction. Existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects. For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the thermal fields achieve held-out surface-temperature MAEs of 0.465 degrees C and 0.592 degrees C at 30 s and 45 s, respectively. The resulting tactile assets support synthetic-to-real object recognition from tactile observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for multimodal sensory simulation and temperature-aware virtual interaction.

I. INTRODUCTION

TouchTherm addresses the lack of contact-scale tactile relief and object-level dynamic thermal representations in conventional visual assets. It combines visual geometry, tactile micro-height fields, and dynamic thermal fields in simulation-ready object assets, which are evaluated through rendering, recognition, and VR feedback.

I. INTRODUCTION

  • Conventional assets often omit contact-scale tactile relief and spatially varying temperature dynamics that evolve over time.Dense polygonal microgeometry can increase storage and simulation costs, motivating separate representations for collision geometry and surface relief.Source: §I · p. 1
  • TouchTherm constructs simulation-ready visuo-tactile-thermal assets from real objects, combining registered visual geometry, tactile micro-height fields, and dynamic thermal fields.The coarse collision mesh is augmented with tangent-space micro-height fields for tactile rendering and a thermal field that supports time-varying surface-temperature queries.Source: §I · pp. 1–2
  • Its tactile pipeline renders optical tactile observations from contact-conditioned micro-height fields.The framework preserves contact-scale relief separately from object-scale collision geometry.Source: §I · p. 2
  • The thermal pipeline reconstructs dynamic fields from multiview infrared cooling videos and identifies effective parameters for time-varying surface-temperature queries.Source: §I · p. 2
  • Validation on 20 objects covers tactile appearance, thermal prediction, synthetic-to-real recognition, and wearable thermal feedback in VR.Source: §I · p. 2

II. RELATED WORK

Prior work provides object-scale visual assets and complementary tactile or thermal measurements, but these sources generally lack a unified object-level representation registered to geometry.

II. RELATED WORK

  • Object reconstruction methods mainly capture object-scale geometry and appearance, overlooking contact-scale relief and transient thermal properties.Source: A. Multimodal Data Collection · p. 2
  • Tactile and thermal sensors provide local surface and temperature information, but these measurements are typically localized observations rather than object-level representations registered to geometry.Source: A. Multimodal Data Collection · p. 2
  • The paper constructs a shared object-frame representation that spatially registers visual, tactile, and thermal information as reusable simulation assets.Source: A. Multimodal Data Collection · p. 2

B. Datasets and Simulation Assets

Existing datasets and simulation assets expand access to digital objects and multisensory observations, yet lack registered tactile microgeometry and dynamic thermal responses. TouchTherm aligns scans, normal maps, and infrared video, then builds a unified asset from tactile and thermal reconstruction branches.

B. Datasets and Simulation Assets

  • Available object datasets provide diverse geometry and visual assets but generally lack spatially registered contact-scale tactile relief and dynamic surface-temperature information.Multisensory datasets also lack the combination of registered tactile microgeometry and spatiotemporal thermal responses in one object asset.Source: B. Datasets and Simulation Assets · p. 2

A. Pipeline Overview

  • TouchTherm aligns object scans, multiview normal maps, and infrared videos to a common object frame before reconstructing tactile microgeometry and a dynamic thermal field in parallel.The resulting representations are integrated into a unified object asset.Source: A. Pipeline Overview · p. 2

B. Hardware Setup

  • Acquisition combines structured-light scanning and smartphone images for geometry and normals, thermal cameras for natural cooling after controlled heating, and GelSight measurements for validation and calibration.Source: B. Hardware Setup · p. 2

C. Reconstruction of Tactile Microgeometry for Rendering

TouchTherm separates collision geometry from tactile microgeometry, using a coarse mesh for physical collision and registered local micro-height fields for high-frequency surface relief. It estimates those fields from multiview normal maps and renders them in contact-conditioned patches alongside coarse-mesh indentation.

1) Coarse Geometry and Microgeometry Decoupling:

2) Normal Estimation and Registration:

  • SDM-UniPS estimates image-aligned normal maps, which are registered to the mesh using manually annotated sparse 2D–3D correspondences and PnP-estimated object-to-camera poses.Object-space points are projected into each view using camera intrinsics and perspective projection.Source: 2) Normal Estimation and Registration · p. 3

3) Contact-conditioned Tactile Rendering:

  • At each contact point, closest-surface projection samples a metric patch, and its normal-map samples are transformed into the local tangent frame.Patch dimensions W and H set the physical sensing area.Source: 3) Contact-conditioned Tactile Rendering · p. 3
  • Poisson reconstruction recovers a least-squares height field from estimated normal-field gradients, and removing its low-frequency component yields the micro-height residual.The least-squares solver is a Poisson/DCT solver.Source: 3) Contact-conditioned Tactile Rendering · p. 3
  • Rendering combines coarse-mesh indentation under the contact mask with the calibrated micro-height residual, scaled by a per-object factor.The factor is selected by aligning rendered output with a GelSight observation from a separate calibration contact.Source: 3) Contact-conditioned Tactile Rendering · p. 4

D. Dynamic Thermal Field Reconstruction

TouchTherm fuses multiview thermal observations, models surface heat diffusion and ambient exchange, and jointly reconstructs temperatures and estimates the governing parameters. The learned field is then replaced by a direct thermal rollout.

D. Dynamic Thermal Field Reconstruction

  • Thermal observations are calibrated, corrected for temporal drift and viewing-angle bias, projected onto the object, and averaged across cameras; points unseen by every camera remain unobserved.Camera poses are estimated with RANSAC-PnP and refined jointly across cameras.Source: D. Dynamic Thermal Field Reconstruction · p. 4
  • Surface diffusion compares each point’s temperature with a distance-weighted neighborhood, with normalized weights reducing the influence of uneven point density.The operator approximates the negative surface Laplacian; its diffusion term cools a point hotter than its neighborhood average and warms one that is colder.Source: D. Dynamic Thermal Field Reconstruction · p. 4
  • The temperature network uses the K lowest-frequency eigenvectors of the surface operator as spatial features that encode location through surface connectivity.Source: D. Dynamic Thermal Field Reconstruction · p. 4
  • The temperature MLP predicts surface temperatures from spatial features and time, while joint optimization fits observations and penalizes violations of the thermal dynamics.The optimization estimates the network parameters together with surface-diffusion and ambient-relaxation coefficients.Source: D. Dynamic Thermal Field Reconstruction · p. 4
  • After estimating the diffusion and ambient-relaxation coefficients, runtime simulation discards the network and advances temperatures with explicit Euler updates accounting for both processes.The rollout retains the surface operator and estimated coefficients.Source: D. Dynamic Thermal Field Reconstruction · p. 4

IV. EXPERIMENTS

The experiments evaluate TouchTherm through asset quality, rendering fidelity, prediction accuracy, and downstream perception and interaction tasks.

IV. EXPERIMENTS

  • The evaluation includes object asset quality.Source: §IV · p. 4
  • The evaluation measures tactile rendering fidelity and thermal prediction accuracy.Source: §IV · p. 4
  • The evaluation also considers downstream utility in perception and interaction tasks.Source: §IV · p. 4

A. TouchTherm Assets Overview

TouchTherm constructs visuo-tactile assets for 20 varied real objects from metric meshes, registered normal maps, and reconstructed micro-height fields. Acquisition uses object-dependent numbers of views and takes roughly half a day per object per modality.

A. TouchTherm Assets Overview

  • Each visuo-tactile asset combines a cleaned metric mesh, registered multiview normal maps, and reconstructed micro-height fields for simulated tactile contact and rendering.Source: A. TouchTherm Assets Overview · p. 4
  • The 20-object collection spans flat, curved, cylindrical, and free-form geometries, with smooth, granular, embossed, and anisotropic surface patterns.Source: A. TouchTherm Assets Overview · p. 4, Figure 3 · p. 5
  • Planar objects use one registered RGB/normal view, while 3D objects use at least two, typically 5–7 and up to 12 for complex shapes; acquisition takes roughly half a day per object per modality.Thermal videos record natural cooling after heat-gun excitation.Source: A. TouchTherm Assets Overview · p. 5

B. Tactile Appearance Evaluation and Ablation

Across 20 objects, TouchTherm’s tactile reconstruction scores are compared with repeated real contacts and with coarse-geometry and image-space ablations. The evaluation pairs real and simulated tactile measurements and uses bounded alignment to account for residual image-plane mismatch.

Tactile appearance evaluation

  • The paired evaluation compares real and simulated tactile measurements after bounded alignment to compensate for residual image-plane mismatch.Physical contacts are manually established; alignment is limited to ±4° rotation and ±16 pixels translation.Source: B. Tactile Appearance Evaluation and… · p. 5, Figure 4 · p. 5
  • Real–Sim comparisons yield G-SSIM/HF-NCC of 0.0712 ± 0.0217/0.0775 ± 0.0575, below the Real–Real reference of 0.0954±0.0471/0.1622±0.0917.The reference averages pairwise similarity across three repeated physical GelSight contacts per object.Source: B. Tactile Appearance Evaluation and… · p. 5

Ablation

  • TouchTherm achieves the highest object-balanced G-SSIM and HF-NCC across 20 objects, scoring 0.0701/0.0911 versus 0.0480/0.0116 for Coarse Geometry and 0.0629/0.0081 for Image-space Height.Table I averages three contacts within each object and weights all objects equally.Source: Table I · p. 5, B. Tactile Appearance Evaluation and… · p. 5

C. Dynamic Thermal Evaluation

The thermal evaluation compares reconstructed and reference fields over time and tests the full model against parameter ablations on 20 objects at two prediction horizons.

Dynamic thermal evaluation

  • The evaluation compares the full α + h model with no-α and no-h variants on 20 objects at 30 s and 45 s prediction horizons.It reports mean surface-temperature change, surface MAE, and Gradient RMSE; Hotspot F1 evaluates spatial agreement of high-temperature regions.Source: C. Dynamic Thermal Evaluation · p. 5
  • Removing h largely suppresses cooling and substantially increases surface MAE, while removing α causes only minor degradation.The reported interpretation is that ambient heat exchange primarily governs global cooling, with surface diffusion providing smaller improvements in local temperature prediction.Source: C. Dynamic Thermal Evaluation · pp. 5–6
  • Figure 5 compares real GelSight Mini measurements with simulated renderings across 20 objects under approximately matched contact regions and sensor configurations, with similarity metrics computed after bounded 2D alignment.Source: Figure 5 · p. 6

D. Downstream Applications

The downstream demonstrations test whether TouchTherm’s tactile representation transfers to real-image recognition and whether reconstructed thermal cues can be explored through wearable VR feedback.

Synthetic-to-real tactile object recognition

  • Recognition uses simulated GelSight images for training and validation, with real images held out for evaluation and no real images used for classifier training or validation.The setup uses 90 simulated contacts per object for training and 10 for validation; real evaluation uses held-out images.Source: D. Downstream Applications · pp. 6–7
  • TouchTherm improves synthetic-to-real recognition over Coarse Geometry across Top-1, Top-3, and Macro-F1, achieving 34.0 ± 2.7%, 63.0 ± 1.7%, and 29.3 ± 2.5%, respectively.Coarse Geometry scores 20.0±4.4%, 35.0±2.3%, and 13.4 ± 3.7% on those metrics.Source: D. Downstream Applications · p. 7

Thermal feedback in VR

  • Ten participants recognized gradual cold–warm–hot changes across object regions, and all reported no discomfort with the wearable thermal-feedback system.Visual–thermal consistency averaged 6.30 ± 0.95 and comfort 6.40 ± 1.26 on a seven-point scale.Source: D. Downstream Applications · p. 7

V. CONCLUSION

TouchTherm combines coarse visual geometry, tactile microgeometry, and a dynamic thermal field in simulation-ready multimodal object assets. Its evaluations report tactile-recognition gains and held-out thermal prediction accuracy, while identifying limits in contact mechanics and heat transfer.

V. CONCLUSION

  • The framework integrates coarse visual geometry, tactile microgeometry, and a dynamic thermal field, separating tactile micro-height fields from collision geometry to preserve contact-scale detail without increasing collision complexity.The reconstructed thermal field is derived from multiview infrared observations.Source: §V · p. 7
  • On 20 objects, TouchTherm improves tactile rendering over coarse-geometry and image-space baselines and achieves held-out surface-temperature MAEs of 0.465 °C at 30 s and 0.592 °C at 45 s.TouchTherm also consistently outperforms coarse geometry across Top-1, Top-3, and Macro-F1 in synthetic-to-real tactile object recognition.Source: §V · p. 8
  • The current framework uses microgeometry only for tactile rendering and models natural cooling, not bidirectional heat transfer during human or robotic contact.Future work is proposed on compliant contact and friction models and contact-dependent bidirectional heat transfer.Source: §V · p. 8
Loading 2610.01943v1…