PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibration-Free Surface Normals Estimation in Vision-Based Tactile Sensing using Universal Photometric Stereo

Zdravko Dugonjic, Stefanie Speidel, Roberto Calandra

arXiv:2609.31754v1cs.CVcs.LGcs.RO

TL;DR

Sensor-specific calibration with known probes does not scale readily across vision-based tactile sensors. This paper uses dynamic illumination and pretrained Universal Photometric Stereo networks, reporting competitive calibrated-baseline results across three sensors.

  • Problem

    Obtaining high-detail contact normals and depth relies on probe-based sensor calibration, which does not scale across sensor designs and can be complex.

    Source: §I · p. 1

  • Method

    The framework captures a fixed tactile contact under dynamic illumination and uses pretrained Universal Photometric Stereo networks to estimate surface normals without tactile-specific training or probe calibration.

    Source: §I · p. 2

  • Results

    Across three sensors, universal methods rival calibrated baselines: sphere MAE 6.56°, in-the-wild discrepancy 10.66°, and Digit 360 MAE 10.2°.

    Source: §VII · p. 6, D. In-the-Wild Contact Surface Normals… · p. 6, E. Dome-Shaped Sensor Digit 360 · p. 6

  • Takeaways & Limitations

    The results support a unified representation of contact surfaces across tactile sensor designs as a foundation for transferable tactile perception.

    Source: §VII · p. 6

    Inference assumes the contact-region mask is provided, and the study does not determine the optimal number of illumination settings or images.

    Source: §VI · p. 6

Abstract

from arXiv · show

Vision-based tactile sensors are a popular solution for capturing rich contact surface geometry. However, to obtain high-detail contact surface normals and depth, it is necessary to calibrate the sensor by physically pressing a probe with known geometry against the sensor elastomer and mapping tactile images onto the ground truth probe's shape. This approach does not scale across different tactile sensors, and the calibration effort can be complex depending on the sensor shape and optical system. Instead, we propose a calibration-free procedure for the estimation of contact surface normals using Universal Photometric Stereo neural networks. In a series of real-world experiments, we evaluate our approach on 3 sensors with different optical systems, demonstrating that universal methods are a suitable approach for estimating surface normals at the contact patch from tactile images, thereby alleviating the need for tactile sensor calibration. Controlled experiments with a metal ball show that universal methods match the calibrated method, with a mean angular error of $6.56^{\Large\circ}$. We show that the proposed framework recovers high-frequency surface details of objects with natural textures, achieving an overall mean angular error of $10.66^{\Large\circ}$. Universal method robustly recovers the contact surface normals captured with dome-shaped Digit 360, achieving a low angular discrepancy of $10.18^{\Large\circ}$ relative to the calibrated baseline. This experiment demonstrates that with sufficient illumination settings surface normals could be estimated using a model trained solely on synthetic data. By providing a unified representation of contact surfaces across different vision-based tactile sensor designs, Universal Photometric Stereo neural networks lay the foundation for transferable tactile perception across sensors.

I. INTRODUCTION

Vision-based tactile sensors capture rich contact information, but sensor-specific image differences make interpretation difficult and calibration burdensome. The paper proposes zero-shot surface-normal estimation using dynamic illumination and pretrained Universal Photometric Stereo networks.

I. INTRODUCTION

  • Vision-based tactile sensors capture contact geometry at high spatial resolution, but interpreting their RGB images is challenging.Their coating and lighting designs vary across implementations.Source: §I · p. 1
  • Differences in coatings, lighting, and sensor form factors can make models trained on one sensor fail on another, prompting retraining and recalibration.Retraining requires additional physical data collection.Source: §I · pp. 1–2
  • The proposed framework uses dynamic illumination and pretrained Universal Photometric Stereo networks to estimate surface normals without tactile-specific training or probe calibration.The framework assumes dynamic illumination and reframes physical sensor calibration as a computational problem.Source: §I · p. 2
  • Experiments report competitive performance against standard calibration and generalization across tactile sensors with different optical systems.Source: §I · p. 2

A. Vision-Based Tactile Sensors for 3D Shape Reconstruction

Prior tactile reconstruction methods estimate depth directly or infer surface normals that can be integrated into depth maps. This paper instead targets local contact-patch normals, using dynamic lighting and pretrained Universal Photometric Stereo networks across tactile sensors.

A. Vision-Based Tactile Sensors for 3D Shape Reconstruction

  • Earlier tactile methods map images to sphere-reference normals or train networks on probe imprints, while other methods regress depth directly.Normal estimates can be integrated into high-quality depth maps; direct depth prediction requires diverse probes and contact configurations.Source: A. Vision-Based Tactile Sensors for 3D… · p. 2
  • The framework estimates local contact-patch normals, which are a basis for global shape estimation through touch.It uses dynamic lighting and pretrained universal models without tactile probes.Source: A. Vision-Based Tactile Sensors for 3D… · p. 2

B. Photometric Stereo

  • Photometric stereo estimates surface normals from images captured from a fixed camera perspective under different illumination conditions.The original calibrated setting assumes a Lambertian surface and known directional lighting; universal photometric stereo removes prior assumptions about physical lighting models.Source: B. Photometric Stereo · p. 2
  • Dynamic illumination lets pretrained Universal Photometric Stereo networks estimate contact-patch normals across three tactile sensors without expensive equipment or laborious probing.Source: B. Photometric Stereo · p. 2

III. DEPTH ESTIMATION FOR VBTS

Depth estimation from vision-based tactile images can be framed as direct depth regression or as indirect estimation through surface normals. The proposed pipeline captures a fixed contact under varied lighting, masks the contact region, and infers its normals.

III. DEPTH ESTIMATION FOR VBTS

  • Depth estimation from tactile images can be formalized as direct depth-map inference or indirect estimation through surface normals.Source: §III · p. 2
  • The pipeline captures a fixed contact under varied lighting, masks the contact region, and feeds multiple tactile images to a Universal Photometric Stereo Transformer for normal inference.Source: Figure 1 · p. 3

A. Direct Depth Estimation

  • Direct depth estimation regresses a depth map from an RGB tactile image, with ground truth obtained from known-geometry probes or simulation.The cited Tactile Transformer is trained on tactile images paired with depth maps.Source: A. Direct Depth Estimation · p. 3

B. Depth Estimation Through Surface Normals

For flat vision-based tactile sensors, small elastomer deformation supports a weak-perspective camera approximation. Surface normals yield gradients for depth integration, while calibration commonly learns gradients from a known object; Lambertian estimation is another, less practical option.

B. Depth Estimation Through Surface Normals

A. Overview

The framework treats a fixed tactile contact as multiple images captured under dynamic illumination. It uses Universal Photometric Stereo Transformers to infer contact-patch normals without sensor calibration or tactile-image pretraining.

A. Overview

  • The framework represents a fixed contact as multiple tactile images captured under dynamic illumination, rather than one image under fixed lighting.Source: A. Overview · p. 4
  • Pre-trained Universal Photometric Stereo Transformer networks infer contact-patch surface normals without further tactile sensor calibration.Source: A. Overview · p. 4
  • The approach does not require tactile images during pre-training, so the physical image collection typically required in current practice is no longer necessary.Source: A. Overview · p. 4

B. Data Collection and Processing

The experiments use pre-trained Universal Photometric Stereo models on tactile images collected from sensors with different illumination designs. On the metal-sphere task, GeoUniPS performs on par with calibrated methods across Digit and GelSight Mini.

B. Data Collection and Processing

  • Data collection fixes contact and captures images under available illumination settings, adapting to each sensor's lighting interface.Digit and Digit 360 use sequential LED captures; GelSight Mini's RGB image is decomposed into three grayscale images, and contact regions are manually masked.Source: B. Data Collection and Processing · p. 4

V. EXPERIMENTAL RESULTS

  • The experiments compare universal estimates with a conventional calibrated baseline on a metal sphere, then assess generalization in the wild using mean angular error.Source: §V · p. 4

A. Flat Sensor Calibrated Baseline

  • The flat-sensor baseline trains a four-layer MLP to regress surface gradients from tactile-image pixel values and locations using a 6 mm metal sphere.The sphere provides ground-truth normals from its known diameter.Source: A. Flat Sensor Calibrated Baseline · p. 4
  • GeoUniPS performs on par with traditional calibrated methods for sphere-normal estimation on Digit and GelSight Mini.The sensors have different optical designs.Source: Figure 2 · p. 5, Figure 3 · p. 5

B. Digit 360 Calibrated Baseline

Digit 360's dome-shaped sensor uses a simplified Lambertian calibration based on its LED positions. In sphere experiments, GeoUniPS matches the calibrated approach on Digit and GelSight Mini, while limited lighting may contribute to higher errors in other universal methods.

B. Digit 360 Calibrated Baseline

  • For dome-shaped Digit 360, the simplified baseline extracts eight LED positions from CAD files and solves the Lambertian reflectance equations across all eight light images.This avoids an external calibration reference object and precise alignment machinery.Source: B. Digit 360 Calibrated Baseline · p. 5

C. Sphere Surface Normals Estimation

  • GeoUniPS reaches the same performance as the calibrated method on sphere-normal estimation across Digit and GelSight Mini.All universal methods recover the global sphere shape despite training only on synthetic scenes; higher angular errors may partly reflect fewer light directions than those used in training scenarios.Source: C. Sphere Surface Normals Estimation · p. 5

D. In-the-Wild Contact Surface Normals Estimation

On natural-texture objects, GeoUniPS closely matches the calibrated baseline across flat tactile sensors; on dome-shaped Digit 360, UniMSPS achieves lower MAE than GeoUniPS. Synthetic-only UniMSPS also recovers fine surface detail on Digit 360.

D. In-the-Wild Contact Surface Normals Estimation

E. Dome-Shaped Sensor Digit 360

  • For Digit 360 inference, the authors remove the elastomer, calibrate the camera to correct fisheye distortion, and evaluate MAE only within the contact region.They mask a centered circular region with an empirically selected radius.Source: E. Dome-Shaped Sensor Digit 360 · p. 6
  • 10.2° mean MAE for UniMSPS outperforms GeoUniPS at 11.37° mean MAE on dome-shaped Digit 360.The authors attribute the advantage to UniMSPS being trained for more diverse illumination settings.Source: E. Dome-Shaped Sensor Digit 360 · p. 6
  • UniMSPS, trained only on synthetic Blender scenes, recovers intricate high-frequency surface details rivaling GeoUniPS on Digit 360.GeoUniPS incorporates substantially more training data and a geometric prior.Source: E. Dome-Shaped Sensor Digit 360 · p. 6

VI. LIMITATIONS

The conclusion reports competitive surface-normal estimates across a metal sphere, natural-texture objects, and Digit 360. The study assumes the contact-region mask is supplied at inference, does not optimize illumination selection, and reports inference costs that preclude real-time integration.

VI. LIMITATIONS

  • Inference assumes the contact-region mask is provided, limiting the study to surface-normal estimation within a known contact patch.Source: §VI · p. 6
  • The study does not optimize the number of illumination settings or tactile images; some settings were selected through trial and error.GelSight Mini used all three available tactile images.Source: §VI · p. 6
  • Transformer-based universal methods have slower inference and higher compute consumption than calibrated methods, making real-time robotic integration infeasible in this study.The authors suggest model distillation as a possible future remedy.Source: §VI · p. 6

VII. CONCLUSION

  • 6.56° MAE on a metal sphere, 10.66° average discrepancy on natural-texture objects, and 10.2° MAE on Digit 360 are reported for universal methods.Source: §VII · p. 6

Related papers

Loading 2609.31754v1…