PaperlayerView exact revision on arXiv

Source-linked AI summary

Tactile-Augmented Radiance Fields

Yiming Dou, Fengyu Yang, Yi Liu, Antonio Loquercio, Andrew Owens

arXiv:2405.04534v1cs.CV

TL;DR

Tactile data collection is costly, and existing real-scene datasets lack spatially aligned visual and tactile signals. TaRF registers touch to a NeRF scene and uses conditional diffusion to estimate touch from rendered RGB-D inputs. The experiments report accurate cross-modal prediction and utility for downstream tasks.

  • Problem

    Collecting touch is expensive, and existing real-scene datasets lack aligned visual-tactile information and provide sparse probes with unclear 3D relationships.

    Source: §1 · p. 1

  • Method

    TaRF registers touch probes to a NeRF scene and trains conditional diffusion to generate touch from rendered visual and depth signals at novel positions.

    Source: §1 · p. 2

  • Results

    Experiments report accurate touch prediction from sight for natural scenes, improved prediction with visually acquired 3D geometry, and benefits from synthetic novel-view probes on tactile localization.

    Source: §1 · p. 2, §5.3 · p. 8

  • Takeaways & Limitations

    The TaRF representation and dataset provide spatially aligned visual-tactile data for 3D touch localization and material recognition.

    Source: §6 · p. 8

    Small centimeter-scale SfM or registration errors can cause multi-pixel misalignment, and TaRF assumes coarse scene structure remains unchanged during touch.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present a scene representation, which we call a tactile-augmented radiance field (TaRF), that brings vision and touch into a shared 3D space. This representation can be used to estimate the visual and tactile signals for a given 3D position within a scene. We capture a scene's TaRF from a collection of photos and sparsely sampled touch probes. Our approach makes use of two insights: (i) common vision-based touch sensors are built on ordinary cameras and thus can be registered to images using methods from multi-view geometry, and (ii) visually and structurally similar regions of a scene share the same tactile features. We use these insights to register touch signals to a captured visual scene, and to train a conditional diffusion model that, provided with an RGB-D image rendered from a neural radiance field, generates its corresponding tactile signal. To evaluate our approach, we collect a dataset of TaRFs. This dataset contains more touch samples than previous real-world datasets, and it provides spatially aligned visual signals for each captured touch signal. We demonstrate the accuracy of our cross-modal generative model and the utility of the captured visual-tactile data on several downstream tasks. Project page: https://dou-yiming.github.io/TaRF

1. Introduction

Tactile sensing captures material properties and microgeometry relevant to physical-contact tasks, but collecting touch is costly and existing scene datasets lack spatially aligned visual-tactile signals. TaRF registers touch to a visual scene and uses conditional diffusion to estimate touch at novel positions, with experiments reporting accurate prediction and downstream utility.

1. Introduction

  • Tactile sensing reveals material properties and microgeometry relevant to applications including robotic locomotion, manipulation, and material simulation.Source: §1 · p. 1
  • Collecting tactile data is expensive, while existing real-scene datasets lack aligned visual signals and contain sparse touch samples whose 3D relationships are unclear.Touch sensors and their holders often occlude large portions of the visual scene.Source: §1 · pp. 1–2
  • TaRF uses a low-cost camera–touch-sensor setup and multi-view geometry to register touch with a visual scene reconstructed by NeRF.The sensors can be calibrated through correspondences between sight and touch.Source: §1 · p. 2
  • A conditional diffusion model estimates touch at novel scene positions, propagating sparse probes to visually and structurally similar regions.It is conditioned on spatially aligned visual and depth information rendered from the NeRF.Source: §1 · p. 2
  • NeRF enables unobstructed visual images spatially aligned with touch signals to be sampled from arbitrary 3D viewpoints.This contrasts with prior visual-tactile data capture described as obstructed in the introduction.Source: Figure 2 · p. 2, §1 · p. 2
  • Experiments report accurate touch prediction from sight for natural scenes, benefits from visually acquired 3D geometry, and improved tactile-localization performance when synthetic probes augment real pairs.The introduction also reports that novel-view touch estimates are qualitatively accurate and useful on downstream tasks.Source: §1 · p. 2

2. Related Work

Prior visual-tactile datasets and models differ in their scale, alignment, and use of scene geometry. TaRF combines scene-level aligned data with geometry-conditioned generative touch prediction.

2. Related Work

  • Unlike Zhong et al., TaRF uses common materials rather than expensive robot proprioception to align touch and vision, enabling larger, more diverse scene-level data and a higher-capacity diffusion model.Zhong et al. generate tactile fields for several small objects using a NeRF.Source: §2 · p. 2
  • Unlike prior scene-level datasets [5, 56] and image- or video-conditioned touch predictors [18] [35] [57], TaRF spatially aligns signals in a 3D representation and uses scene-specific geometry.Touch is tied to a precise scene position, unlike language supervision; related work also studies joint vision-touch embeddings [28] [36] [56] [58] [61].Source: §2 · p. 3

3. Method

TaRF combines a NeRF scene reconstruction with a tactile model in shared 3D space. The method registers touch measurements to the scene, then uses conditional diffusion to estimate touch at novel locations.

3. Method

  • TaRF associates a NeRF visual representation with a model that generates the tactile signal for a touch at the image center, placing both in shared 3D space.The NeRF maps 3D points and viewing directions to color and density.Source: §3 · p. 3

3.1. Capturing vision and touch signals

  • The capture process builds a NeRF from scene video and uses a camera mounted with a vision-based touch sensor to collect synchronized signals and estimate their relative pose.Calibration uses sight-touch correspondences and NeRF-derived 3D structure to solve a six-degree-of-freedom resectioning problem by minimizing reprojection error; the camera intrinsics are known, and 6–15 annotated correspondences are used.Source: §3.1 · pp. 3–4

3.2. Imputing the missing touch

  • A conditional diffusion model predicts tactile images from NeRF-rendered views and depth maps, with a sensor background image as an additional input that empirically improves performance.Training first pre-trains a cross-modal encoder with self-supervised contrastive learning, then trains a conditional latent diffusion model; at inference, rendered visual signals and depth are used to estimate touch at a novel location.Source: §3.2 · p. 4, Figure 4 · p. 4

4. A 3D Visual-Tactile Dataset

This section describes a two-stage pipeline for collecting synchronized visual-touch data and reports a scene-scale dataset of spatially aligned vision-touch pairs.

4.1. Data Collection Procedure

  • The collection procedure captures scene views before synchronized visual-touch probes, then estimates camera poses and derives touch poses using the mount calibration matrix.Approximately 500 frames are collected around intended touch areas.Source: §4.1 · p. 4
  • Translating sensor poses upward by 0.4 meters and querying the NeRF at a 50° field of view yields approximately 1,500 temporally aligned image pairs per scene.The authors describe this procedure as scalable because it requires neither specific expertise nor equipment.Source: §4.1 · p. 4

4.2. Dataset Statistics

  • The dataset covers 13 ordinary scenes, typically with 1k to 2k tactile probes per scene, for a total of 19.3k image pairs.The scenes include indoor settings and four outdoor scenes.Source: §4.2 · p. 4
  • The dataset includes varied geometries and textures, and is presented as the first to capture full, scene-scale spatially aligned vision-touch image pairs.Collectors probed distinct object areas using different sensor poses.Source: §4.2 · p. 5, Figure 5 · p. 5

5. Experiments

The experiments assess dense touch estimation, then test aligned real and synthesized touch data through tactile localization and material classification.

5.1. Implementation Details

  • Nerfacto NeRFs use approximately 2,000 images and 200,000 training steps; the diffusion model conditions on RGB-D, with multiscale conditioning and contrastive reranking.Source: §5.1 · p. 5

5.2. Dense Touch Estimation

  • Up to 4x lower FID and 80x higher CVTP; methods are comparable on PSNR and SSIM, where L1 slightly leads.Source: Table 2 · p. 6, §5.2 · p. 6

5.3. Downstream Task I: Tactile Localization

  • Training on aligned real pairs plus estimated samples from new scene locations achieves the highest mAP across all distance thresholds.Source: Table 4 · p. 8, §5.3 · p. 8

5.4. Downstream Task II: Material Classification

  • Adding estimated touch probes improves all three material tasks, especially smoothness classification; TaRF data also improves contrastive pretraining despite distribution differences.Source: §5.4 · p. 8, Table 5 · p. 8

6. Conclusion

The conclusion presents TaRF as a shared 3D representation for vision and touch that generates touch probes at novel scene locations. It reports a large spatially aligned dataset, evaluates touch localization and material recognition, and identifies registration and scene-structure assumptions as limitations.

  • TaRF brings vision and touch into a shared 3D space and generates touch probes for novel scene locations.Source: §6 · p. 8
  • The work collects the largest dataset of spatially aligned vision and touch probes and evaluates it on 3D touch localization and material recognition.Source: §6 · p. 8
  • The conclusion frames TaRF as a first step toward scene representations that capture how things look and feel, with possible uses in robotics and virtual worlds.Source: §6 · p. 8
  • Centimeter-scale SfM or visual-tactile registration errors can cause pixel misalignment, and the representation assumes touch does not change coarse scene structure, which may fail for some inelastic surfaces.Source: §6 · p. 8
Loading 2405.04534v1…