PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Visual Scenes from Touch

Fengyu Yang, Jiacheng Zhang, Andrew Owens

arXiv:2309.15117v1cs.CV

TL;DR

Prior visuo-tactile methods address limited tasks, and cross-modal synthesis lags behind other domains. This paper uses a visuo-tactile diffusion model across synthesis tasks, generating images from touch without scene-image conditioning and outperforming prior stylization work while addressing additional tasks.

  • Problem

    Existing visuo-tactile methods cover narrow tasks and lag behind cross-modal synthesis in other domains.

    Source: §1 · p. 1

  • Method

    A visuo-tactile diffusion model uses temporal multimodal representations, masks hand pixels during training, and conditions shading estimation on reflectance and touch.

    Source: §3.1.1 · p. 3, §3.2.1 · p. 4, §3.2.3 · p. 4

  • Results

    The model generates natural-scene images from touch without image conditioning, improves tactile-driven stylization over prior work, and produces hand-less images and tactile-driven shading estimates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work is a step toward integrating tactile sensing and generative modeling across visuo-tactile synthesis tasks.

    Source: §5 · pp. 8–9

    Because touch mainly conveys material properties and microgeometry, generated images will often differ semantically from ground truth.

    Source: §5 · p. 9

Abstract

from arXiv · show

An emerging line of work has sought to generate plausible imagery from touch. Existing approaches, however, tackle only narrow aspects of the visuo-tactile synthesis problem, and lag significantly behind the quality of cross-modal synthesis methods in other domains. We draw on recent advances in latent diffusion to create a model for synthesizing images from tactile signals (and vice versa) and apply it to a number of visuo-tactile synthesis tasks. Using this model, we significantly outperform prior work on the tactile-driven stylization problem, i.e., manipulating an image to match a touch signal, and we are the first to successfully generate images from touch without additional sources of information about the scene. We also successfully use our model to address two novel synthesis problems: generating images that do not contain the touch sensor or the hand holding it, and estimating an image's shading from its reflectance and touch.

1. Introduction

Visuo-tactile synthesis has received comparatively little attention, and existing methods address separate tasks while lagging behind cross-modal synthesis in other domains. The paper adapts diffusion models to unify several tasks, including image generation from touch and tactile-guided editing.

1. Introduction

  • Touch-to-sight translation requires models to connect physical and material properties, but touch has received comparatively little attention in multimodal learning.Source: §1 · p. 1
  • The framework adapts diffusion models to multiple visuo-tactile synthesis problems and unifies tasks previously handled by separate methods.Source: §1 · pp. 1–2
  • The model first generates natural-scene images from touch without image conditioning, beats prior arm-addition work, and yields higher-fidelity, closer-matching tactile stylization with input-content control.Source: §1 · p. 2
  • The model also produces images without the pressing hand or sensor and estimates shading conditioned on reflectance and touch.Source: §1 · p. 2
  • Experiments use natural scenes from Touch and Go and robot-collected data from VisGel.Source: §1 · p. 2

2. Related Work

Latent diffusion reduces the dimensionality of diffusion generation, while diffusion models have been explored across image, language, and audio tasks. Prior visuo-tactile work used image-output tactile sensors and addressed narrower translation or embedding tasks.

Cross-modal synthesis with diffusion models

  • Latent diffusion works in a compressed, lower-dimensional space to accelerate diffusion, which has been applied to image synthesis, editing, and other modalities.Source: §2 · p. 2

Tactile sensing

  • Vision-based tactile sensors such as GelSight image membrane deformation, providing shape and material information in an output format usable by image-network architectures.Source: §2 · p. 2

Cross-modal models for vision and touch

  • Prior visuo-tactile methods used GANs for image-conditioned arm addition or limited-style editing, while other work learned joint embeddings.Source: §2 · p. 2

3. Method

The method adapts latent diffusion for cross-modal vision–touch synthesis, using contrastive visuo-tactile representations to condition image generation. It extends that framework to hand- and sensor-free generation, tactile-driven stylization, and shading estimation from reflectance and touch.

3.1.1 Contrastive Visuo-tactile Pretraining (CVTP)

  • Temporal contrastive pretraining represents visual and tactile clips with embeddings that incorporate changes over time, which can convey material properties such as surface hardness or softness.The method uses synchronized clips and a symmetric contrastive objective to distinguish corresponding visuo-tactile pairs from other samples.Source: §3.1.1 · p. 3

3.1.2 Touch-conditioned Image Generation

  • Touch-conditioned generation uses latent diffusion to denoise an image representation conditioned on a tactile representation, with classifier-free guidance trading off sample quality and diversity.The method describes sampling noise at inference and iteratively predicting noise to obtain a denoised latent; experiments use Touch and Go and VisGel data.Source: §3.1.2 · pp. 3–4

3.2.1 Generating realistic images without hands

  • To generate images without hands or touch sensors, training masks the diffusion loss on pixels overlapping hands, depriving the model of supervision for those pixels.Hand masks are downsampled to match the image latent dimensions.Source: §3.2.1 · p. 4

3.2.2 Tactile-driven Image Stylization

  • Tactile-driven stylization edits an input image toward a desired touch signal while controlling how much input content is preserved through the denoising start step N.The method adapts guided image synthesis by adding noise to the input latent and denoising it conditioned on the desired tactile signal.Source: §3.2.2 · p. 4

3.2.3 Tactile-driven Shading Estimation

  • Tactile-driven shading estimation conditions on reflectance and touch to reconstruct the original image, inferring shading as the missing component.The method downsamples estimated reflectance and concatenates it with the noisy latent at each denoising step.Source: §3.2.3 · p. 5

4. Results

Experiments evaluate visuo-tactile synthesis on Touch and Go and VisGel, reporting gains over prior methods and examining tactile conditioning and controllable stylization.

4.1. Implementation details

  • The diffusion model is based on Stable Diffusion and uses a frozen pretrained VQ-GAN latent representation, with 200 denoising steps at inference.Training uses batch size 96 on four RTX A40 GPUs.Source: §4.1 · p. 5

4.2. Experimental Setup

  • The Touch and Go dataset supplies 13,900 touches across roughly 4,000 object instances and 20 material categories for all three tasks.The study also uses robot-collected VisGel data, containing 195 household objects, 12k touches, and around 3M frames.Source: §4.2 · pp. 5–6

4.3. Cross-modal Generation

  • Our methods outperform existing state-of-the-art methods [38] by a large margin across evaluation metrics on Touch and Go and VisGel.The comparison covers cross-modal generation in both directions; the VisGel setup uses a reference scene photo.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 7

4.2. Experimental Setup

  • Evaluation combines image-distribution, tactile-visual embedding, and material-class metrics for touch-to-image generation, with SSIM and PSNR used for image-to-touch generation.CVTP, material classification consistency, and FID are restricted to Touch and Go touch-to-image evaluation because pretrained natural-image metrics may not generalize to other modalities or robot-collected data.Source: §4.2 · p. 6, §4.3 · p. 7

4.4. Tactile-Driven Image Stylization

  • Touch and Go stylization produces higher-fidelity images that match touch more closely than prior methods and captures styles they fail to represent.The evaluation compares against a modified CycleGAN and Yang et al. [64].Source: Table 3 · p. 7, §4.4 · pp. 7–8

4.5. Tactile-driven Shading Estimation

  • Adding touch improves every evaluated shading-estimation metric, especially for roughness and smoothness on Lambertian surfaces such as bricks.The comparison is against reflectance-only image prediction.Source: Table 4 · p. 8, §4.5 · p. 8

4.6. Analysis

  • More tactile frames improve generation, with the five-frame model performing best; material-class conditioning also beats conditioning on a single frame.Conditioned generation improves over unconditional generation, and pretraining improves over no pretraining.Source: §4.6 · p. 8, Figure 9 · p. 8
  • Larger denoising starting points change image appearance more drastically, while N = T/2 provides an empirically observed trade-off between tactile matching and preserving structure.At N = T, stylization becomes equivalent to touch-to-image generation; small N produces little change.Source: Figure 8 · p. 8, §4.6 · p. 8

5. Conclusion

The paper presents a visuo-tactile diffusion model for prior and novel synthesis tasks, including touch-driven image generation, hand-less imagery, and shading estimation. It identifies semantic divergence from ground truth as a limitation and flags potential disinformation use.

5. Conclusion

  • The model unifies previous cross-modal synthesis tasks and supports novel visuo-tactile synthesis problems.The paper reports realistic natural-scene images generated from touch without image-based conditioning, as well as hand-less imagery and tactile-driven shading estimation.Source: §5 · pp. 8–9
  • The paper reports more realistic tactile-driven stylization results than prior work.Source: §5 · p. 9
  • Because touch mainly conveys material properties and microgeometry, generated images may differ semantically from ground truth; the imagery could also be used for disinformation.Source: §5 · p. 9

A. Model Architecture and Implementation Details

The appendix points to additional latent-diffusion hyperparameter details and a table covering first-stage, condition, and LDM model settings.

A. Model Architecture and Implementation Details

B. More Qualitative Results

The section presents additional qualitative results across visuo-tactile generation, tactile-driven stylization, and tactile-driven shading estimation.

Loading 2309.15117v1…