PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visual Styles from Audio-Visual Associations

Tingle Li, Yichen Liu, Andrew Owens, Hang Zhao

arXiv:2205.05072v1cs.CVcs.MMcs.SDeess.AS

TL;DR

Human annotations for image styles can be costly and may miss important scene properties. This paper learns to restyle images from unlabeled audio-visual data, and reports better performance than label- and image-conditioned baselines, with predictable visual changes from audio adjustments.

  • Problem

    Human-provided style annotations can be expensive to collect and may fail to capture important scene properties.

    Source: §1 · p. 2

  • Method

    The model learns audio-driven image stylization from unlabeled audio-visual data, changing image textures to match target sounds while preserving structure.

    Source: §1 · p. 2

  • Results

    The model outperforms label- and image-conditioned baselines in quantitative and qualitative evaluations, and audio volume or mixture changes yield predictable visual changes.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Audio can serve as an intuitive style control: adjusting its volume or mixing sounds corresponds to predictable changes in image style.

    Source: §1 · p. 2

    Unexpected sounds, especially human speech, can reduce stylization to minor input adjustments, and performance may suffer when the area to convert is small.

    Source: §5 · p. 14

Abstract

from arXiv · show

From the patter of rain to the crunch of snow, the sounds we hear often convey the visual textures that appear within a scene. In this paper, we present a method for learning visual styles from unlabeled audio-visual data. Our model learns to manipulate the texture of a scene to match a sound, a problem we term audio-driven image stylization. Given a dataset of paired audio-visual data, we learn to modify input images such that, after manipulation, they are more likely to co-occur with a given input sound. In quantitative and qualitative evaluations, our sound-based model outperforms label-based approaches. We also show that audio can be an intuitive representation for manipulating images, as adjusting a sound's volume or mixing two sounds together results in predictable changes to visual style. Project webpage: https://tinglok.netlify.app/files/avstyle

1 Introduction

Human-provided style annotations can be costly and miss important scene properties; the paper learns audio-driven image stylization from unlabeled audio-visual data. Sounds specify visual styles while preserving image structure, and audio adjustments produce predictable changes in style.

1 Introduction

  • Existing label- and language-based style specifications require human annotations, which can be expensive and may miss important scene properties.The proposed approach learns from unlabeled audio-visual data instead.Source: §1 · p. 2
  • Audio-driven image stylization changes an input image’s texture to match a target sound while preserving its structural content.Examples pair bird chirps with blue skies, crunching footsteps with snow, and rain with dark skies.Source: §1 · p. 2
  • Audio provides a natural learning signal that can distinguish scene properties not captured by labels, and volume or sound mixtures offer intuitive style controls.For example, rain audio can indicate light versus heavy rain and whether dark, stormy skies are likely.Source: §1 · p. 2
  • The model combines audio-visual adversarial training with contrastive learning to match generated images to target sounds while preserving source-image structure.It uses an audio-visual discriminator and a multi-scale patch-wise structure discriminator, and is trained on egocentric hiking videos.Source: §1 · p. 2
  • Evaluations and perceptual studies demonstrate the model’s ability to stylize images from sound, while changing audio volume or mixing sounds yields predictable visual changes.The reported evaluation includes quantitative and qualitative results.Source: §1 · p. 2

2 Related Work

Prior image-translation methods use images, labels, or text to specify style, while audio-visual work has explored correspondence and sound-to-image synthesis. This paper uses natural sounds to restyle plausible source images and learns styles solely from unlabeled audio-visual data.

Image translation

  • Text- and label-based image stylization relies on human-provided or weak supervision, whereas this approach learns styles from audio without human labeling.The paper describes this as a complementary learning signal to text and labels.Source: §2 · p. 3

Audio-visual correspondence

  • Prior audio-visual correspondence work spans representation learning, source separation, grounding, spatialization, speech recognition, and scene classification; this paper uses correspondence for sound-guided image translation.Source: §2 · p. 3

Audio-visual synthesis

  • Unlike prior sound-to-image synthesis work, this paper restyles plausible source images using natural sounds and learns styles solely from unlabeled audio-visual data.Concurrent work by Lee et al. [40] uses sound to guide text-based image manipulation.Source: §2 · p. 3

3 Audio-driven Image Stylization

Audio-driven image stylization (ADIS) learns visual styles specified by sound while preserving scene structure. It addresses visual differences that can remain within a semantic category, such as rain.

3 Audio-driven Image Stylization

  • ADIS changes image texture to match a sound while preserving the image’s structural content.The task is motivated by visual variation among videos with the same category, such as heavy versus light rain.Source: §3 · pp. 3–5
  • The generator combines a source image with target-audio features, and an audio-visual discriminator encourages the output image to co-occur with that sound.The discriminator concatenates the audio spectrogram with the generated image before judging them; this fusion strategy empirically yields better visual quality.Source: §3 · p. 4
  • A contrastive objective preserves structure by matching source and generated-image patches, encouraging the encoder to discard source texture while the decoder adds target texture.The patch-wise setup assumes global correspondence is determined by local correspondences.Source: §3 · pp. 4–6
  • The final training objective combines adversarial loss with contrastive losses on source and identity-generation data to discourage unexpected changes.Parameters λ and µ adjust the strengths of the contrastive and identity losses.Source: §3 · p. 6

4 Experiments

The experiments compare audio-driven stylization with label- and image-conditioned baselines, then examine model variants and how audio changes affect image style. The results report stronger baseline comparisons and predictable visual changes when sound volume or mixtures are manipulated.

4.1 Experimental Setup

  • Into the Wild contains 94 untrimmed egocentric hiking videos totaling 50 hours, while Greatest Hits contributes selected outdoor material with less diverse backgrounds.Evaluation uses AVC, FID, human AMT rankings, and CLIP; the baselines include automatic class prediction, human keywords, and image-conditioned AdaIN.Source: §4.1 · pp. 6–8

4.2 Comparison to Baselines

  • On Into the Wild, the model outperforms three baselines on AVC, FID, and CLIP, and participants consistently prefer its audio-visual correlations.The comparison covers only Into the Wild; the authors could not meaningfully evaluate Greatest Hits with AVC because its impact sounds are poorly modeled by AudioSet.Source: §4.2 · pp. 8–9, Table 1 · p. 9

4.3 Ablation Study and Analysis

  • NCE yields results comparable to cycle-consistency while requiring one generator rather than CycleGAN’s two, which can reduce training time.The comparison is reported for Into the Wild and is supported by qualitative examples as well as Table 3.Source: §4.3 · p. 11, Table 3 · p. 11
  • The late-fusion discriminator induces generator collapse, leaving it too weak to sustain image structure and producing unsatisfactory results.The authors compare late fusion with their early-fusion discriminator in the ablation study.Source: §4.3 · pp. 10–11, Table 3 · p. 11
  • Removing identity loss worsens performance, particularly for conversions between similar landscapes, while SeLaVi pretraining yields a small improvement.The sunny-to-rainy forest example illustrates the identity-loss difference.Source: §4.3 · p. 11, Table 3 · p. 11

4.4 Audio Manipulation for Image Manipulation

  • Increasing sound volume makes image texture more prominent, and mixing crunching-snow with muddy-footstep audio gradually shifts snowy imagery toward muddy or mixed hues.The mixed audio combinations were not present during training.Source: §4.4 · p. 12, Figure 6 · p. 12, Figure 7 · p. 12
  • The qualitative examples include restyling Places images with VGG-Sound snow and rain, while reported failures include speech-containing audio and certain scene objects.The paper presents these as qualitative generalization examples and failure cases, not quantified evaluations.Source: Figure 8 · p. 13, Figure 9 · p. 13
  • An editor can adjust a video’s existing soundtrack and propagate the change to its frames; increasing light-rain audio volume produces images with more rain.The paper demonstrates this with videos from VGG-Sound.Source: §4.4 · p. 14, Figure 10 · p. 14

5 Discussion and Limitations

The model’s results are not uniformly positive: unexpected ambient sounds and small regions needing conversion pose challenges. The authors identify broader mixture-sound handling, particularly urban sound, as important future work.

5 Discussion and Limitations

  • Unexpected ambient sounds can upset the model, and human speech interference can reduce translation to minor adjustments of the input.The authors highlight handling a wider range of mixture sounds, particularly urban sound, as important future work.Source: §5 · p. 14
  • When the region to convert is small, the model may produce an odd result because it transforms the whole scene rather than locating that region.The example involves trees and sky each occupying half of the input image.Source: §5 · p. 14
  • The authors describe the results as promising in various cases but far from uniformly positive.Source: §5 · p. 14

6 Conclusion

The paper introduces audio-driven image stylization, a task for learning visual styles from paired audio-visual data, and proposes a contrastive-based audio-visual GAN with the Into the Wild dataset. It reports better quantitative and qualitative evaluations than label- and image-conditioned baselines, and predictable visual changes when audio volume or mixture changes.

6 Conclusion

  • The paper introduces audio-driven image stylization, learning visual styles from paired audio-visual data.Source: §6 · p. 14
  • The proposed approach combines a contrastive-based audio-visual GAN model with the Into the Wild egocentric hiking dataset.Source: §6 · p. 14
  • The model outperforms label- and image-conditioned baselines in quantitative and qualitative evaluations.Source: §6 · p. 14
  • Changing audio volume or mixture results in predictable visual changes.Source: §6 · p. 14

A.1 Into the Wild dataset

Into the Wild is an egocentric hiking-video dataset introduced for audio-driven image stylization, selected because hiking has strong nature-related audio-visual associations. The videos were collected from YouTube and soundtrack tags were checked with YAMNet; category labels come from YAMNet predictions, not human annotation.

A.1 Into the Wild dataset

  • Into the Wild comprises egocentric hiking videos, chosen because hiking features strong audio-visual associations with nature.Source: §A.1 · p. 20
  • Videos were collected from YouTube using hiking-related search terms, and YAMNet tags were used to check that soundtracks contain actual sounds rather than replacement background music.Source: §A.1 · p. 20
  • The dataset contains 94 untrimmed videos, with category labels derived from YAMNet predictions rather than human annotation.The listed categories include crunching snow, gravel and dirt; rain; birds chirping; ocean; stream; and human speech.Source: §A.1 · p. 20

A.2 Training Details

Training uses a ResNet-based generator and PatchGAN discriminator, with training settings intentionally matched to CycleGAN and CUT except for batch size and audio network. The appendix also describes the datasets’ clip sampling and train-test splits, alongside figures for dataset statistics and the AMT rating interface.

A.2 Training Details

  • Training uses a 9-residual-block ResNet generator, PatchGAN discriminator, least-squares GAN loss, ResNet18 audio encoder, batch size 16, and Adam at learning rate 0.002.The model trains for 50 epochs, holding the learning rate constant for 30 and linearly decaying it over the final 20.Source: §A.2 · p. 20
  • The structure loss uses features from five scales, samples 256 random locations per layer, and maps them through a two-layer MLP to 256-dimensional features.The sampled features include input RGB pixels, two downsampling convolution features, and the first and fifth residual-block features.Source: §A.2 · p. 20
  • Into the Wild is split into three-second clips, yielding 454,560 images and 56,820 audios, with 20% of audios randomly reserved for testing.Source: §A.2 · p. 20
  • The outdoor subset of Greatest Hits covers dirt, grass, gravel, leaf, and water, yielding 32,172 images and 8,043 audios, with 15% of audios reserved for testing.Source: §A.2 · p. 20
  • The appendix includes a figure of Into the Wild dataset statistics and a screenshot of the AMT audio-visual correspondence rating interface.Source: Figure 11 · p. 21, Figure 12 · p. 21

A.3 Evaluation Details

This section specifies the objective and human evaluation measures used to assess audio-visual correspondence and image-conversion quality.

A.3 Evaluation Details

  • AVC uses OpenL3 audio and visual features to compute average cosine similarity for each image-audio pair.The features come from an “env” content-type model with a 512-dimensional linear spectrogram representation.Source: §A.3 · p. 21
  • FID is computed from Inception V3 average-pooling activations after images are resized to 299-by-299.Clean-FID is used to avoid errors from implementation differences such as resizing functions.Source: §A.3 · p. 21
  • CLIP conversion quality is assessed by cosine similarity between image features and labels using an off-the-shelf ViT-B/32 model.CLIP is used as a feature extractor rather than for zero-shot prediction.Source: §A.3 · p. 21
  • Table 4 compares quantitative results for different pre-training methods on the Into the Wild dataset.The supplied caption identifies the table’s scope but does not provide its values.Source: Table 4 · p. 22
  • Twenty MTurk workers ranked audio-image correlations for 1,000 random test samples, assigning scores from 1 to 4.The best-ranked sample received 4 points and the worst received 1; final scores were averaged.Source: §A.3 · p. 22

A.4 Additional Results

Additional results report qualitative comparisons, out-of-distribution audio generalization, and a pre-training comparison in which Wav2CLIP surpasses the alternatives across three metrics.

A.4 Additional Results

  • Additional qualitative comparisons show the model produces results described as better or competitive, with versatility relative to label-based baselines.Figure 13 extends the main-paper qualitative comparison of the model, baselines, and ablations.Source: §A.4 · p. 22, Figure 13 · p. 23
  • Using Places images and VGG-Sound audio, the model generates plausible images matching out-of-distribution audio content.Figure 14 extends the qualitative generalization experiment.Source: §A.4 · p. 22, Figure 14 · p. 24
  • Wav2CLIP surpasses training from scratch and SeLaVi pre-training on AVC, FID, and CLIP metrics.The authors interpret this comparison as indicating stronger representation ability; they suggest Wav2CLIP may benefit from exposure to well-annotated image-text pairs.Source: §A.4 · p. 22

Related papers

Loading 2205.05072v1…