PaperlayerView exact revision on arXiv

Source-linked AI summary

I Hear Your True Colors: Image Guided Audio Generation

Roy Sheffer, Yossi Adi

arXiv:2211.03089v2cs.SDeess.AS

TL;DR

Open-domain image-to-audio generation lacks the large paired datasets available for text-to-image modeling, motivating methods that exploit image-audio data in videos. IM2WAV uses CLIP-conditioned hierarchical audio token modeling with two Transformer stages and classifier-free guidance, and it outperforms evaluated baselines on fidelity and relevance while introducing ImageHear for future evaluation.

  • Problem

    Open-domain visually guided audio generation is constrained by the difficulty of collecting large text-audio datasets, while prior methods have limited generalization or require labels.

    Source: §1 · p. 1

  • Method

    IM2WAV encodes audio with a hierarchical VQ-VAE, uses CLIP image representations, generates LOW tokens with one autoregressive Transformer, upsamples them with a second Transformer, and applies classifier-free guidance.

    Source: §1 · p. 1, §2 · p. 2, §2.3 · p. 2

  • Results

    IM2WAV significantly outperforms evaluated baselines on both fidelity and relevance metrics.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    ImageHear provides an out-of-domain benchmark intended to make future image-to-audio model comparisons more standardized and accessible.

    Source: §4 · p. 4

    The method assumes evenly spaced input frames and future work includes evaluating complex multi-scene and multi-object image sequences.

    Source: §2.3 · p. 2, §4 · p. 4

Abstract

from arXiv · show

We propose Im2Wav, an image guided open-domain audio generation system. Given an input image or a sequence of images, Im2Wav generates a semantically relevant sound. Im2Wav is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-VAE based model. We first produce a low-level audio representation using a language model. Then, we upsample the audio tokens using an additional language model to generate a high-fidelity audio sample. We use the rich semantics of a pre-trained CLIP (Contrastive Language-Image Pre-training) embedding as a visual representation to condition the language model. In addition, to steer the generation process towards the conditioning image, we apply the classifier-free guidance method. Results suggest that Im2Wav significantly outperforms the evaluated baselines in both fidelity and relevance evaluation metrics. Additionally, we provide an ablation study to better assess the impact of each of the method components on overall performance. Lastly, to better evaluate image-to-audio models, we propose an out-of-domain image dataset, denoted as ImageHear. ImageHear can be used as a benchmark for evaluating future image-to-audio models. Samples and code can be found inside the manuscript.

1. INTRODUCTION

IM2WAV addresses open-domain visually guided audio generation by using image or video data and proposes a label-free, CLIP-conditioned Transformer system. It uses hierarchical audio modeling and reports superior fidelity and relevance against evaluated baselines.

1. INTRODUCTION

  • Limited availability of large text-audio datasets motivates using web videos, which naturally contain paired image-audio data.DALL-E 2 used approximately 650M text-image pairs, whereas AUDIOGEN used approximately 3M text-audio pairs.Source: §1 · p. 1
  • The system first generates low-level discrete audio tokens and then upsamples them to higher-resolution tokens before decoding both sequences into time-domain audio.CLIP features extracted from the input images condition the autoregressive generation stages.Source: Figure 1 · p. 1, §1 · p. 2
  • Prior visually guided audio methods rely on class-aware training, labeled data, or fixed class sets, limiting generalization to unseen classes.SpecVQGAN provides open-domain generation across multiple classes but generates mel-spectrograms that require neural-vocoder conversion.Source: §1 · p. 1
  • IM2WAV generates semantically relevant audio from a single image or image sequence using a label-free Transformer language model conditioned on image representations.The system is inspired by prior audio language-model approaches and is designed for natural images.Source: §1 · pp. 1–2
  • IM2WAV significantly outperforms evaluated baselines across fidelity and relevance metrics, with additional label analysis and ablations assessing its components.Figure 1 provides a visual description of the proposed architecture.Source: Figure 1 · p. 1, §1 · p. 2

2. METHOD

The method represents audio with hierarchical VQ-VAE tokens, encodes image sequences with CLIP, and models audio generation using two autoregressive Transformers. Classifier-free guidance steers generation toward the visual input while balancing audio quality and diversity.

2. METHOD

  • IM2WAV comprises an audio encoder-decoder, a pretrained image encoder, and an audio language model operating over discrete audio tokens.The overall objective is to generate an audio file from a sequence of images.Source: §2 · p. 2

2.1. Audio Encoder & Decoder.

  • The hierarchical VQ-VAE encodes raw audio into discrete multi-level representations and decodes quantized codebook vectors back into a time-domain signal.The representation uses decreasing sequence lengths across levels, with each level learning its own codebook.Source: §2.1 · p. 2

2.2. Image Encoder.

  • CLIP encodes each image, and the resulting sequence of image features is averaged over time and projected through MLP layers into a single conditioning vector.The choice of CLIP is intended to leverage semantic information from multimodal image-text training.Source: §2.2 · p. 2

2.3. Sequence Modeling.

  • Two autoregressive Transformers model discrete audio at different resolutions: LOW predicts semantically grounded tokens, while UP reconstructs higher-resolution tokens from LOW outputs and image features.The LOW level provides semantic foundations, whereas UP completes fine details; evenly spaced frames support arbitrary image counts.Source: §2.3 · p. 2

2.4. Classifier Free Guidance.

  • Classifier-free guidance is applied during training and inference to steer generation toward the input images by combining visually conditioned and unconditioned token distributions.The authors report that applying CFG only to LOW is sufficient to greatly improve performance, with η = 3 used as the guidance scale.Source: §2.4 · p. 3

3. EXPERIMENTS

Experiments compare IM2WAV with SpecVQGAN using fidelity and visual-relevance metrics on VGGSound videos and IMAGEHEAR single images, then assess component effects through ablation. IM2WAV is reported to outperform evaluated baselines, while UP primarily improves fidelity and CFG primarily improves relevance.

3.1. Experimental Setup

  • The evaluation uses 4-second, 16-kHz audio from VGGSound and 100 web images spanning 30 classes in the out-of-distribution IMAGEHEAR benchmark.IMAGEHEAR generates 120 audios per image class; VGGSound contains approximately 200,000 ten-second videos, evaluated here using their initial four seconds.Source: §3.1 · p. 3
  • Fidelity is measured with FAD, while relevance is assessed using KL divergence, classifier accuracy, and CLIP-based audio-image similarity.The CLIP-based score replaces the text encoder with Wav2Clip and averages image-wise scores for videos.Source: §3.1 · pp. 3–4

3.2. Results

  • IM2WAV is superior to evaluated baselines on both fidelity and relevance metrics across VGGSound videos and IMAGEHEAR single images.All evaluated models achieve better relevance metrics with IMAGEHEAR single images than with VGGSound videos, and the relevance metrics preserve the same model ranking.Source: Table 1 · p. 3, §3.2 · p. 4
  • IM2WAV produces diverse sounds across more classes than SpecVQGAN, according to per-class accuracy analysis on IMAGEHEAR and VGGSound.Figure 2 reports per-class accuracy for single-image IMAGEHEAR conditions and image-sequence VGGSound conditions.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Ablation study

  • UP improves fidelity because every model using UP achieves lower FAD than models without UP, regardless of CFG or EVERY.The ablation attributes semantic abstraction mainly to LOW and natural-sound refinement to the higher-resolution UP level.Source: §3.3 · p. 4, Table 2 · p. 4
  • CFG improves visual relevance across UP and EVERY settings, whereas EVERY has a relatively small effect and improves when combined with CFG.The ablation compares CFG, UP, and temporally corresponding-frame conditioning at every token.Source: §3.3 · p. 4, Table 2 · p. 4

4. CONCLUSION & FUTURE WORK

The paper concludes that IM2WAV enables open-domain image-to-audio generation with stronger fidelity and relevance than evaluated baselines. It also introduces IMAGEHEAR as an out-of-domain benchmark and identifies multimodal features and complex multi-scene sequences as future directions.

  • IM2WAV is presented as an open-domain image-to-audio method that is superior to evaluated baselines on fidelity and relevance metrics.Source: §4 · p. 4
  • IMAGEHEAR is proposed as an out-of-domain benchmark intended to standardize and broaden future evaluation of image-to-audio models.Source: §4 · p. 4
  • Future work will explore multimodal features and evaluate IM2WAV on complex multi-scene and multi-object image sequences.Source: §4 · p. 4

Related papers

Loading 2211.03089v2…