Taming Visually Guided Sound Generation

Vladimir Iashin; Esa Rahtu

Source-linked AI summary

Taming Visually Guided Sound Generation

Vladimir Iashin, Esa Rahtu

arXiv:2110.08791v1Oct 17, 2021cs.CVcs.AIcs.LGcs.SDeess.AS

TL;DR

Visually induced audio generation has been limited by short, low-fidelity, class-specific samples, slow sampling, and inadequate automatic evaluation. The paper combines a transformer over a compact spectrogram codebook with a perceptual loss, fast waveform decoding, and Melception-based fidelity and relevance metrics. Experiments report substantial improvements in quality, model size, and computation time over the state of the art.

Problem
Visually guided audio generation is limited by short, low-fidelity, one-class samples, slow sampling, and a lack of automatic metrics for fidelity and relevance.
Method
The paper trains a transformer to sample spectrograms from a video-conditioned codebook, using LPAPS for perceptual reconstruction and Melception-based FID and MKL for evaluation.
Results
The authors report substantial improvements in quality, model size, and computation time compared with the state of the art across qualitative and quantitative studies.
Takeaways & Limitations
A single spectrogram-based model supports efficient multi-class visually guided sound generation while evaluating both fidelity and relevance on small- and large-scale datasets.
Takeaways & Limitations
The model can confuse visually similar sounds and struggle to distinguish fine-grained classes such as bird species, instruments, or human voices.

Abstract

from arXiv · show

Recent advances in visually-induced audio generation are based on sampling short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio from the state-of-the-art model takes minutes on a high-end GPU. In this work, we propose a single model capable of generating visually relevant, high-fidelity sounds prompted with a set of frames from open-domain videos in less time than it takes to play it on a single GPU. We train a transformer to sample a new spectrogram from the pre-trained spectrogram codebook given the set of video features. The codebook is obtained using a variant of VQGAN trained to produce a compact sampling space with a novel spectrogram-based perceptual loss. The generated spectrogram is transformed into a waveform using a window-based GAN that significantly speeds up generation. Considering the lack of metrics for automatic evaluation of generated spectrograms, we also build a family of metrics called FID and MKL. These metrics are based on a novel sound classifier, called Melception, and designed to evaluate the fidelity and relevance of open-domain samples. Both qualitative and quantitative studies are conducted on small- and large-scale datasets to evaluate the fidelity and relevance of generated samples. We also compare our model to the state-of-the-art and observe a substantial improvement in quality, size, and computation time. Code, demo, and samples: v-iashin.github.io/SpecVQGAN

1 Introduction

Visually guided open-domain sound generation remains difficult because existing approaches are costly to scale, while automatic evaluation of generated audio is still unresolved. The paper addresses these gaps with a single efficient spectrogram-based model and new fidelity and relevance metrics.

Existing visually guided audio methods are few, often one-model-per-class, and prohibitively expensive to scale to hundreds of classes.Most prior audio work focuses on music, with limited open-domain attempts.
The paper targets a single model that generates sounds from visual input across multiple classes within a restricted time budget.It learns a codebook-based prior over spectrograms and introduces a spectrogram perceptual loss.
A transformer samples shorter codebook sequences to autoregressively construct high-fidelity spectrograms conditioned on visual cues.A MelGAN variant then converts the generated spectrogram into a waveform.
Human evaluation is expensive and tedious, while automatic evaluation of sound generation remains an open question.Existing image-generation metrics motivate automatic fidelity assessment, but audio evaluation remains unresolved.
The proposed Melception-based metrics evaluate both fidelity and relevance for open-domain generated sounds.They address the limited applicability of 1-second audio FID and the challenge of estimating visual relevance.
The work contributes an efficient multi-class synthesis approach, LPAPS perceptual loss, and Melception-based FID and MKL metrics, evaluated on VAS and VGGSound.The authors report comparisons with prior work and ablations on small- and large-scale datasets.

2 Related Work

Related work uses codebook-based priors for content generation and explores automatic audio evaluation, instrument-focused visual-audio synthesis, and open-domain audio generation. Existing open-domain systems remain limited by class-specific training and slow sampling.

Codebook-based Content Generation: VQVAE codebooks reduce the sampling space for generative algorithms across images, audio, and videos.Transformers can sample codebook representations under rich conditions such as text or visual maps.
Automatic Evaluation of Audio Synthesis: Automatic audio evaluation remains open, with prior FID methods operating on 1-second windows and missing long-term cues.Related work also uses human-judgment models as perceptual losses, but collecting training material is challenging.
Instrument Music Generation With Visual Cues: Visual-audio generation research includes short music tasks using video features, with early work based on drumstick interactions and AlexNet-LSTM models.These studies primarily target instrument or music settings.
Open-domain Audio Generation Based on Visual Cues: Open-domain visual-audio generation is challenging, and prior systems include class-specific SampleRNN or RegNet models trained separately for data classes.Although these models produce relevant and high-fidelity sounds, slow sampling and per-class training limit applicability.

3 Framework

The framework compresses spectrograms into a perceptually rich discrete codebook, autoregressively predicts codebook indices from visual features, and vocodes the decoded spectrogram into audio. Its design reduces transformer sequence length while preserving reconstruction quality and supports conditional relevance assessment.

3 Framework: The pipeline predicts a concise spectrogram codebook sequence from video features, decodes it into a spectrogram, and reconstructs a waveform with a vocoder.A transformer samples indices autoregressively, followed by codebook lookup, spectrogram decoding, and waveform synthesis.
3.1 Perceptually-rich Spectrogram Codebook: Spectrogram VQVAE encodes an input spectrogram into a reduced representation, quantizes each element using the nearest codebook entry, and decodes the quantized representation.The reconstructed spectrogram is defined as ˆx = G(zq) = G(q(E(x))).
3.1 Perceptually-rich Spectrogram Codebook: The VQVAE bottleneck remains too large for transformer sampling, while aggressive downsampling can degrade reconstruction quality.This motivates a more compact codebook representation for autoregressive generation.
3.1 Perceptually-rich Spectrogram Codebook: Spectrogram VQGAN combines codebook, reconstruction, patch-based adversarial, and LPAPS losses to preserve high-fidelity reconstructions from smaller representations.LPAPS replaces image-based perceptual guidance because spectrograms require audio-specific features and longer temporal context.
3.2 Vision-based Conditional Cross-modal Autoregressive Sampler: The transformer autoregressively samples codebook indices conditioned on frame-wise RGB and optical-flow features, then unflattens the sequence into a 2D representation for decoding.Column-major unflattening avoids the poor samples observed with row-major ordering when prompted by real audio.
3.3 Spectrogram Vocoder: MelGAN provides non-autoregressive waveform reconstruction in 2 seconds per sample on a CPU, compared with WaveNet’s 25 minutes per 10-second sample on a GPU.Because speech- or music-trained MelGAN models are unsuitable for open-domain audio, the authors train MelGAN on VGGSound.

4 Experiments

Experiments evaluate the model on VAS and VGGSound using Melception-based metrics, qualitative inspection, and comparisons of conditioning, losses, and prior systems. Results indicate strong reconstruction and visually guided generation, with relevance improving as conditioning increases but sampling slowing accordingly.

Datasets: VAS contains approximately 12.5k 6.73-second clips across 8 classes, while VGGSound contains over 200k 10-second clips spanning 309 classes.
Metrics: Melception-based MKL and FID measure relevance and fidelity in the quantitative evaluation.
Reconstruction: Spectrogram VQGAN reconstructs high-fidelity, relevant samples across classes, with VGGSound-pretrained codebooks outperforming VAS-pretrained codebooks on VAS validation.
Reconstruction: Adversarial-loss removal causes substantial metric degradation and visibly blurry reconstructed spectrograms.
Comparison: Compared with the state of the art, the model produces higher-fidelity samples faster while maintaining similar relevance with or without class labels.
Visually-Guided Sound Generation: More visual features generally improve relevance, but conditioning reduces sampling speed; visual conditioning also produces a large mean-MKL improvement over empty conditioning.
Qualitative Analysis: Human inspection of over 2k VGGSound samples found relevant audio for most classes, while errors often reflected poor audio-visual correspondence or confusion within similar instrument classes.
Qualitative Analysis: The model struggles when sound sources are unobservable and may confuse visually similar categories, including different birds and instruments.

5 Conclusion

The conclusion presents an efficient spectrogram-based approach for multi-class visually guided sound generation, together with perceptual reconstruction training and automatic fidelity and relevance metrics. Experiments on small- and large-scale datasets report quantitative and qualitative performance improvements over the state of the art.

The proposed approach combines a codebook-based spectrogram prior, LPAPS perceptual loss, and Melception-based metrics for multi-class visually guided sound generation.
Experiments on small- and large-scale datasets show improved power and efficiency compared with the state of the art in quantitative and qualitative studies.

Supplementary Material

The supplementary material provides training guidance, additional experimental results, qualitative analysis, and generated audio packages used in the figures.

Section 6.1 gives training guidance for VGGish-ish, Melception, and MelGAN, while Section 6.2 reports additional experiment results.
The supplementary material includes qualitative analysis of model properties and generated audios in MP3 format used in the figures.

6.1 Implementation Details

The implementation uses compact spectrogram codebooks, transformer-based autoregressive sampling, and neural vocoding, with dataset-specific training settings and substantial evaluation costs.

Spectrogram codebook: The spectrogram codebook encodes 80×848 mel-spectrograms into 5×53×256 representations, using 1024 codes for VGGSound and 128 for VAS.The codebook dimensions and vocabulary sizes are dataset-specific.
Perceptual loss: LPAPS uses a pretrained VGGish-ish classifier to provide perceptually rich features for compact spectrogram reconstruction.VGGish-ish is trained on VGGSound and contains 137.6M parameters.
Conditional sampler: The autoregressive sampler is a 24-layer, 16-head transformer that predicts the next codebook index from visual features and prior indices.Its output uses a K-way softmax, with K matching the codebook vocabulary size.
Waveform synthesis: The model vocodes spectrograms with a fully convolutional MelGAN variant trained on short windows, enabling arbitrary temporal dimensions at test time.Training uses random 8192-sample sequences at 22050 Hz.
Evaluation: Evaluation generates 140k VGGSound samples and 80k VAS samples, making one reported setting take 18 hours on four A100 GPUs.Sampling is parallelized across as many as six nodes.

6.2 Additional Results

Additional experiments show strong reconstruction, multi-class generation, long-form continuation, and controllable diversity, while exposing dataset and scene-level limitations.

Reconstruction: Spectrogram VQGAN shows strong reconstruction on VAS and VGGSound, including cross-dataset use without finetuning.Figures 6 and 7 include random samples and reconstructions from both datasets.
Multi-class generation: The model generates relevant, high-fidelity spectrograms across multiple classes, including VAS classes with only 314–802 training videos.Reported examples include gun, cough, and hammer.
Baseline comparison: The model produces higher-quality VAS results than RegNet while using more than two times fewer parameters.The comparison retains support for multiple classes in a single model.
Long-form generation: The sampler generates lengthy, relevant, high-fidelity samples despite training on approximately 10-second segments.The paper illustrates this capability with “the great drum solo.”
Diversity control: Top-X sampling controls temporal diversity: lower X yields repetitive texture-like sounds, whereas higher X produces more random and less relevant samples.The paper notes that Melception may respond weakly to low-diversity samples because such examples were absent from its training data.
Scene ambiguity: For ambiguous scenes, the model can confuse a person’s gender or age while generating diverse interpretations of the same visual sequence.The paper frames these errors as reasonable given the scene difficulty.
Dataset limitations: VGGSound dataset issues include silent or irrelevant videos, duplicate-like class distinctions, and visually or acoustically difficult class pairs.The authors call for a more curated large-scale dataset.
Neural audio codec: Spectrogram VQGAN achieves comparable music reconstruction to SoundStream at an order-of-magnitude smaller bitrate but struggles with fine human-speech details.The authors attribute the speech limitation to mel-scale preprocessing and the absence of narrow-domain pretraining.