PaperlayerView exact revision on arXiv

Source-linked AI summary

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Simian Luo, Chuanhao Yan, Chenxu Hu, Hang Zhao

arXiv:2306.17203v1cs.SDcs.CVcs.LGeess.AS

TL;DR

V2A methods have struggled to produce audio that is both semantically relevant and temporally synchronized with silent video, limiting their effectiveness for Foley production. DIFF-FOLEY addresses this with CAVP-aligned visual features, latent diffusion in spectrogram space, and double guidance, achieving strong benchmark performance and downstream applicability. Its scalability to billion-scale datasets remains untested, and diffusion inference is slower than GANs.

  • Problem

    Existing V2A methods often neglect temporal synchronization and struggle to capture nuanced audio-visual correlations while generating relevant sounds.

  • Method

    DIFF-FOLEY learns aligned audio-visual features with CAVP and conditions a spectrogram-latent LDM on aligned visual features, using double guidance during reverse diffusion.

  • Results

    DIFF-FOLEY achieves state-of-the-art V2A performance, including IS of 62.37 versus 30.01 for SpecVQGAN and 94.05% Align Acc, while generating 64 samples per batch at 0.38 seconds per sample with 25 inference steps.

  • Takeaways & Limitations

    The results support DIFF-FOLEY's practical applicability and generalization through downstream finetuning, while double guidance improves audio-visual alignment.

  • Takeaways & Limitations

    Scalability to billion-scale datasets remains untested because of limited data and computation resources, and diffusion models are slower than GANs.

Abstract

from arXiv · show

The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and generalization capabilities via downstream finetuning. Project Page: see https://diff-foley.github.io/

1 Introduction

Video-to-Audio generation supports practical Foley production but must address both audio-visual relevance and precise temporal synchronization. DIFF-FOLEY uses contrastive audio-visual pretraining and latent diffusion to improve these properties.

  • Motivation: Neural Foley can accelerate video production by generating synchronized and realistic sound effects, reducing the workload of skilled Foley artists.Traditional Foley requires laborious studio manipulation of objects to recreate sounds during post-production.
  • Motivation: V2A offers readily available audio-video training pairs and direct control over temporal synchronization, unlike text-based audio generation.The paper identifies audio-video pairs as easier to collect than text-audio pairs and emphasizes synchronization as an additional control dimension.
  • Problem: Semantic content matching and temporal synchronization are the two major goals of V2A, but many existing methods neglect synchronization.For example, drum sounds may be generated without matching the exact snare or cymbal strikes shown in the video.
  • Problem: Pretrained image and optical-flow features can fail to capture nuanced audio-video correlations, limiting simultaneous synchronization and relevance.RegNet and SpecVQGAN condition generation on RGB, flow, or ResNet features that do not adequately represent these relationships.
  • Approach: DIFF-FOLEY learns temporally and semantically aligned features with CAVP, then conditions a spectrogram-latent LDM on aligned visual features.Double guidance combines classifier-free and alignment classifier guidance to improve sample quality during the reverse process.

2 Related Work

Prior V2A work uses visual representations and generative models for sound synthesis, while DIFF-FOLEY emphasizes stronger audio-visual alignment. Its overview combines CAVP feature learning with latent diffusion in spectrogram latent space.

  • Video-to-Audio Generation: RegNet conditions a GAN on pretrained RGB and optical-flow features, while SpecVQGAN uses a Transformer conditioned on ResNet50 or RGB-and-flow features.These methods represent earlier approaches to video-conditioned sound synthesis.
  • Contrastive Pretraining: Contrastive pretraining aligns representations across modalities, motivating CAVP for learning temporally and semantically aligned audio-visual features.The paper relates CAVP to contrastive systems such as CLIP and CLAP used in other multimodal generation tasks.
  • DIFF-FOLEY Overview: DIFF-FOLEY first learns aligned audio-visual features with CAVP, then trains an LDM conditioned on aligned visual features in spectrogram latent space.The two-stage design is summarized as CAVP alignment followed by latent diffusion generation.
  • Latent Diffusion Model: Latent diffusion performs diffusion processes in data latent space, supporting more efficient computation and accelerated inference than operating directly in the original data space.The paper presents latent diffusion as a practical generative modeling choice.

3 Method

DIFF-FOLEY uses contrastive audio-visual pretraining to align temporal and semantic features, then conditions a latent diffusion model on aligned visual representations to synthesize audio. Its method also uses latent-space denoising, split-and-merge augmentation, and double guidance for audio-visual alignment.

  • 3.1 Contrastive Audio-Visual Pretraining: CAVP aligns audio and visual features by contrasting semantic pairs across videos and temporal pairs across segments of the same video.The final objective combines semantic and temporal contrast losses as L = LS + λLT, with λ = 1.
  • 3.2 LDM with Aligned Visual Representation: The LDM compresses Mel-spectrograms into low-dimensional latents, learns to denoise them conditioned on CAVP visual features, and decodes the generated latent into audio.DIFF-FOLEY generates audio samples with a duration of 8 seconds.
  • 3.2 LDM with Aligned Visual Representation: CAVP visual features contain audio-related information that enables the LDM to synthesize synchronized and relevant audio.The model projects and positionally encodes visual features before conditioning the diffusion process.
  • 3.3 Split and Merge Augmentation: Split-and-merge augmentation creates additional audio-visual feature pairs by temporally concatenating clips, helping prevent overfitting and learn temporal correspondence.Clips are randomly extracted at different time lengths from two videos before their visual features are merged.
  • 3.4 Double Guidance: Double guidance combines classifier guidance and classifier-free guidance during reverse diffusion to guide semantic and temporal alignment.The classifier predicts whether an audio-visual pair is real, while condition dropping trains conditional and unconditional noise estimates.

4 Experiments

Experiments evaluate DIFF-FOLEY on VGGSound and EPIC-Kitchens, testing visual features, temporal augmentation, guidance, and sampling choices. The model improves synchronization and audio-visual relevance while supporting downstream finetuning and faster inference.

  • Video-to-Audio Generation Results: DIFF-FOLEY achieves 62.37 IS and 94.05% Align Acc on VGGSound, while maintaining comparable MKL/FID to baseline methods.It generates 64 samples per batch at an average of 0.38 seconds per sample using 25 DPM-Solver inference steps.
  • Video-to-Audio Generation Results: In a silent golf video, DIFF-FOLEY generates the hitting sound at the 4th second, whereas baseline methods fail at that timestamp.The example highlights synchronization with a specific visual event.
  • Visual Features Analysis: CAVP visual features improve synchronization and audio-visual relevance compared with alternative visual features, although CLIP features have advantages in IS and KL.The drum example shows four generated sound spikes matching the four drum hits in the ground-truth spectrogram.
  • Temporal Augmentation Analysis: Temporal Split & Merge Augmentation improves all reported metrics, particularly Align Acc, while increasing training pairs and mitigating overfitting.The method incorporates prior knowledge of audio-visual temporal alignment.
  • Downstream Finetuning: Finetuning on EPIC-Kitchens produces synchronized sounds whose timing matches knife cutting, water flow, and plate clinking in the original videos.These qualitative results support application beyond the original VGGSound setting.
  • Ablation Study: Double guidance achieves the best IS and Align Acc, while DPM-Solver converges in 25 inference steps and enables faster DIFF-FOLEY inference.All evaluated samplers converge at 250 steps, whereas DPM-Solver reaches convergence in 25 steps.

5 Limitations and Broader Impact

DIFF-FOLEY demonstrates strong audio-visual synchronization on VGGSound and EPIC-Kitchens, but its scalability to billion-scale datasets remains untested. Diffusion-based generation is also slower than GAN-based generation, and V2A systems require caution because of potential misuse and misinformation.

  • Scalability to super large, billion-scale datasets remains untested because of limited data and computation resources.
  • Diffusion models are slower than GANs, creating a practical speed trade-off for DIFF-FOLEY.
  • V2A models can accelerate video production, but misuse and misinformation remain broader-impact concerns.

6 Conclusion

The paper presents DIFF-FOLEY as a V2A approach for highly synchronized, audio-visually relevant audio generation. It reports improved generation quality, double-guidance gains in alignment, downstream applicability, and analyses of key training and sampling factors.

  • DIFF-FOLEY generates highly synchronized audio with strong audio-visual relevance for V2A.
  • The method demonstrates superior generation quality and further improves audio-visual alignment through double guidance during the LDM reverse process.
  • Downstream finetuning demonstrates DIFF-FOLEY’s practical applicability and generalization capabilities.
  • An ablation study analyzes pretrained dataset size, guidance techniques, and diffusion samplers.

A.1 Model Architectures

DIFF-FOLEY uses a two-stage architecture: CAVP learns temporally and semantically aligned audio-visual features, then an LDM generates spectrogram latents conditioned on aligned visual features. The appendix details encoders, latent processing, cross-attention, model scales, vocoding, and alignment evaluation.

  • Two-stage architecture: CAVP first learns temporally and semantically aligned audio-visual features, followed by LDM training in Mel-spectrogram latent space.
  • Stage1 CAVP: The CAVP stage uses two-stream audio and video encoders that produce time-aligned audio and visual features from paired inputs.
  • Stage2 LDM: The LDM encodes spectrograms into compressed latents, projects CAVP visual features with positional encoding, and conditions denoising through cross-attention.
  • Model configurations: DIFF-FOLEY is evaluated in three model sizes: DIFF-FOLEY-S with 335M, DIFF-FOLEY-M with 553M, and DIFF-FOLEY-L with 859M parameters.
  • Vocoder: A vocoder converts generated Mel-spectrograms into waveform signals, and the method is compatible with deep learning-based vocoders.
  • Alignment evaluation: An alignment classifier evaluates generated-audio synchronization and audio-visual relevance by predicting whether audio-visual pairs are real.

A.2 Data Processing

The appendix specifies data preparation for V2A training, including video and spectrogram preprocessing, latent-model inputs, downstream finetuning data, and training configurations for CAVP and LDM stages.

  • Video-to-Audio Generation: V2A videos are sampled at 4 FPS into 40 frames per 10-second clip and resized to 224 × 224.
  • Video-to-Audio Generation: Audio is sampled at 16KHz and converted into 640 × 128 Mel-spectrograms using FFT Num 1024, mel basis Num 128, and hop size 250.
  • Video-to-Audio Generation: The LDM receives noisy spectrogram latents zt ∈R4×16×64, a time embedding, and CAVP-aligned visual features Ev ∈R32×512.
  • Downstream Finetuning: EPIC-Kitchens provides approximately 100 hours of low-noise egocentric kitchen activities with sound categories significantly different from VGGSound.
  • Training: CAVP training runs for 1.4M steps on 8 A100 GPUs, while LDM training runs for 24.4K steps on the same hardware.
  • Downstream Finetuning: Downstream finetuning of DIFF-FOLEY-L on EPIC-Kitchens runs for 3.2K steps on 8 A100 GPUs and takes around 9 hours.

A.4 Evaluation Details

Evaluation compares DIFF-FOLEY with SpecVQGAN and Im2Wav using established audio-generation metrics on VGGSound, with differences in video frame rates and sample durations.

  • Evaluation uses Inception Score (IS), Frechet Distance (FID), and Mean Kullback–Leibler Divergence (MKL).
  • SpecVQGAN and Im2Wav serve as the two current state-of-the-art V2A comparison models.
  • Both comparison models use pretrained weights trained on the VGGSound dataset.
  • SpecVQGAN generates 10-second audio from 10-second video features sampled at 21.5 FPS, whereas Im2Wav uses 30 FPS video.

B.1 V2A Generation Results on VGGSound.

On VGGSound examples, DIFF-FOLEY generates sounds at the appropriate moments and better matches video content than other methods.

  • DIFF-FOLEY generates corresponding sounds, including gunshots and underwater bubbling, at times matching the ground-truth audio.
  • Other methods fail to generate highly synchronized audio aligned with the video content.

B.2 Downstream Finetuning Results on EPIC-Kitchens.

The section combines EPIC-Kitchens finetuning examples with the double-guidance design, which jointly uses classifier-free and classifier guidance for semantic and temporal alignment.

  • Downstream Finetuning Results: DIFF-FOLEY generates drawer-opening sounds at the 1st and 3rd seconds in an EPIC-Kitchens video.
  • Downstream Finetuning Results: It generates plate-clinking and placing sounds at the 1st and 4th seconds in another EPIC-Kitchens video.
  • Guidance Techniques: Double guidance simultaneously leverages classifier-free guidance and classifier guidance during the diffusion reverse process.
  • Guidance Techniques: The classifier predicts whether noisy audio latents and CAVP visual features form a semantically and temporally aligned real pair.
  • Guidance Techniques: In DIFF-FOLEY, classifier-free guidance uses CAVP visual features as the general condition, while classifier guidance targets semantic and temporal alignment.
  • Guidance Techniques: The method scales the classifier-free and classifier-guidance terms with ω and γ, respectively, in the improved score estimation.
  • Guidance Techniques: The paper explains that classifier-free guidance can outperform classifier guidance because classifier gradients may lead to incorrect samples despite good classification.

C.4 The Effect of Guidance Techniques

The guidance study reports improved audio quality and synchronization as guidance scales increase, while the frozen Stable Diffusion encoder and decoder reconstruct Mel-spectrograms effectively.

  • Guidance Effects: Increasing CFG scale ω improves audio quality and synchronization in the double-guidance visualizations.
  • Guidance Effects: The visualizations vary CFG scale ω horizontally and CG scale γ vertically against ground-truth Mel-spectrograms and video frames.
  • Mel-spectrogram Reconstruction: The frozen SD-V1.4 latent encoder and decoder are used to encode and decode Mel-spectrograms without retraining.
  • Mel-spectrogram Reconstruction: The frozen encoder and decoder demonstrate strong Mel-spectrogram reconstruction capability.
Loading 2306.17203v1…