PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo

arXiv:2212.09478v2cs.CV

TL;DR

Single-modality generators do not jointly provide the visual and auditory experiences found in multimodal content, while audio and video also differ structurally and must remain temporally aligned. MM-Diffusion addresses this gap with coupled audio and video denoising subnetworks and random-shift cross-modal attention. It reports superior unconditional results, zero-shot conditional generation, and strong human-evaluation outcomes on Landscape and AIST++.

  • Problem

    Single-modality generation does not jointly model the visual and auditory content common in human-created media, while audio and video require parallel processing and temporal alignment.

    Source: §1 · pp. 1–2

  • Method

    MM-Diffusion couples audio and video denoising autoencoders with a random-shift cross-modal attention block for joint generation.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    56.7% FVD and 37.7% FAD gains are reported on AIST++ over previous SOTA models, with zero-shot conditional generation and 10k-vote Turing-test support.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MM-Diffusion generates realistic audio-video pairs jointly and extends multimodal diffusion beyond single-modality generation.

    Source: §5 · p. 8

    The forward processes for audio and video are assumed independent because the modalities have different distributions.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic consistency across modalities, we propose a novel random-shift based attention block bridging over the two subnets, which enables efficient cross-modal alignment, and thus reinforces the audio-video fidelity for each other. Extensive experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks (e.g., video-to-audio). In particular, we achieve the best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of 10k votes further demonstrate dominant preferences for our model. The code and pre-trained models can be downloaded at https://github.com/researchmm/MM-Diffusion.

1. Introduction

The paper studies joint audio-video generation to address the limits of single-modality content generation and the challenge of processing distinct modalities together. MM-Diffusion uses coupled denoising autoencoders and achieves strong unconditional and zero-shot conditional results.

Motivation

  • Single-modality generators provide either visual or auditory experiences, whereas much human-created Web content combines both modalities.This motivates studying joint audio-video generation for more engaging perception.Source: §1 · pp. 1–2

Challenges

  • Joint diffusion must process video’s 3D spatial-temporal RGB signals and audio’s 1D temporal waveform digits in parallel.The differing data patterns make unified multi-modal processing a central design challenge.Source: §1 · p. 2

Method

  • MM-Diffusion uses two-coupled denoising autoencoders whose outputs jointly produce less-noisy audio and video samples across diffusion steps.The design learns a joint distribution over both modalities.Source: §1 · p. 2

Results

  • 25.0% visual and 32.9% audio gains by FVD and FAD, respectively, are reported on Landscape against SOTA modality-specific unconditional models.The comparison evaluates video and audio generation quality separately using their respective metrics.Source: §1 · p. 2
  • 56.7% and 37.7% gains by FVD and FAD, respectively, are reported on AIST++ over previous SOTA models.The paper also demonstrates zero-shot conditional generation without task-driven fine-tuning.Source: §1 · p. 2
  • Turing tests receiving 10k votes further support the high-fidelity performance of the generated results for common users.The evaluation combines benchmark results with human preference testing.Source: §1 · p. 2

2. Related Work

Prior cross-modality generation methods addressed directional or bidirectional transfer but generally generated one modality at a time. The paper positions its approach as simultaneous generation of audio and video.

Diffusion Models

  • Diffusion probabilistic models use forward and reverse processes that map signals to noise and noise back to signals.These processes can also be formulated by solving differential equations.Source: §2 · p. 2

Cross-Modality Generation

  • Earlier cross-modal work covered text-to-visual, text-to-audio, audio-to-visual, visual-to-audio, and visual transfer tasks.The related literature spans multiple modality directions and generation settings.Source: §2 · p. 2
  • Previous methods generated one modality at a time, while MM-Diffusion generates two modalities simultaneously.The contrast distinguishes simultaneous joint generation from prior conditional or transfer frameworks.Source: §2 · p. 3

3. Approach

MM-Diffusion extends diffusion modeling to jointly generate semantically consistent audio-video pairs through coupled denoising streams. Its architecture combines modality-specific processing with efficient random-shift cross-modal attention during reverse diffusion.

3. Approach

  • MM-Diffusion jointly reconstructs audio and video within one diffusion process rather than generating a single modality.Independent forward processes map each modality toward noise, while a unified reverse model reconstructs consistent pairs.Source: §3.2 · pp. 3–4
  • The unified reverse model takes noisy audio and video as inputs, allowing each modality to reinforce the other's generation quality.The model learns a joint distribution over paired modalities through coupled denoising.Source: §3.2 · pp. 3–4

3.3. Coupled U-Net for Joint Audio-Video Denoising

  • The coupled U-Net uses separate audio and video streams at every denoising step, accommodating their different tensor shapes and patterns.Audio uses channel-temporal tensors, while video uses frame-channel-height-width tensors.Source: §3.3 · p. 4, §3.2 · p. 4
  • Efficient multi-modal blocks use dilated 1D convolutions for audio and decomposed spatial-temporal convolutions for video.The video design replaces heavy 3D convolutions with stacked 1D and 2D convolutions.Source: §3.3 · p. 4
  • Random-shift multi-modal attention aligns audio segments with sampled video-frame windows instead of computing the full cross-modal attention map.The method splits audio along video-frame time steps, samples shifted windows, and computes symmetric cross-attention between the segments.Source: §3.3 · pp. 4–5
  • Using a window size S reduces cross-modal attention complexity to O((S × H × W) × (S × T)).The reduction exploits temporal redundancy in both audio and video; smaller and larger windows are used at different U-Net depths for fine-grained and semantic correspondence.Source: §3.3 · pp. 4–5

4. Experiments

Experiments evaluate MM-Diffusion on high-quality audio-video datasets using objective metrics, user studies, ablations, and zero-shot modality transfer. The model outperforms single-modal baselines, while coupled learning and random-shift attention improve generation and alignment.

Evaluation Setup

  • Experiments use Landscape and AIST++ datasets, with FVD and KVD for video quality and separate audio metrics for audio quality.Objective evaluation uses 2,048 samples per model, 64 × 64 resolution, and averages six runs for the main results.Source: §4.1 · pp. 5–6, §4.2 · p. 6, §4.3 · p. 6

User Studies

  • User studies and Turing tests support the realism and relevance of generated audio-video pairs, including 10k votes across Landscape and AIST++.Over 80% of generated sound videos in Landscape fooled subjects, while almost half did so in AIST++.Source: §4.3 · p. 6, §4.6 · p. 8, Table 5 · p. 8

Objective Comparison

  • MM-Diffusion significantly outperforms SOTA single-modal methods on both video and audio generation across Landscape and AIST++.The authors report that Ours-v also exceeds DIGAN and TATS-base on most metrics, while coupled U-Nets further benefit both modalities.Source: §4.4 · pp. 6–7

Ablation Studies

  • Larger multi-modal attention windows improve performance, with adaptive window sizes selected by U-Net channel scale producing the best results in the ablation.These experiments use 80K training steps and compare window-size scales [2] [4].Source: §4.5 · p. 7, Table 3 · p. 8
  • Random shift improves audio quality and accelerates audio convergence, with larger gains for audio than video.The authors attribute this asymmetry to video appearance providing more information for its paired audio.Source: §4.5 · p. 7, Table 4 · p. 8

Zero-Shot Conditional Generation

  • Zero-shot transfer generates high-quality video-conditioned audio and audio-conditioned videos without task-driven fine-tuning.For audio-to-video generation, gradient guidance produces videos that are more semantically and temporally aligned than replacement-based conditioning.Source: §4.5 · pp. 7–8, Figure 5 · p. 8

5. Conclusion

MM-Diffusion advances content generation beyond single-modality diffusion by jointly generating realistic audio and video. It achieves superior objective and human-evaluation performance on audio-video benchmarks.

  • MM-Diffusion generates realistic audio and videos jointly rather than separately.Source: §5 · p. 8
  • The model’s multimodal diffusion formulation and coupled U-Net are identified as contributors to its benchmark performance.Source: §5 · p. 8
  • Objective evaluations and Turing tests show superior performance over widely used audio-video benchmarks.Source: §5 · p. 8

Supplementary Material

The supplementary material provides additional visualization results and reports that the model generates diverse sea scenes conditioned on wave sounds.

A. Algorithm Details

The algorithm-details section documents implementation choices intended to support reproducibility, including the architecture, diffusion process, training settings, and super-resolution network.

B. Details of Human Study

The human-study details describe MOS evaluations of perceptual quality and alignment, alongside a Turing test measuring whether users identify videos as generated or real.

MOS

Turing test

C. Additional Samples

The section presents additional unconditional and zero-shot conditional audio-video generation results across Landscape, AIST++, and open-domain AudioSet examples.

  • All reported samples use 1,000 diffusion steps for best quality.Source: C. Additional Samples · p. 11
  • Unconditional results include Landscape scenes such as wind, underwater motion, rain, waterfalls, thunder, and fire cracking.Source: Figure C.1 · p. 13
  • AIST++ results pair generated dance videos with matched audio exhibiting periodic rhythms.Source: Figure C.2 · p. 14
  • Open-domain AudioSet examples cover concerts, game streaming, marathons, news playback, surfing, and first-person driving.Source: Figure C.3 · p. 15
  • The supplementary results also include visual examples of zero-shot conditional generation.Source: Figure C.4 · p. 16

Related papers

Loading 2212.09478v2…