Source-linked AI summary
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo
TL;DR
Single-modality generators do not jointly provide the visual and auditory experiences found in multimodal content, while audio and video also differ structurally and must remain temporally aligned. MM-Diffusion addresses this gap with coupled audio and video denoising subnetworks and random-shift cross-modal attention. It reports superior unconditional results, zero-shot conditional generation, and strong human-evaluation outcomes on Landscape and AIST++.
Problem
Single-modality generation does not jointly model the visual and auditory content common in human-created media, while audio and video require parallel processing and temporal alignment.
Method
MM-Diffusion couples audio and video denoising autoencoders with a random-shift cross-modal attention block for joint generation.
Results
56.7% FVD and 37.7% FAD gains are reported on AIST++ over previous SOTA models, with zero-shot conditional generation and 10k-vote Turing-test support.
Takeaways & Limitations
MM-Diffusion generates realistic audio-video pairs jointly and extends multimodal diffusion beyond single-modality generation.
Takeaways & Limitations
The forward processes for audio and video are assumed independent because the modalities have different distributions.
Abstract
from arXiv · showhide
We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic consistency across modalities, we propose a novel random-shift based attention block bridging over the two subnets, which enables efficient cross-modal alignment, and thus reinforces the audio-video fidelity for each other. Extensive experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks (e.g., video-to-audio). In particular, we achieve the best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of 10k votes further demonstrate dominant preferences for our model. The code and pre-trained models can be downloaded at https://github.com/researchmm/MM-Diffusion.
1. Introduction
The paper studies joint audio-video generation to address the limits of single-modality content generation and the challenge of processing distinct modalities together. MM-Diffusion uses coupled denoising autoencoders and achieves strong unconditional and zero-shot conditional results.
- Motivation: Single-modality generators provide either visual or auditory experiences, whereas much human-created Web content combines both modalities.This motivates studying joint audio-video generation for more engaging perception.
- Challenges: Joint diffusion must process video’s 3D spatial-temporal RGB signals and audio’s 1D temporal waveform digits in parallel.The differing data patterns make unified multi-modal processing a central design challenge.
- Method: MM-Diffusion uses two-coupled denoising autoencoders whose outputs jointly produce less-noisy audio and video samples across diffusion steps.The design learns a joint distribution over both modalities.
- Results: 25.0% visual and 32.9% audio gains by FVD and FAD, respectively, are reported on Landscape against SOTA modality-specific unconditional models.The comparison evaluates video and audio generation quality separately using their respective metrics.
- Results: 56.7% and 37.7% gains by FVD and FAD, respectively, are reported on AIST++ over previous SOTA models.The paper also demonstrates zero-shot conditional generation without task-driven fine-tuning.
- Results: Turing tests receiving 10k votes further support the high-fidelity performance of the generated results for common users.The evaluation combines benchmark results with human preference testing.
2. Related Work
Prior cross-modality generation methods addressed directional or bidirectional transfer but generally generated one modality at a time. The paper positions its approach as simultaneous generation of audio and video.
- Diffusion Models: Diffusion probabilistic models use forward and reverse processes that map signals to noise and noise back to signals.These processes can also be formulated by solving differential equations.
- Cross-Modality Generation: Earlier cross-modal work covered text-to-visual, text-to-audio, audio-to-visual, visual-to-audio, and visual transfer tasks.The related literature spans multiple modality directions and generation settings.
- Cross-Modality Generation: Previous methods generated one modality at a time, while MM-Diffusion generates two modalities simultaneously.The contrast distinguishes simultaneous joint generation from prior conditional or transfer frameworks.
3. Approach
MM-Diffusion extends diffusion modeling to jointly generate semantically consistent audio-video pairs through coupled denoising streams. Its architecture combines modality-specific processing with efficient random-shift cross-modal attention during reverse diffusion.
- 3. Approach: MM-Diffusion jointly reconstructs audio and video within one diffusion process rather than generating a single modality.Independent forward processes map each modality toward noise, while a unified reverse model reconstructs consistent pairs.
- 3. Approach: The unified reverse model takes noisy audio and video as inputs, allowing each modality to reinforce the other's generation quality.The model learns a joint distribution over paired modalities through coupled denoising.
- 3.3. Coupled U-Net for Joint Audio-Video Denoising: The coupled U-Net uses separate audio and video streams at every denoising step, accommodating their different tensor shapes and patterns.Audio uses channel-temporal tensors, while video uses frame-channel-height-width tensors.
- 3.3. Coupled U-Net for Joint Audio-Video Denoising: Efficient multi-modal blocks use dilated 1D convolutions for audio and decomposed spatial-temporal convolutions for video.The video design replaces heavy 3D convolutions with stacked 1D and 2D convolutions.
- 3.3. Coupled U-Net for Joint Audio-Video Denoising: Random-shift multi-modal attention aligns audio segments with sampled video-frame windows instead of computing the full cross-modal attention map.The method splits audio along video-frame time steps, samples shifted windows, and computes symmetric cross-attention between the segments.
- 3.3. Coupled U-Net for Joint Audio-Video Denoising: Using a window size S reduces cross-modal attention complexity to O((S × H × W) × (S × T)).The reduction exploits temporal redundancy in both audio and video; smaller and larger windows are used at different U-Net depths for fine-grained and semantic correspondence.
4. Experiments
Experiments evaluate MM-Diffusion on high-quality audio-video datasets using objective metrics, user studies, ablations, and zero-shot modality transfer. The model outperforms single-modal baselines, while coupled learning and random-shift attention improve generation and alignment.
- Evaluation Setup: Experiments use Landscape and AIST++ datasets, with FVD and KVD for video quality and separate audio metrics for audio quality.Objective evaluation uses 2,048 samples per model, 64 × 64 resolution, and averages six runs for the main results.
- User Studies: User studies and Turing tests support the realism and relevance of generated audio-video pairs, including 10k votes across Landscape and AIST++.Over 80% of generated sound videos in Landscape fooled subjects, while almost half did so in AIST++.
- Objective Comparison: MM-Diffusion significantly outperforms SOTA single-modal methods on both video and audio generation across Landscape and AIST++.The authors report that Ours-v also exceeds DIGAN and TATS-base on most metrics, while coupled U-Nets further benefit both modalities.
- Ablation Studies: Larger multi-modal attention windows improve performance, with adaptive window sizes selected by U-Net channel scale producing the best results in the ablation.These experiments use 80K training steps and compare window-size scales [2] [4].
- Ablation Studies: Random shift improves audio quality and accelerates audio convergence, with larger gains for audio than video.The authors attribute this asymmetry to video appearance providing more information for its paired audio.
- Zero-Shot Conditional Generation: Zero-shot transfer generates high-quality video-conditioned audio and audio-conditioned videos without task-driven fine-tuning.For audio-to-video generation, gradient guidance produces videos that are more semantically and temporally aligned than replacement-based conditioning.
5. Conclusion
MM-Diffusion advances content generation beyond single-modality diffusion by jointly generating realistic audio and video. It achieves superior objective and human-evaluation performance on audio-video benchmarks.
- MM-Diffusion generates realistic audio and videos jointly rather than separately.
- The model’s multimodal diffusion formulation and coupled U-Net are identified as contributors to its benchmark performance.
- Objective evaluations and Turing tests show superior performance over widely used audio-video benchmarks.
Supplementary Material
The supplementary material provides additional visualization results and reports that the model generates diverse sea scenes conditioned on wave sounds.
- On Landscape, the model generates diverse sea video scenes from the given wave sound.
- The supplementary visualizations include results for audio-based video generation on Landscape.
- The supplementary material presents algorithm details, human-study details, and additional visualization results.
A. Algorithm Details
The algorithm-details section documents implementation choices intended to support reproducibility, including the architecture, diffusion process, training settings, and super-resolution network.
- Implementation details are introduced to ensure reproducibility of the reported results.
- The documented components include the model architecture and diffusion process.
- Training settings for the Coupled U-Net and the super-resolution network are formally listed.
B. Details of Human Study
The human-study details describe MOS evaluations of perceptual quality and alignment, alongside a Turing test measuring whether users identify videos as generated or real.
- MOS: MOS testers rate video quality, audio quality, and video-audio alignment.
- Turing test: Turing-test participants vote whether each video is machine-generated, indeterminate, or real.
- Turing test: The study treats indeterminate and real votes as evidence relevant to the Turing-test assessment.
C. Additional Samples
The section presents additional unconditional and zero-shot conditional audio-video generation results across Landscape, AIST++, and open-domain AudioSet examples.
- All reported samples use 1,000 diffusion steps for best quality.
- Unconditional results include Landscape scenes such as wind, underwater motion, rain, waterfalls, thunder, and fire cracking.
- AIST++ results pair generated dance videos with matched audio exhibiting periodic rhythms.
- Open-domain AudioSet examples cover concerts, game streaming, marathons, news playback, surfing, and first-person driving.
- The supplementary results also include visual examples of zero-shot conditional generation.