PaperlayerView exact revision on arXiv

Source-linked AI summary

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji

arXiv:2412.15322v2cs.CVcs.LGcs.SDeess.AS

TL;DR

Video-to-audio Foley synthesis requires high-quality audio that is semantically and temporally aligned, but audio-visual data are limited and costly to curate. MMAudio addresses this with multimodal joint training over video, audio, and text plus a conditional synchronization module. It reports state-of-the-art public-model video-to-audio performance, competitive text-to-audio results, and efficient 157M-parameter inference, while human speech remains a limitation.

  • Problem

    Foley synthesis must produce high-quality audio that is semantically and temporally aligned with video, despite limited and costly-to-curate audio-visual data.

  • Method

    MMAudio jointly trains video, audio, and text in one multimodal transformer with missing-modality masking and adds a conditional synchronization module using high-frame-rate visual features.

  • Results

    MMAudio achieves new state-of-the-art video-to-audio performance among public models and competitive text-to-audio performance, with reported gains in audio quality, semantic alignment, and synchronization.

  • Takeaways & Limitations

    Joint multimodal training enables data scaling and cross-modal understanding without hindering single-modality text-to-audio performance.

  • Takeaways & Limitations

    The model generates unintelligible human-speech mumbles and is aimed at general audio effects rather than speech.

Abstract

from arXiv · show

We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio. In contrast to single-modality training conditioned on (limited) video data only, MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples. Additionally, we improve audio-visual synchrony with a conditional synchronization module that aligns video conditions with audio latents at the frame level. Trained with a flow matching objective, MMAudio achieves new video-to-audio state-of-the-art among public models in terms of audio quality, semantic alignment, and audio-visual synchronization, while having a low inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio also achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance. Code and demo are available at: https://hkchengrex.github.io/MMAudio

1. Introduction

MMAudio targets Foley generation that is both semantically and temporally aligned with video, while addressing limited audio-visual data through multimodal joint training. It combines shared multimodal learning with conditional synchronization to improve audio quality and alignment.

  • Foley synthesizes ambient sounds and event-induced effects, excluding background music and human speech.
  • Audio-visual Foley requires semantic alignment between visible concepts and sounds, plus temporal alignment because humans detect misalignment as slight as 25 ms.
  • Existing approaches are constrained by scarce, costly-to-curate audio-visual data or by control modules that complicate pretrained text-to-audio models.
  • MMAudio jointly trains video, audio, and text in one transformer while masking missing modalities, enabling training from scratch on audio-visual and audio-text datasets.
  • 10% lower Fréchet Distance, 15% higher Inception Score, 4% higher ImageBind score, and 14% better synchronization score are reported with joint training.
  • A conditional synchronization module uses high-frame-rate visual features and adaptive-layer-normalization scales and biases, producing a reported 50% relative synchronization improvement.
  • The smallest 157M-parameter model achieves new state-of-the-art video-to-audio performance among public models, while text-to-audio remains competitive.

2. Related Works

Prior work learns semantic alignment from audio-visual pairs and increasingly uses audio-text data for richer semantics. For temporal alignment, MMAudio uses deep features from a pretrained desynchronization detector rather than handcrafted proxy features, while supporting longer contexts than a cited autoregressive method.

  • Semantic alignment: Audio semantics can be learned from paired audio-text data, which joint training uses to build a shared semantic space and richer representations.
  • Temporal alignment: Temporal-alignment methods have used handcrafted proxies such as onsets, energy, and waveform RMS; MMAudio instead uses deep embeddings from Synchformer.
  • Temporal alignment: Compared with V-AURA, MMAudio combines multimodal text training with longer-term 8–10-second temporally consistent generation rather than a 2.56-second context window.
  • The authors position their multimodal formulation and architecture as a foundation for future multimodal conditioned audio generation.

3. MMAudio

MMAudio uses conditional flow matching to generate audio latents from video and text conditions, combining multimodal and audio-only transformer blocks. Its architecture adds temporal-position and frame-aligned conditioning mechanisms for audio-visual synchrony and supports joint multimodal training.

  • Conditional flow matching: Conditional flow matching learns a time-dependent velocity field that numerically transports Gaussian noise to audio latents conditioned on video and text.At inference, an ODE solver integrates from t = 0 to t = 1; the resulting latent is decoded into audio.
  • Audio encoding: Audio waveforms are converted to mel spectrograms, encoded into latent representations with a pretrained VAE, then decoded and vocoded during testing.Modeling in latent space is used for computational efficiency.
  • Multimodal transformer: MMAudio combines multimodal transformer blocks with audio-only transformer blocks so video, text, and audio can interact while preserving deeper audio processing.The architecture uses N1 multimodal blocks followed by N2 audio-only blocks, enabling joint training across audio-visual and audio-text data.
  • Temporal modeling: Aligned RoPE embeddings and ConvMLPs encode cross-stream timing and local temporal structure for visual and audio sequences with different frame rates.RoPE is applied to visual and audio streams, while ConvMLPs use one-dimensional convolutions with kernel size 3 and padding 1.
  • Conditioning: Global conditioning modulates every token through adaLN scales and biases derived from the flow timestep and pooled visual and text features.The same condition is broadcast across the sequence, distinguishing global conditioning from token-level modulation.
  • Synchronization conditioning: A synchronization module extracts 24 fps visual features, upsamples them to the audio-latent frame rate, and injects per-token adaLN modulation.The frame-aligned condition is derived from Synchformer features and provides fine-grained control in the audio stream.

4. Experiments

MMAudio is evaluated across distribution matching, audio quality, semantic alignment, and temporal alignment for video-to-audio and text-to-audio generation. Experiments also examine modality usage, training-data scale, synchronization features, architecture choices, and limitations.

  • Metrics: The video-to-audio evaluation measures distribution matching with FD and KL, audio quality with IS and IB-score, and temporal alignment with DeSync.FD is computed with PaSST, PANNs, and VGGish embeddings.
  • Video-to-audio: MMAudio’s 157M model improves distribution matching, audio quality, semantic alignment, and temporal alignment over prior video-to-audio methods on VGGSound, while remaining fast.The comparison uses the VGGSound test set and 8-second generations; Seeing&Hearing remains stronger on IB-score because it directly optimizes that metric at test time.
  • Temporal alignment: Table 2 evaluates onset accuracy, average precision, and F1-score on Greatest Hits to provide a model-free synchronization assessment alongside DeSync on VGGSound.Greatest Hits contains distinct labeled sound events and is not used to train the evaluated models.
  • Text-to-audio: MMAudio achieves state-of-the-art semantic alignment and audio quality in text-to-audio generation, although its FD_VGG score is worse than recent works.The evaluation uses 10-second AudioCaps samples and reports FD_PANNs, FD_VGG, IS, and CLAP.
  • Training modalities: Masking text from either audio-visual or audio-text data worsens results, while adding uncaptioned audio improves performance, indicating benefits from a shared text space and natural-audio distribution learning.The default AVT+AT setting uses audio-visual-text and audio-text data; omitting audio-text data can cause overfitting.
  • Ablations: More multimodal training data improves distribution matching, semantic alignment, and temporal alignment with diminishing returns, while ConvMLP improves synchronization by capturing local temporal structure.Architecture comparisons use roughly matched parameter budgets.
  • Limitation: MMAudio generates unintelligible mumbles for human-speech prompts, reflecting a scope boundary for a model designed around general Foley effects.The authors attribute this limitation to the greater complexity of human speech, including languages, tones, and grammars.

5. Conclusion

MMAudio jointly models audio, video, and text for scalable cross-modal semantic alignment, while a conditional synchronization module improves audio-visual synchrony. The resulting system achieves new state-of-the-art performance among public video-to-audio models.

  • MMAudio jointly considers audio, video, and text, enabling effective data scaling and cross-modal semantic alignment.

A. User Study

A user study evaluates audio quality, semantic alignment, and temporal alignment through ratings from 23 participants. MMAudio receives significantly higher ratings across all three aspects, consistent with the paper’s objective metrics.

  • The study evaluates audio quality, semantic alignment, and temporal alignment between generated audio and video.
  • 920 responses per aspect were collected from 23 participants.
  • MMAudio receives significantly higher user ratings in all three evaluated aspects.
  • Temporal misalignment includes delayed or advanced audio and events occurring at incorrect times relative to the video.

B. Comparisons with Movie Gen Audio

MMAudio is compared with Movie Gen Audio on synthetic MovieGen videos because accessible outputs lack corresponding ground-truth audio. The two systems have similar audio quality, while their relative semantic and temporal alignment strengths differ.

  • Movie Gen Audio is an industry state-of-the-art video-to-audio system with 13 billion parameters trained on non-public data over 100 times larger than MMAudio’s.
  • The Movie Gen Audio Bench contains 5,272 accessible generations, but its synthetic videos differ from real-world video distributions.
  • MMAudio and Movie Gen Audio obtain the same audio quality IS score, while Movie Gen Audio leads on IB-score and CLAP and MMAudio leads on DeSync.
  • Subjectively, MMAudio is slightly worse in semantic alignment and slightly better in temporal alignment than Movie Gen Audio.
  • Movie Gen Audio performs more consistently in the low-IB-score regime, especially for video concepts not covered by MMAudio’s training data.

C. Evaluation on the Greatest Hits Dataset

The Greatest Hits experiment evaluates temporal alignment without relying on the model-based DeSync metric by comparing generated audio onsets with labeled sound events. MMAudio performs significantly better on these model-free metrics.

  • The Greatest Hits test set contains 244 videos with distinct, labeled sound events, and neither evaluated models nor baselines trained on it.
  • Temporal alignment is assessed by comparing generated-audio onsets with ground-truth event labels using accuracy, average precision, and F1-score.
  • MMAudio achieves significantly better performance than the baselines on the model-free temporal-alignment metrics.

D. Ablations on Filling in Missing Modalities

MMAudio fills missing modalities with learnable embeddings or an empty text string, while randomly dropping modalities during training to support robust conditional generation. Experiments found no significant difference between learnable embeddings and zeros.

  • Missing-modality handling: MMAudio replaces missing visual modalities with learnable embeddings and missing text with an empty string.Random modality dropping during training enables classifier-free guidance and improves robustness to missing modalities.
  • Ablation result: Learnable embeddings and zero-filled missing modalities produce no significant performance difference.The comparison is summarized in Table A3.

E. Details on Data Overlaps

The appendix documents dataset contamination risks, implementation details, and model configurations, while describing how synchronization features and multimodal network components are constructed. The authors remove identified overlaps from training data and avoid AVSync15 evaluation because of contamination concerns.

  • Data overlaps: Training and testing data overlap across AudioSet, VGGSound, AudioCaps, and AVSync15 can bias video-to-audio evaluations.The authors identify overlaps involving pretrained audio components, text-to-audio backbones, and jointly used datasets.
  • Data cleaning: The authors remove overlaps from training data and do not evaluate on AVSync15 because of severe cross-contamination with VGGSound.They also remove the Clotho test set from training data.
  • Network details: MMAudio uses separate projection layers for text, visual, synchronization, and audio features before multimodal processing.Visual and audio streams use convolutional projections, while text uses a linear layer followed by an MLP.
  • Conditional modulation: Global gating broadcasts condition-dependent scales across all sequence tokens, whereas per-token gating applies frame-aligned scales independently to each token.Both mechanisms modulate the audio stream through MLP-generated scales.
  • Synchronization features: Synchformer extracts synchronization features from 25-fps video using overlapping 16-frame clips with stride 8, producing eight features per clip.For 8- and 10-second videos, the resulting synchronization feature rate is exactly 24 fps.

G.5. Illustration of the “sum sync with visual” Ablation

The ablation compares a synchronization-feature pathway with visual features and examines temporal affinity under different positional embeddings. Aligned RoPE achieves temporal alignment, while the summed synchronization-with-visual variant trades worse several metrics for a better inception score.

  • Sum sync with visual: The “sum sync with visual” ablation upsamples visual features to match the synchronization-feature frame rate.Nearest-neighbor upsampling increases the number of visual tokens used in the computation.
  • Ablation outcome: The ablation has worse FD_PaSST, IB-score, and synchronization scores but a better inception score than the corresponding alternative.The authors hypothesize that the inception-score gain comes from finer-grained computation after visual-token upsampling.
  • Aligned positional embeddings: Aligned RoPE produces temporal alignment between audio and visual sequences with different frame rates, unlike default RoPE.The affinity visualization compares 250 audio positions with 64 visual positions for 8-second sequences.

H. Training Details

The appendix details training resources, multimodal data balancing, model variants, and qualitative comparisons. It also provides figures illustrating synchronization behavior and generated spectrograms across methods.

  • Training setup: MMAudio trains with a 1e-4 learning rate, 1K-step warm-up, 300K iterations, batch size 512, AdamW, and weight decay 1e-6.Using the default beta_2=0.999 occasionally caused training collapse to NaN, so beta_2=0.95 was used.
  • Model scaling: The appendix reports model variants and training resources for different MMAudio sizes and configurations.Table A6 summarizes model variants and generation time, while Table A7 summarizes GPU resources.
  • Synchronization analysis: Aligned RoPE visualizes temporal alignment between sequences with different frame rates, while synchronization and visual ablations compare alternative conditioning designs.Figures A3 and A4 illustrate synchronization-feature extraction and the “sum sync with visual” architecture.
  • Data balancing: Audio-text data outnumbers audio-text-visual data 951K to 180K, so audio-text-visual samples are duplicated to balance training.The default duplication ratio is 5X, reduced to 3X for medium and large models to mitigate overfitting.
  • Qualitative comparisons: Qualitative spectrogram figures show precise capture of visible events, while ambiguous scenarios can yield plausible audio without exact ground-truth timing.Examples include golf-ball strikes, dog barks, string playing, and complex scenes.
  • Dataset comparison: The appendix presents comparisons with prior methods on the Greatest Hits dataset.These comparisons are summarized in Figure A8.
Loading 2412.15322v2…