PaperlayerView exact revision on arXiv

Source-linked AI summary

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji

arXiv:2412.15322v2cs.CVcs.LGcs.SDeess.AS

TL;DR

Foley synthesis requires high-quality audio that matches a video's meaning and timing, while video-to-audio training is constrained by limited audiovisual data. MMAudio jointly trains across video, audio, and text and adds conditional synchronization; it achieves new state-of-the-art among public video-to-audio models and remains competitive on text-to-audio.

  • Problem

    Foley synthesis requires high-quality audio that is semantically and temporally aligned with video, while existing video-to-audio training is constrained by limited audiovisual data.

    Source: §1 · p. 1

  • Method

    MMAudio jointly trains a multimodal model on audiovisual and audio-text data, with a conditional synchronization module aligning video conditions to audio latents.

    Source: §1 · p. 2

  • Results

    The 157M model outperforms prior public models on distribution matching, audio quality, semantic alignment, and temporal alignment; its synchronization module yields 50% relative improvement in synchronization score.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    MMAudio also achieves competitive text-to-audio performance, showing that joint training does not hinder single-modality performance.

    Source: §1 · p. 2

    MMAudio generates unintelligible mumbles when prompted for human speech, which its authors attribute to the model's focus on general Foley audio.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio. In contrast to single-modality training conditioned on (limited) video data only, MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples. Additionally, we improve audio-visual synchrony with a conditional synchronization module that aligns video conditions with audio latents at the frame level. Trained with a flow matching objective, MMAudio achieves new video-to-audio state-of-the-art among public models in terms of audio quality, semantic alignment, and audio-visual synchronization, while having a low inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio also achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance. Code and demo are available at: https://hkchengrex.github.io/MMAudio

1. Introduction

MMAudio targets Foley’s need for semantically and temporally aligned video-conditioned audio through multimodal joint training and a synchronization module. The paper reports gains in audio quality, alignment, and public video-to-audio performance.

1. Introduction

  • Foley targets high-quality ambient sounds and event effects rather than music or speech, requiring semantic and temporal alignment with video.Source: §1 · p. 1
  • Prior audio-visual-only methods [1] are data-limited, while control-module finetuning [20] [66] [69] [73] complicates architecture and leaves sufficient degrees of freedom for all scenarios unclear.The commonly used VGGSound dataset [1] contains around 550 hours of videos.Source: §1 · pp. 1–2
  • Joint training yields 10% lower Fréchet Distance, 15% higher Inception Score, 4% higher ImageBind score, and 14% better synchronization score.Source: §1 · p. 2
  • MMAudio jointly trains one multimodal transformer on audio-visual and audio-text data, masking missing modalities to learn a unified semantic space and natural-audio distribution.Source: §1 · p. 2
  • High-frame-rate features from an audiovisual desynchronization detector [19] condition adaLN scales and biases, yielding 50% relative improvement in synchronization score.Source: §1 · p. 2
  • The smallest MMAudio model (157M parameters) achieves new state-of-the-art video-to-audio performance among public models and competitive text-to-audio performance.The paper reports that joint training does not hinder single-modality text-to-audio performance.Source: §1 · p. 2

2. Related Works

Related work motivates MMAudio’s joint multimodal training for shared semantics and direct temporal alignment, while distinguishing multimodal-conditioned audio generation from broader multimodal generation.

Semantic alignment

  • Prior audio-visual work learns semantic alignment using generation [2] [6] [18] [44] [49] [60] [66] [71] or contrastive [42] objectives; the authors additionally train on audio-text pairs and argue that these semantics transfer to video-text pairs through joint training in a shared semantic space.Source: §2 · p. 2

Temporal alignment

  • MMAudio aligns video and audio directly through Synchformer [19] embeddings, rather than handcrafted proxies, and uses higher visual positional-embedding frequencies for temporal alignment.Unlike V-AURA [65], it supports multimodal text training and 8–10-second generations rather than a 2.56-second context window; Mei et al. concurrently subsample high-frequency audio positional embeddings, an equivalent effect under the stated frequency condition.Source: §2 · p. 2

Multimodal conditioning

  • Unlike control-module approaches [13] [17] [20] [34] [47] [54] [73], MMAudio trains modalities simultaneously for joint semantics and omnidirectional feature sharing.Seeing-and-Hearing [69] optimizes alignment at test time but can be slow and yield low-quality, temporally misaligned audio. VATT [40] requires video during training; compared with MultiFoley [3], MMAudio uses 24 FPS rather than 8 FPS visual features, which the paper says improves synchronization.Source: §2 · p. 2

Multimodal generation

  • MMAudio focuses on multimodal-conditioned audio generation; existing multimodal generation approaches [25] [55] [61] [62] are not yet competitive with dedicated video-to-audio models.The paper presents its formulation and architecture as a foundation for future multimodal generation work.Source: §2 · pp. 2–3

3. MMAudio

MMAudio learns a conditional flow to generate audio latents from noise, using video and text conditions to target high-quality audio that is semantically and temporally aligned. Its multimodal transformer combines the modalities, while a frame-aligned synchronization module adds fine-grained visual conditioning.

3.1. Preliminaries

  • Conditional flow matching trains the velocity field to move noise toward data-conditioned audio latents, which are decoded into waveforms at inference.At test time, an ODE solver integrates from t = 0 to t = 1; audio is represented using mel spectrograms encoded by a pretrained VAE, then decoded and vocoded.Source: §3.1 · p. 3, §3.2 · p. 3

3.3. Multimodal Transformer

  • MMAudio combines visual, text, and audio branches in multimodal transformer blocks, followed by audio-only blocks, to estimate the flow from conditions and timestep.Joint attention lets modalities communicate; the subsequent audio-only blocks enable a deeper network at the same parameter count and compute. Inputs are tokenized at their respective frame rates, with no absolute positional encoding to support variable durations.Source: §3.2 · p. 3, §3.3 · p. 3
  • Aligned RoPE scales visual positional frequencies by 31.25/8 to align 8-fps video with 31.25-fps audio, while ConvMLPs use 1D convolutions to capture local temporal structure.RoPE is applied to visual and audio queries and keys, but not text; global conditioning combines timestep and pooled visual and text features through adaptive layer normalization.Source: §3.3 · p. 4

3.4. Conditional Synchronization Module

  • A conditional synchronization module injects 24-fps Synchformer video features as frame-aligned conditioning into audio-stream normalization, providing per-token modulation for precise synchrony.The features are upsampled to match the audio-latent frame rate; the module supplements cross-modal attention, whose soft aggregation can hamper precision.Source: §3.4 · pp. 4–5

3.5.1. Multimodal Datasets

  • Training combines about 180K 10-second VGGSound videos with 951K audio-clip/text pairs from AudioCaps, Clotho, and WavCaps, using empty visual and synchronization tokens for audio-text data.The VGGSound class names serve as text input, and test sets of VGGSound and AudioCaps are excluded from training to address minor train/test contamination.Source: §3.5.1 · p. 5

3.5.2. Implementation Details

  • The default S-16kHz model uses 20-dimensional latents at 31.25 fps; inference uses 25-step Euler integration with guidance strength 4.5 and permits either condition to be omitted.The model has N1 = 4 multimodal blocks, N2 = 8 audio-only blocks, and hidden dimension h = 448. Training randomly masks video or text with 10% probability for classifier-free guidance.Source: §3.5.2 · p. 5

4. Experiments

MMAudio is evaluated for audio distribution, quality, semantic alignment, and synchronization, with experiments showing strong video-to-audio results, competitive text-to-audio performance, and benefits from multimodal training and architectural choices. The section also identifies a limitation in generating human speech.

4.1. Metrics

  • Evaluation measures distribution matching with FD and KL, audio quality with IS, semantic alignment with IB-score, and temporal alignment with DeSync; DeSync averages two crops for 8-second audio.The crop-based DeSync scores are not directly comparable with Viertola et al. [65].Source: §4.1 · pp. 5–6

4.2. Main Results

  • The 157M-parameter model improves on prior methods across video-to-audio evaluation dimensions, except that Seeing-and-Hearing has a higher IB-score; larger models further improve FD_PaSST and IB-score with diminishing returns.Evaluation uses 8-second generations on the VGGSound test set; ReWaS is evaluated at 5 seconds. Seeing-and-Hearing directly optimizes IB-score at test time, which MMAudio does not.Source: §4.2 · p. 6, Table 1 · p. 6
  • Without fine-tuning, MMAudio achieves state-of-the-art text-to-audio semantic alignment (CLAP) and audio quality (IS), but has a worse FD_VGG score than recent works.The authors hypothesize that VGGish’s local 0.96-second features favor local rather than globally and semantically consistent audio.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3. Ablations

  • Masking text from either training data stream worsens results, uncaptioned audio helps, and VGGSound class labels alone have little effect, supporting a shared text feature space and large-scale multimodal training.The default AVT+AT setting uses audio-video-text data with VGGSound class labels and audio-text data.Source: §4.3 · pp. 7–8
  • Increasing multimodal training data improves FD_PaSST, IB-score, and DeSync with diminishing returns; without audio-text data, the model overfits and training is stopped early.Audio-visual and audio-text data are sampled at roughly a 1:1 ratio except when audio-text data is omitted.Source: §4.3 · p. 8, Table 5 · p. 8
  • The conditional synchronization module gives the best temporal alignment, while aligned RoPE embeddings improve audio-visual synchrony.Summing upsampled synchronization features with visual features yields higher IS, which the authors hypothesize may result from a longer visual-token sequence.Source: §4.3 · p. 8, Table 6 · p. 8
  • ConvMLP better captures local temporal structure than MLP, especially for synchronization; the default transformer-block allocation performs similarly to more single-modal blocks and better than fewer.The compared allocations use roughly the same parameter budget; the authors attribute the advantage to deeper networks from single-modal blocks.Source: §4.3 · p. 8, Table 7 · p. 8

4.4. Limitations

  • When prompted for human speech from mouth movement or text, MMAudio generates unintelligible mumbles, which the authors attribute to the complexity of speech and the model’s Foley focus.Source: §4.4 · p. 8

5. Conclusion

MMAudio jointly trains across audio, video, and text with conditional synchronization, achieving state-of-the-art performance among public models and comparable performance with Movie Gen Audio.

5. Conclusion

  • MMAudio achieves state-of-the-art performance among public models and comparable performance with Movie Gen Audio.The authors attribute this result to combining multimodal joint training with a conditional synchronization module.Source: §5 · p. 8
  • Jointly considering audio, video, and text enables effective data scaling and cross-modal semantic alignment.The authors describe MMAudio as the first multimodal training pipeline to jointly consider all three modalities.Source: §5 · p. 8
  • The authors argue that multimodal formulation is key to synthesizing data in any modality, and that MMAudio lays a foundation in the audio-video-text space.Source: §5 · p. 8

A. User Study

In a VGGSound user study, MMAudio received significantly higher ratings than the selected baselines for audio quality, semantic alignment, and temporal alignment.

A. User Study

  • The comparison used MMAudio-L-44.1kHz and four baselines selected for leading objective metrics: Seeing and Hearing [69], V-AURA [65], VATT [40], and V2A-Mapper [66].The baselines were selected for strong ImageBind, DeSync, KL, or Fréchet-distance scores, respectively, among methods other than MMAudio.Source: A. User Study · p. 13
  • Twenty-three participants evaluated 40 samples each, drawn from eight VGGSound test videos across five methods, with sample order shuffled within each video group.Videos below 360p or containing human speech were excluded; the study collected 920 responses for each aspect.Source: A. User Study · p. 13
  • Participants used a 1–5 Likert scale to rate audio quality, whether audio fit the video scenario, and whether its events were synchronized in time.Instructions defined low quality as noisy, unclear, or muffled, and temporal misalignment as delayed, advanced, or mistimed events.Source: A. User Study · p. 13
  • MMAudio received significantly higher user ratings than the baselines in all three aspects, consistent with the paper’s objective metrics.The aspects were audio quality, semantic alignment with video, and temporal alignment with video.Source: Table A1 · p. 13, A. User Study · p. 13

B. Comparisons with Movie Gen Audio

On Movie Gen Audio Bench, MMAudio matches Movie Gen Audio [52] on audio quality and leads on synchrony, while Movie Gen Audio leads on semantic alignment; user ratings show a similar tradeoff.

B. Comparisons with Movie Gen Audio

  • Movie Gen Audio [52] represents the industry’s current state of the art, with a 13-billion-parameter model trained on non-public data more than 100× larger than MMAudio’s.The comparison benchmarks this private model against MMAudio as a public model.Source: B. Comparisons with Movie Gen Audio · p. 13
  • MMAudio-L-44.1kHz was compared on Movie Gen Audio Bench videos using the corresponding prompts; absent ground-truth audio, objective evaluation was limited to IS, IB-score, DeSync, and CLAP.At the time of writing, 527 Movie Gen Audio outputs had been released; the user study used five videos and collected 230 responses per aspect from 23 participants, excluding very quiet Movie Gen Audio samples.Source: B. Comparisons with Movie Gen Audio · pp. 13–14
  • MMAudio ties Movie Gen Audio [52] on IS, while Movie Gen Audio leads on IB-score and CLAP and MMAudio leads on DeSync; user ratings are comparable, with slight semantic and temporal advantages, respectively.The subjective comparison found MMAudio slightly worse in semantic alignment and slightly better in temporal alignment.Source: Table A2 · p. 14, B. Comparisons with Movie Gen Audio · p. 14
  • Movie Gen Audio [52] consistently performs better in the low-performance IB-score regime, but the gap narrows in the high-performance region.The authors attribute the gap to MMAudio's limited VGGSound training coverage of 310 classes and hypothesize that open-world training data could reduce it; they report the same pattern on CLAP at a smaller scale.Source: Figure A1 · p. 14, B. Comparisons with Movie Gen Audio · p. 14

C. Evaluation on the Greatest Hits Dataset

On the unseen Greatest Hits test set, MMAudio performs significantly better than baselines on model-free onset-alignment metrics, addressing potential bias from the model-based DeSync metric.

C. Evaluation on the Greatest Hits Dataset

  • MMAudio significantly outperforms baselines on Greatest Hits onset-alignment accuracy, average precision (AP), and F1-score.The test uses 244 videos with labeled sound events; models are evaluated on each video’s first 8 seconds by comparing generated-audio onsets with ground-truth labels.Source: C. Evaluation on the Greatest Hits… · p. 14
  • High AP alone does not imply accurate onset alignment: it can result from generating very few onsets, as with Seeing&Hearing.The models and baselines were not trained on Greatest Hits.Source: C. Evaluation on the Greatest Hits… · p. 14

D. Ablations on Filling in Missing Modalities

Replacing missing modalities with learnable embeddings or zeros yields no significant difference, while random modality dropout supports robustness to missing modalities. The training setup uses learnable visual and synchronization embeddings and empty strings for missing text.

D. Ablations on Filling in Missing Modalities

E. Details on Data Overlaps … G.4. Details on Synchronization Features

The appendix documents dataset contamination and its mitigation, then specifies audio-latent and model variants alongside network-conditioning and synchronization implementations.

E. Details on Data Overlaps

  • Overlaps across AudioSet, AudioCaps, VGGSound, and AVSync15 can bias VGGSound and AVSync15 evaluations; the authors remove training-set overlaps with VGGSound and AudioCaps test sets and omit AVSync15 evaluation.Table A4 summarizes the observed overlaps. The training data also excludes the Clotho test set.Source: E. Details on Data Overlaps · p. 15, Table A4 · p. 16

F. Details on the Audio Latents

  • Audio latents come from STFT mel spectrograms encoded by a pretrained VAE, then are decoded to spectrograms and vocoded into waveforms.The VAE follows Make-An-Audio 2 [15] with reconstruction, adversarial, and KL objectives; magnitude-preserving layers remove extreme latent values without significant performance difference. The 16kHz model uses BigVGAN [35], while the 44.1kHz model uses BigVGAN-v2 [35].Source: F. Details on the Audio Latents · p. 16

G.1. Model Variants

  • The model family ranges from the default S-16kHz to higher-rate S-, M-, and L-44.1kHz variants, scaling latent dimension, hidden dimension, and layer counts.S-16kHz uses 20-dimensional, 31.25fps latents with N1 = 4, N2 = 8, h = 448; S-44.1kHz uses 40-dimensional, 43.07fps latents. M-44.1kHz doubles h to 896, while L-44.1kHz also increases layers to N1 = 7 and N2 = 14.Source: §G.1 · p. 16

G.2. Projection Layers

  • Projection layers map text, visual, synchronization, and audio features to hidden dimension h, with modality-specific linear, convolutional, activation, and ConvMLP stages.Text uses a linear layer followed by an MLP; visual features use a linear layer followed by a kernel-3 ConvMLP. Sync features use kernel-7 convolution, SELU [28], then kernel-3 ConvMLP; audio features use kernel-7 convolution, SELU [28], then kernel-7 ConvMLP.Source: §G.2 · pp. 16–17

G.3. Gating

  • Global gating broadcasts condition-derived scales across all sequence tokens, whereas per-token gating applies frame-aligned condition-derived scales separately to each token.Both scale the input features through an MLP-derived transformation; only global gating broadcasts the scales along sequence length.Source: §G.3 · p. 17

G.4. Details on Synchronization Features

  • Synchronization features use the pretrained Synchformer [19] visual encoder on 25-fps frames, producing eight features per overlapping 16-frame clip at stride 8.For the tested 8- and 10-second videos, synchronization features are exactly 24 FPS, and a learnable positional embedding of length 8 is added to them.Source: §G.4 · p. 17

G.5. Illustration of the “sum sync with visual” Ablation · G.6. Visualization of Aligned RoPE

The appendix illustrates the “sum sync with visual” ablation and compares default with aligned RoPE for sequences of different temporal lengths. The ablation has mixed metric outcomes, while aligned RoPE yields temporal alignment.

G.5. Illustration of the “sum sync with visual” Ablation

  • Figure A4 illustrates the “sum sync with visual” ablation architecture, while Figure A3 depicts Synchformer feature extraction.Visual features are upsampled with nearest-neighbor interpolation to match the synchronization-feature frame rate.Source: §G.5 · p. 17, Figure A3 · p. 18
  • The “sum sync with visual” architecture has worse FD_{\text {PaSST}}, IB-score, and DeSync, but a better inception score (IS).Source: §G.5 · p. 17
  • The authors hypothesize that increased visual-token counts from upsampling enable finer-grained computations.Source: §G.5 · p. 17

G.6. Visualization of Aligned RoPE

  • The aligned RoPE visualization uses audio and visual sequences of lengths 250 and 64, respectively, with channel size C = 64.These lengths correspond to 31.25 fps audio and 8 fps video over 8 seconds.Source: §G.6 · p. 18
  • The default-RoPE affinity compares the 250-step audio sequence with the 64-step visual sequence encoded using RoPE_default.Source: §G.6 · p. 18
  • The aligned-RoPE affinity uses the same sequences but encodes the visual sequence using RoPE_aligned.Source: §G.6 · p. 18
  • Aligned RoPE attains temporal alignment in the visualization shown in Figure A5.Source: §G.6 · p. 18

H. Training Details · I. Additional Visualizations

MMAudio’s training uses shared hyperparameters, balanced multimodal sampling, and measures to address overfitting and duplicate videos. Additional visualizations show temporal alignment improvements alongside examples where generations capture events but may not match ground truth precisely.

H. Training Details

  • Training uses a 1e-4 base learning rate for 300K iterations, batch size 512, and reductions to 1e-5 and 1e-6 after 80% and 90% of training.The schedule includes a 1K-step linear warm-up.Source: H. Training Details · p. 18
  • AdamW uses β1=0.9, β2=0.95, and 1e-6 weight decay; the default β2=0.999 occasionally caused NaN collapse.Training also uses post-hoc EMA with relative width 0.05, bf16 mixed precision, and offline precomputed audio latents and visual embeddings.Source: H. Training Details · p. 18
  • Table A7 summarizes training resources by model size, with H100 GPUs used in all settings.Source: Table A7 · p. 18
  • Figures A4–A5 illustrate the “sum sync with visual” ablation and show aligned RoPE temporally aligns sequences with different frame rates, unlike default RoPE.Source: Figure A4 · p. 19, Figure A5 · p. 19
  • To balance 951K audio-text samples against 180K audio-text-visual samples, training duplicates the latter 5X, or 3X for medium and large models to mitigate overfitting.Source: H. Training Details · pp. 19–20
  • VGGSound [1] contains repeated uploads across train and test sets; video-ID overlap removal does not catch all duplicates, but prior works used the same dataset.The authors state that their training scheme therefore remains a fair comparison.Source: H. Training Details · p. 20

I. Additional Visualizations

  • Additional spectrogram visualizations compare MMAudio with prior works, including on the Greatest Hits [48] dataset.Generated samples and comparisons are also provided on the project page.Source: §I · p. 20, Figure A8 · p. 22
  • Figures A6–A7 show precise golf-strike and visible-string event capture, dog-bark bursts despite imperfect timing, and plausible complex-scene audio that may not match ground truth.Source: Figure A6 · p. 20, Figure A7 · p. 21

Related papers

Loading 2412.15322v2…