MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation

• Existing training-free methods improve multimodal alignment without training but may degrade single-modal fidelity, whereas training-based methods improve both but require substantial computation.Source: §1 · p. 1
• These approaches also depend heavily on base-model architectures, so updating the base models can require manual redesign and trial and error.Source: §1 · pp. 1–2
• The proposed method is training-based yet model-agnostic, using a lightweight joint guidance module that adjusts outputs from audio and video base models.Source: §1 · p. 2
• The method treats the base diffusion models as black boxes and avoids backpropagation through them during optimization.Source: §1 · p. 2
• Joint guidance is computed from the gradient of an optimal discriminator distinguishing real audio-video pairs from independently generated fake pairs, with DLSM-inspired regularization.Source: §1 · p. 2
• Experiments show that the method integrates the single-modal models while maintaining their generation performance without significant computational cost.Source: §1 · p. 2

2.1 AUDIO-VIDEO JOINT GENERATION BY DIFFUSION MODELS

• Existing joint audio-video methods include architectures trained from scratch or extensions of pre-trained models that remain strongly tied to specific diffusion-model designs.Source: §2.1 · p. 2
• MMDisCo treats base models as black boxes and depends only on their outputs, allowing application across different base-model architectures.Source: §2.1 · p. 2
• Embedding-based universal guidance can provide semantic alignment but does not explicitly sample from the actual joint audio-video distribution.Source: §2.1 · p. 2

2.2 GUIDANCE FOR PRE-TRAINED DIFFUSION MODELS

• Figure 1 shows a joint discriminator distinguishing real pairs from independently generated pairs, alongside a denoising objective that matches its input gradient to residual noise.Source: Figure 1 · p. 3
• The paper is the first described extension of classifier guidance to joint multimodal generation, deriving guidance that enables sampling from the joint distribution.Source: §2.2 · p. 3
• The method extends DLSM-style regularization to stabilize discriminator gradients and address score mismatch during multimodal generation.Source: §2.2 · p. 3
• Unlike discriminator guidance designed for single-modality quality improvement, this method bridges single-modal and joint distributions using one discriminator for both domains.Source: §2.2 · p. 3

3.1 PRELIMINARY: DIFFUSION MODELS

• Diffusion models reverse a data-to-noise process by predicting noise and iteratively sampling from the learned reverse transitions.The predicted noise estimates the score of the noisy data distribution and is used from timestep T down to 1.Source: §3.1 · p. 4
• Classifier guidance extends diffusion generation by adding a conditional-score term estimated from a classifier gradient.The unconditional score comes from the diffusion model, while the classifier gradient supplies conditioning information.Source: §3.1 · p. 4

3.2 JOINT SCORE ESTIMATION ON THE TOP OF PRE-TRAINED DIFFUSION MODELS

• MMDisCo decomposes each joint score into an independently estimated marginal score and a conditional score for the other modality.For audio-video variables x and y, the joint score uses the two Bayes-rule decompositions for gradients with respect to x_t and y_t.Source: §3.2 · pp. 4–5
• Instead of training costly conditional generative models, the method trains a lightweight discriminator to distinguish real noisy pairs from independently generated fake pairs.The discriminator is trained on the joint distribution versus the product of marginal distributions and outputs one for real pairs and zero for fake pairs.Source: §3.2 · p. 5
• The optimal discriminator gradient approximates the conditional-score terms needed to convert the two marginal scores into a joint score.The approximation uses gradients of log Dθ/(1−Dθ) with respect to each modality’s noisy sample.Source: §3.2 · p. 5
• During inference, MMDisCo sums base-model noise estimates with discriminator guidance at every timestep before each joint denoising step.The method can also be applied when base-model marginals do not exactly equal the joint-data marginals, and this setting was tested experimentally.Source: §3.2 · pp. 5–6

3.3 RESIDUAL SCORE ESTIMATION FROM AN OPTIMAL DISCRIMINATOR

• Denoising regularization addresses fidelity degradation caused by score-estimation mismatch when discriminator guidance is trained alone.The regularizer matches discriminator-gradient guidance to residuals between true noises and base-model noise predictions, while keeping base-model parameters fixed.Source: §3.3 · p. 6
• The final discriminator objective combines the discriminator loss with denoising regularization using λ as the balancing weight, set to 1 throughout the paper.Algorithms 1 and 2 summarize discriminator training and joint generation.Source: §3.3 · p. 6

4.1 PRELIMINARY EXPERIMENTS WITH TOY DATASETS

• The method successfully guides independently trained base models toward target joint distributions in both in-domain and out-of-domain toy settings, with both Ldisc and Ldenoise improving NLL.The toy experiments use duplicated scalar diffusion models to generate two-dimensional samples and evaluate negative log-likelihood against target distributions.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 EXPERIMENTS WITH BENCHMARK DATASETS UNDER THE IN-DOMAIN SETTING

• In the in-domain benchmark setting, guidance improves both FVD and FAD on Landscape and AIST++, while marginally improving IB-AV alignment scores.The experiments adapt MM-Diffusion outputs using a 12.7M-parameter discriminator while freezing the 133M-parameter base model.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 EXPERIMENTS WITH BENCHMARK DATASETS UNDER THE OUT-OF-DOMAIN SETTING

• In the out-of-domain setting, guidance consistently improves IB-AV and generally improves audio fidelity, although IB-TV and FVD degrade for VGGSound.Compared with sequential T2V followed by V2A generation, the method generally performs better on audio fidelity and IB-AV.Source: §4.3 · pp. 9–10, Table 3 · p. 9, Table 4 · p. 9
• Qualitative VGGSound results show that guidance tends to produce players whose generated audio is temporally aligned with their motion, unlike independently generated samples.The comparison uses the same captions and random seed for AnimateDiff/AudioLDM and the proposed method.Source: §4.3 · p. 10, Figure 3 · p. 10

5 CONCLUSION

• The guidance module modifies scores from two pretrained diffusion models to match the joint audio-video distribution using an optimal-discriminator gradient.A denoising objective regularizes the discriminator gradient to provide a stable generation signal.Source: §5 · p. 10
• Across benchmark datasets, the method improves alignment and single-modal fidelity without requiring a large parameter increase over the base models.The approach is designed to integrate pretrained modality-specific models without backpropagation through them.Source: §5 · p. 10
• Generated quality strongly depends on the pretrained base models, although replacing them with improved modality-specific models can extend the joint generator without huge computational cost.This dependence is the paper’s stated limitation.Source: §5 · p. 10

A.1 PROOF OF DISCRIMINATOR GUIDANCE

• The discriminator loss separates into timestep-wise classification of real paired samples versus fake samples from the independently trained base models.The real distribution is q(x_t,y_t), while the fake distribution factorizes into modality-specific base-model distributions.Source: §A.1 · p. 15
• At its optimum, the discriminator estimates the density ratio between the real joint distribution and the independently generated fake distribution.The ratio is represented by Dθ*(x_t,y_t,t)/(1−Dθ*(x_t,y_t,t)).Source: §A.1 · p. 15
• Assuming the base models match the real marginal distributions, gradients of the conditional distributions can be computed from the joint-distribution ratio and used in the guidance equations.The assumption is q(x_t)=pϕ(x_t) and q(y_t)=pψ(y_t).Source: §A.1 · p. 15

• The derivation extends equations previously obtained under matching marginal assumptions to cases where q(x_t) and q(y_t) differ from the base-model marginals.Source: §A.2 · pp. 15–16
• For audio, the joint score is approximated by the audio base-model score plus a discriminator-gradient correction.Source: §A.2 · p. 16
• For video, the joint score is analogously approximated by the video base-model score plus the discriminator-gradient correction.Source: §A.2 · p. 16
• The optimal discriminator can guide OOD generation when trained to distinguish real paired samples from fake pairs generated independently by the base models.Source: §A.2 · p. 16

Experiments with toy dataset

• Toy experiments compute two-dimensional residual noise predictions from discriminator input gradients and sample 4000 examples for evaluation and visualization.Source: §A.3 · p. 16

Experiments with benchmark datasets

• The benchmark discriminator uses two-stream residual blocks followed by a linear layer, with attention removed after preliminary tests found attention destabilized training.Source: §A.3 · p. 16
• Benchmark evaluation generated 2048 samples and computed FVD, FAD, and IB-score using the corresponding released evaluation implementations.Source: §A.3 · p. 16
• During inference, discriminator gradients are added without scaling except for Auffusion and VideoCrafter2, where a grid search selected video-latent scale 8.Source: §A.3 · p. 16

• Training used 16GB V100 × 4 GPUs for in-domain experiments and 40GB A100 × 8 GPUs for out-of-domain experiments, with runtimes ranging from 1.5 to 30 hours.Source: §A.3 · p. 17

• The original Landscape labels were too ambiguous for the base models, so InstructBLIP generated short captions from each video’s first frame for the OOD experiment.Source: §A.4 · p. 17
• The generated captions omit audio information, and the resulting text condition may reduce audio-generation suitability, as reflected by IB-TA performing worse than IB-TV.Source: §A.4 · p. 17
• The authors identify designing and manipulating input conditions as future work for joint generation.Source: §A.4 · p. 17

• The visualization places IND samples in the top row and OOD samples in the bottom row, with target samples at the left and loss-function variants in the remaining columns.Source: §A.5 · p. 17, Figure 4 · p. 19
• In the OOD setting, combining Ldisc and Ldenoise yields the best visual result and substantially improves concentration at a single peak.Source: §A.5 · pp. 17, 19, Figure 4 · p. 19
• Using only Ldisc roughly concentrates OOD samples at four corners but struggles to force them into one peak.Source: §A.5 · p. 19
• In the IND setting, Ldisc and Ldenoise produce similar results.Source: §A.5 · p. 19

A.6 ABLATION STUDY ABOUT THE LOSS FUNCTIONS ON REAL DATA

• Lall achieves both better fidelity scores and cross-modal alignment than Ldisc and Ldenoise on the Landscape dataset.Ldisc favors alignment, whereas Ldenoise favors fidelity; Lall provides the best of both worlds.Source: §A.6 · p. 19

A.7 VALIDATION OF THE PARAMETER SIZE AND THE TRAINING EPOCH

• Less than 1% additional parameters still improves the base model’s performance, while C = 128 is the most appropriate setting for multimodal guidance.Increasing channel size generally improves fidelity and alignment, but alignment slightly degrades at C = 256; C = 128 requires less than 10% additional parameters.Source: §A.7 · p. 20, Table 8 · p. 20
• Increasing L mainly improves FAD, likely because dilated convolutions expand the audio encoder’s receptive field and capacity.Other metrics change only marginally as the number of ResBlocks increases.Source: §A.7 · p. 20, Table 9 · p. 20
• The guidance module converges around 100 training epochs, with no clear improvement from training longer.This result uses C = 128, L = 2, five random seeds, and reports metric averages and standard deviations.Source: §A.7 · p. 20, Figure 5 · p. 21

A.8 DISCRIMINATOR STRUCTURE VARIANTS

• Transformer-based feature fusion improves fidelity and crossmodal alignment, whereas replacing ResBlocks with vision transformers does not improve performance.The tested fusion variants are Naive Transformer and Frieren; further discriminator-structure exploration remains future work.Source: §A.8 · pp. 20–21, Table 10 · p. 21

A.9 COMPUTATIONAL COST FOR THE TRAINING AND INFERENCE

• Training is approximately 1.5 to 2 times faster than the baseline, while inference takes around 10 to 20% longer because of discriminator computation.The training baseline estimates fine-tuning base models without additional learnable or cross-modal layers.Source: §A.9 · p. 22, Table 11 · p. 22, Table 12 · p. 22

A.10.2 EVALUATION ON THE TEMPORAL AND SEMANTIC ALIGNMENT

• MMDisCo improves both AV-Align and IB-AV for both base models, with sophisticated bases reaching IB-AV substantially closer to GT and AV-Align comparable to GT.AudioLDM / AnimateDiff plus the default discriminator achieves the best AV-Align score among the compared models.Source: §A.10.2 · p. 23, Table 13 · p. 23

A.11 SUBJECTIVE EVALUATION

• Human evaluations consistently favor the proposed method across video quality, audio quality, semantic alignment, and temporal alignment.The study collected 320 responses from 8 evaluators across 10 randomly sampled VGGSound labels.Source: §A.11 · p. 24, Table 14 · p. 24