PaperlayerView exact revision on arXiv

Source-linked AI summary

UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

Guozhen Zhang, Zixiang Zhou, Teng Hu, Ziqiao Peng, Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, Limin Wang

arXiv:2511.03334v2cs.CV

TL;DR

Existing audio-video generation methods lack effective cross-modal modeling, limiting semantic consistency and synchronization. UniAVGen addresses this with unified dual-branch synthesis, asymmetric cross-modal interaction, face-aware modulation, and modality-aware guidance, supporting multiple generation tasks and stronger audio-video consistency.

  • Problem

    Existing methods rely on decoupled pipelines or lack robust human-centric audio-visual alignment and fine-grained temporal synchronization.

  • Method

    UniAVGen uses parallel video and audio Diffusion Transformer branches with asymmetric cross-modal interaction, face-aware modulation, and modality-aware classifier-free guidance.

  • Results

    UniAVGen demonstrates advantages in audio-video synchronization, timbre consistency, and emotion consistency, while improving emotional intensity and video motion dynamics.

  • Takeaways & Limitations

    UniAVGen unifies joint generation, video-to-audio dubbing, and audio-driven video synthesis within one framework, without task-specific model designs.

  • Takeaways & Limitations

    UniAVGen lacks video-aligned ambient sound generation and remains constrained in multi-person audio generation by its text encoder.

Abstract

from arXiv · show

Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.

1. Introduction

Existing open-source audio-video generators often decouple modalities, limiting semantic and emotional alignment, while joint methods remain weak on human speech and fine-grained synchronization. UniAVGen addresses these gaps with unified cross-modal synthesis, targeted interaction modules, and multi-task adaptability.

  • Motivation: Existing open-source approaches commonly use sequential audio-video pipelines, which impede cross-modal interplay and yield inadequate semantic consistency and emotional alignment.Other end-to-end methods are described as limited by ambient-sound-only generation, weak alignment, or insufficient fine-grained temporal synchronization.
  • Architecture: UniAVGen uses two parallel Diffusion Transformers in a symmetric dual-branch architecture to establish representational parity and a cohesive latent space for joint audio-video generation.The video and audio streams use identical architectural designs, with modal-specific backbone configurations.
  • Cross-modal interaction: Asymmetric Cross-Modal Interaction provides bidirectional, temporally aligned attention, transferring audio semantics to video and video dynamics and identity details to audio.The mechanism uses audio-to-video and video-to-audio aligners for modality-specific information exchange.
  • Alignment modules: Face-Aware Modulation focuses cross-modal interaction on salient human-related regions, while Modality-Aware Classifier-Free Guidance amplifies cross-modal signals during inference.The guidance strategy is reported to enhance emotional intensity in audio and motion dynamics in video.
  • Unified tasks: UniAVGen unifies joint generation, video-to-audio dubbing, and audio-driven video synthesis within one framework, avoiding task-specific model designs.The framework also supports joint continuation and multi-task training to improve joint-generation performance.

3. Method

UniAVGen combines parallel audio and video DiT branches with asymmetric, temporally aligned cross-modal interaction, face-aware modulation, and modality-aware guidance. The framework supports joint generation, continuation, dubbing, and audio-driven video synthesis through multiple input configurations.

  • Dual-branch joint synthesis: UniAVGen uses parallel video and audio Diffusion Transformer branches as a symmetric dual-branch joint synthesis framework.The video branch adopts Wan 2.2-5B, while the audio branch follows the Wan 2.1-1.3B architectural template with different channel counts.
  • Asymmetric Cross-Modal Interaction: Asymmetric Cross-Modal Interaction uses modality-specific bidirectional aligners to exchange temporally aligned audio and video information.The A2V aligner injects fine-grained audio cues into video, while the V2A aligner transfers visual semantics such as timbre and emotion into audio.
  • Audio-to-video alignment: The A2V aligner gives each video frame a contextual audio window, enabling frame-wise cross-attention over neighboring audio tokens.Neighboring frames are included to account for preceding and succeeding phonemes, with boundary features replicated at sequence ends.
  • Video-to-audio alignment: The V2A aligner interpolates between adjacent video latents so each audio token receives more precise temporal visual context before cross-attention.For an audio token mapped to video latent i, the context is formed from video latents i and i+1 using an interpolation weight.
  • Face-aware modulation: Face-Aware Modulation steers cross-modal interaction toward salient facial regions, while zero-initialized output matrices protect each modality’s initial generative capability.The module uses an auxiliary mask-prediction head and mask supervision to constrain interaction without disrupting backgrounds during early training.
  • Modality-aware guidance: Modality-Aware Classifier-Free Guidance strengthens cross-modal conditioning by using a shared unconditional estimate for both modalities.The method nullifies both cross-modal conditioning signals in one forward pass and contrasts this estimate with the cross-modal estimate.
  • Multi-task unification: UniAVGen unifies joint generation, reference-audio control, audio-video continuation, video-to-audio dubbing, and audio-driven video synthesis within one framework.Reference and conditional latents are selectively kept unaffected or included in cross-modal interaction to preserve timbre and temporal continuity.

4. Experiment

UniAVGen is evaluated against representative methods and through ablations of interaction design, face-aware modulation, classifier-free guidance, and training strategy. Results show benefits from temporally aligned asymmetric interaction, supervised and decaying face-aware modulation, MA-CFG, and multi-task training.

  • 4.1. Experimental setup: The evaluation uses 100 newly constructed test samples and measures audio quality, video quality, and audio-video consistency.The compared methods include two-stage generation approaches and recent joint-generation models; the supplied evaluation passage specifies the sample construction and metric dimensions.
  • 4.3.1. Cross-modal interaction design: ATI improves both A2V timbre and emotion consistency and V2A lip synchronization over STI.Compared with symmetric temporal-aligned interaction, ATI strengthens audio perception of facial expressions and movements while improving how video captures neighboring audio cues.
  • 4.3.2. Effectiveness of face-aware modulation: Supervised FAM improves overall audio-video consistency, while decaying mask-loss weights further enhance timbre and emotion consistency.The ablation compares no FAM, unsupervised FAM, fixed-weight FAM, and decaying-weight FAM.
  • 4.3.3. Effectiveness of MA-CFG: MA-CFG produces facial expressions, body movements, and lip synchronization that more closely follow audio emotions than generation without it.Without MA-CFG, audio-video consistency remains general but emotional and motion alignment is insufficient.
  • 4.3.4. Training strategy: Multi-task learning raises the performance ceiling over joint-generation-only training, although training all tasks throughout converges more slowly.The comparison includes JGO, JFML, and MTO strategies.

5. Conclusion

UniAVGen combines asymmetric temporal alignment, face-aware modulation, and modality-aware guidance for joint audio-video generation. The framework is presented as supporting high-quality, practical, and versatile multimodal generation.

  • 5. Conclusion: UniAVGen combines ATI, FAM, and MA-CFG to strengthen cross-modal interaction and audio-video generation.ATI provides modality-specific temporal alignment, FAM localizes facial regions, and MA-CFG strengthens cross-modal influences during inference.
  • 5. Conclusion: The framework is presented as a practical and versatile basis for audio-video generation systems.The conclusion characterizes UniAVGen as setting a new benchmark and supporting more practical multimodal generation systems.

6. Additional implementation details

The implementation details explicitly align audio and video contexts despite video-latent frame grouping, while using Euler sampling, Vocos decoding, and modality-specific guidance settings.

  • 6.2. Temporal alignment in interaction: Audio-to-video alignment uses a two-segment audio context around each video latent to provide phoneme information for lip synchronization.Boundary contexts are padded by replicating the first or last audio-frame features.
  • 6.2. Temporal alignment in interaction: Video latents after the first represent four consecutive frames, so temporal alignment accounts for unequal latent-to-frame correspondence.For audio-to-video alignment, audio windows are computed against actual video-frame counts; for video-to-audio alignment, latents are replicated to audio resolution.
  • 6.2. Temporal alignment in interaction: Video-to-audio alignment first upsamples video latents by replicating each noninitial latent four times before computing the video context.This matches the video representation to audio’s finer temporal resolution.
  • 6.3. Inference details: Inference uses 50 Euler ODE sampling steps, Vocos waveform decoding, modality-aware guidance scales s_v = 3 and s_a = 2, and high-frequency-only CFG.The CFG interval is [0.5, 1].

7. System prompt for evaluation

Evaluation prompts separately score timbre and emotion consistency on a 0–1 scale, with repeated independent judgments averaged for each sample.

  • 7. System prompt for evaluation: The evaluation prompt asks an expert audio-video judge to score timbre and emotion consistency between an audio and video clip.Scores are returned on a 0–1 scale with reasons limited to 100 words.
  • 7. System prompt for evaluation: Timbre consistency is graded by gender and age matching, from complete inconsistency at 0 to perfect matching at 1.The five grades use 0.25-point increments.
  • 7. System prompt for evaluation: Emotion consistency is graded by frame-level emotional matching and body-language correspondence, from no correspondence at 0 to perfect correspondence at 1.The rubric likewise uses five levels in 0.25-point increments.
  • 7. System prompt for evaluation: Each sample receives three independent evaluations, and the reported score is their average.The evaluator returns separate timbre and emotion scores with a brief reason.

8. User study

A 34-participant user study compared generated videos on audio quality, video quality, and overall audio-visual coherence, with results reported in Table 4.

  • 8. User study: 34 participants evaluated and selected top-generated videos using audio quality, video quality, and overall audio-visual coherence.The study reports results in Table 4.
  • 8. User study: UniAVGen achieved superior overall audio-visual quality and enhanced audio-video consistency compared with recent methods.This is the reported conclusion of the user-study comparison.

9. Evaluation on conditional tasks

After multi-task joint training, UniAVGen was evaluated on video-to-audio dubbing and audio-to-video synthesis benchmarks, showing strong consistency and generalization without task-specific designs.

  • Video-to-audio dubbing: On GRID dubbing under Setting 3.0 with unseen reference speakers, UniAVGen achieved superior consistency and lower WER without complex task-specific designs.The comparison uses LSE-C, LSE-D, and WER, with results reported in Table 5.
  • Audio-to-video synthesis: On the EMTD half-body animation benchmark, UniAVGen attained near-SOTA performance after only simple multi-task fine-tuning.The comparison is reported in Table 6 against state-of-the-art audio-driven models.
  • Cross-task evaluation: These conditional-task results support UniAVGen’s practicality and generalization capability after multi-task joint training.The model is evaluated beyond its primary joint audio-visual generation setting.

10. Extended ablation studies

The extended ablations examine where cross-modal interaction should be inserted and how classifier-free guidance affects image quality and audio-visual alignment. Interleaved layer insertion, pre-self-attention placement, and constrained MA-CFG provide the strongest reported configurations.

  • 10.1. Rational integration of the interaction module: Interleaved cross-modal interaction across layers yields the best results among the four tested layer-level insertion schemes.The comparison includes insertion into all layers, the first half, the last half, and interleaved layers.
  • 10.1. Rational integration of the interaction module: Inserting the interaction module before self-attention achieves the best operation-level performance among the three tested positions.The alternatives place the module before cross-attention or before the FFN.
  • 10.2. Validation of MA-CFG’s effectiveness: Constrained MA-CFG improves both image quality and modal alignment, unlike vanilla CFG, whose alignment gains are negligible.MA-CFG alone significantly improves audio-visual alignment but slightly degrades image quality; restricting its interval to [0.5, 1] avoids that trade-off.

11. Limitations

UniAVGen’s current scope is strongest for speech-video generation but does not cover video-aligned ambient sounds. Multi-person audio generation is also constrained by the text encoder.

  • 11. Limitations: UniAVGen lacks video-aligned ambient sound generation despite performing well in speech-video generation.The authors identify broader high-quality audio-video data collection as a future direction.
  • 11. Limitations: Audio generation for multi-person scenarios remains constrained by the inflexible text encoder.The authors plan to improve the audio-branch text encoder with multimodal large language models.
Loading 2511.03334v2…