PaperlayerView exact revision on arXiv

Source-linked AI summary

UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

Guozhen Zhang, Zixiang Zhou, Teng Hu, Ziqiao Peng, Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, Limin Wang

arXiv:2511.03334v2cs.CV

TL;DR

Existing audio-video generation methods lack effective cross-modal modeling, limiting semantic consistency and synchronization. UniAVGen addresses this with unified dual-branch synthesis, asymmetric cross-modal interaction, face-aware modulation, and modality-aware guidance, supporting multiple generation tasks and stronger audio-video consistency.

  • Problem

    Existing methods rely on decoupled pipelines or lack robust human-centric audio-visual alignment and fine-grained temporal synchronization.

    Source: §1 · p. 2

  • Method

    UniAVGen uses parallel video and audio Diffusion Transformer branches with asymmetric cross-modal interaction, face-aware modulation, and modality-aware classifier-free guidance.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    UniAVGen demonstrates advantages in audio-video synchronization, timbre consistency, and emotion consistency, while improving emotional intensity and video motion dynamics.

    Source: §1 · p. 2, §5 · p. 8, §4.3.1 · p. 8

  • Takeaways & Limitations

    UniAVGen unifies joint generation, video-to-audio dubbing, and audio-driven video synthesis within one framework, without task-specific model designs.

    Source: §1 · p. 2

    UniAVGen lacks video-aligned ambient sound generation and remains constrained in multi-person audio generation by its text encoder.

    Source: §11 · p. 13

Abstract

from arXiv · show

Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.

1. Introduction

Existing open-source audio-video generators often decouple modalities, limiting semantic and emotional alignment, while joint methods remain weak on human speech and fine-grained synchronization. UniAVGen addresses these gaps with unified cross-modal synthesis, targeted interaction modules, and multi-task adaptability.

Motivation

  • Existing open-source approaches commonly use sequential audio-video pipelines, which impede cross-modal interplay and yield inadequate semantic consistency and emotional alignment.Other end-to-end methods are described as limited by ambient-sound-only generation, weak alignment, or insufficient fine-grained temporal synchronization.Source: §1 · p. 2

Architecture

  • UniAVGen uses two parallel Diffusion Transformers in a symmetric dual-branch architecture to establish representational parity and a cohesive latent space for joint audio-video generation.The video and audio streams use identical architectural designs, with modal-specific backbone configurations.Source: §1 · p. 2

Cross-modal interaction

  • Asymmetric Cross-Modal Interaction provides bidirectional, temporally aligned attention, transferring audio semantics to video and video dynamics and identity details to audio.The mechanism uses audio-to-video and video-to-audio aligners for modality-specific information exchange.Source: §1 · p. 2

Alignment modules

  • Face-Aware Modulation focuses cross-modal interaction on salient human-related regions, while Modality-Aware Classifier-Free Guidance amplifies cross-modal signals during inference.The guidance strategy is reported to enhance emotional intensity in audio and motion dynamics in video.Source: §1 · p. 2

Unified tasks

  • UniAVGen unifies joint generation, video-to-audio dubbing, and audio-driven video synthesis within one framework, avoiding task-specific model designs.The framework also supports joint continuation and multi-task training to improve joint-generation performance.Source: §1 · p. 2

3. Method

UniAVGen combines parallel audio and video DiT branches with asymmetric, temporally aligned cross-modal interaction, face-aware modulation, and modality-aware guidance. The framework supports joint generation, continuation, dubbing, and audio-driven video synthesis through multiple input configurations.

Dual-branch joint synthesis

  • UniAVGen uses parallel video and audio Diffusion Transformer branches as a symmetric dual-branch joint synthesis framework.The video branch adopts Wan 2.2-5B, while the audio branch follows the Wan 2.1-1.3B architectural template with different channel counts.Source: §3.1 · p. 3

Asymmetric Cross-Modal Interaction

  • Asymmetric Cross-Modal Interaction uses modality-specific bidirectional aligners to exchange temporally aligned audio and video information.The A2V aligner injects fine-grained audio cues into video, while the V2A aligner transfers visual semantics such as timbre and emotion into audio.Source: §3.2 · p. 4

Audio-to-video alignment

  • The A2V aligner gives each video frame a contextual audio window, enabling frame-wise cross-attention over neighboring audio tokens.Neighboring frames are included to account for preceding and succeeding phonemes, with boundary features replicated at sequence ends.Source: §3.2 · p. 4

Video-to-audio alignment

  • The V2A aligner interpolates between adjacent video latents so each audio token receives more precise temporal visual context before cross-attention.For an audio token mapped to video latent i, the context is formed from video latents i and i+1 using an interpolation weight.Source: §3.2 · pp. 4–5

Face-aware modulation

  • Face-Aware Modulation steers cross-modal interaction toward salient facial regions, while zero-initialized output matrices protect each modality’s initial generative capability.The module uses an auxiliary mask-prediction head and mask supervision to constrain interaction without disrupting backgrounds during early training.Source: §3.3 · p. 5, §3.2 · p. 5

Modality-aware guidance

  • Modality-Aware Classifier-Free Guidance strengthens cross-modal conditioning by using a shared unconditional estimate for both modalities.The method nullifies both cross-modal conditioning signals in one forward pass and contrasts this estimate with the cross-modal estimate.Source: §3.4 · p. 5

Multi-task unification

  • UniAVGen unifies joint generation, reference-audio control, audio-video continuation, video-to-audio dubbing, and audio-driven video synthesis within one framework.Reference and conditional latents are selectively kept unaffected or included in cross-modal interaction to preserve timbre and temporal continuity.Source: §3.5 · pp. 5–6

4. Experiment

UniAVGen is evaluated against representative methods and through ablations of interaction design, face-aware modulation, classifier-free guidance, and training strategy. Results show benefits from temporally aligned asymmetric interaction, supervised and decaying face-aware modulation, MA-CFG, and multi-task training.

4.1. Experimental setup

  • The evaluation uses 100 newly constructed test samples and measures audio quality, video quality, and audio-video consistency.The compared methods include two-stage generation approaches and recent joint-generation models; the supplied evaluation passage specifies the sample construction and metric dimensions.Source: §4.2 · pp. 6–7

4.3.1. Cross-modal interaction design

  • ATI improves both A2V timbre and emotion consistency and V2A lip synchronization over STI.Compared with symmetric temporal-aligned interaction, ATI strengthens audio perception of facial expressions and movements while improving how video captures neighboring audio cues.Source: §4.3.1 · p. 8

4.3.2. Effectiveness of face-aware modulation

  • Supervised FAM improves overall audio-video consistency, while decaying mask-loss weights further enhance timbre and emotion consistency.The ablation compares no FAM, unsupervised FAM, fixed-weight FAM, and decaying-weight FAM.Source: §4.3.2 · p. 8, Table 3 · p. 8

4.3.3. Effectiveness of MA-CFG

  • MA-CFG produces facial expressions, body movements, and lip synchronization that more closely follow audio emotions than generation without it.Without MA-CFG, audio-video consistency remains general but emotional and motion alignment is insufficient.Source: §4.3.3 · p. 8, Figure 6 · p. 8

4.3.4. Training strategy

  • Multi-task learning raises the performance ceiling over joint-generation-only training, although training all tasks throughout converges more slowly.The comparison includes JGO, JFML, and MTO strategies.Source: §4.3.4 · p. 8, Figure 7 · p. 8

5. Conclusion

UniAVGen combines asymmetric temporal alignment, face-aware modulation, and modality-aware guidance for joint audio-video generation. The framework is presented as supporting high-quality, practical, and versatile multimodal generation.

5. Conclusion

  • UniAVGen combines ATI, FAM, and MA-CFG to strengthen cross-modal interaction and audio-video generation.ATI provides modality-specific temporal alignment, FAM localizes facial regions, and MA-CFG strengthens cross-modal influences during inference.Source: §5 · p. 8
  • The framework is presented as a practical and versatile basis for audio-video generation systems.The conclusion characterizes UniAVGen as setting a new benchmark and supporting more practical multimodal generation systems.Source: §5 · p. 8

6. Additional implementation details

The implementation details explicitly align audio and video contexts despite video-latent frame grouping, while using Euler sampling, Vocos decoding, and modality-specific guidance settings.

6.2. Temporal alignment in interaction

  • Audio-to-video alignment uses a two-segment audio context around each video latent to provide phoneme information for lip synchronization.Boundary contexts are padded by replicating the first or last audio-frame features.Source: §6.1 · p. 12
  • Video latents after the first represent four consecutive frames, so temporal alignment accounts for unequal latent-to-frame correspondence.For audio-to-video alignment, audio windows are computed against actual video-frame counts; for video-to-audio alignment, latents are replicated to audio resolution.Source: §6.2 · p. 12
  • Video-to-audio alignment first upsamples video latents by replicating each noninitial latent four times before computing the video context.This matches the video representation to audio’s finer temporal resolution.Source: §6.2 · p. 12

6.3. Inference details

  • Inference uses 50 Euler ODE sampling steps, Vocos waveform decoding, modality-aware guidance scales s_v = 3 and s_a = 2, and high-frequency-only CFG.The CFG interval is [0.5, 1].Source: §6.3 · p. 12

7. System prompt for evaluation

Evaluation prompts separately score timbre and emotion consistency on a 0–1 scale, with repeated independent judgments averaged for each sample.

7. System prompt for evaluation

  • The evaluation prompt asks an expert audio-video judge to score timbre and emotion consistency between an audio and video clip.Scores are returned on a 0–1 scale with reasons limited to 100 words.Source: §7 · p. 12
  • Timbre consistency is graded by gender and age matching, from complete inconsistency at 0 to perfect matching at 1.The five grades use 0.25-point increments.Source: §7 · p. 12
  • Emotion consistency is graded by frame-level emotional matching and body-language correspondence, from no correspondence at 0 to perfect correspondence at 1.The rubric likewise uses five levels in 0.25-point increments.Source: §7 · p. 12
  • Each sample receives three independent evaluations, and the reported score is their average.The evaluator returns separate timbre and emotion scores with a brief reason.Source: §7 · p. 12

8. User study

A 34-participant user study compared generated videos on audio quality, video quality, and overall audio-visual coherence, with results reported in Table 4.

8. User study

  • 34 participants evaluated and selected top-generated videos using audio quality, video quality, and overall audio-visual coherence.The study reports results in Table 4.Source: §8 · pp. 12–13
  • UniAVGen achieved superior overall audio-visual quality and enhanced audio-video consistency compared with recent methods.This is the reported conclusion of the user-study comparison.Source: §8 · p. 13

9. Evaluation on conditional tasks

After multi-task joint training, UniAVGen was evaluated on video-to-audio dubbing and audio-to-video synthesis benchmarks, showing strong consistency and generalization without task-specific designs.

Video-to-audio dubbing

  • On GRID dubbing under Setting 3.0 with unseen reference speakers, UniAVGen achieved superior consistency and lower WER without complex task-specific designs.The comparison uses LSE-C, LSE-D, and WER, with results reported in Table 5.Source: §9 · p. 13

Audio-to-video synthesis

  • On the EMTD half-body animation benchmark, UniAVGen attained near-SOTA performance after only simple multi-task fine-tuning.The comparison is reported in Table 6 against state-of-the-art audio-driven models.Source: §9 · p. 13

Cross-task evaluation

  • These conditional-task results support UniAVGen’s practicality and generalization capability after multi-task joint training.The model is evaluated beyond its primary joint audio-visual generation setting.Source: §9 · p. 13

10. Extended ablation studies

The extended ablations examine where cross-modal interaction should be inserted and how classifier-free guidance affects image quality and audio-visual alignment. Interleaved layer insertion, pre-self-attention placement, and constrained MA-CFG provide the strongest reported configurations.

10.1. Rational integration of the interaction module

  • Interleaved cross-modal interaction across layers yields the best results among the four tested layer-level insertion schemes.The comparison includes insertion into all layers, the first half, the last half, and interleaved layers.Source: §10.1 · p. 13, Table 7 · p. 13
  • Inserting the interaction module before self-attention achieves the best operation-level performance among the three tested positions.The alternatives place the module before cross-attention or before the FFN.Source: §10.1 · p. 13, Table 8 · p. 13

10.2. Validation of MA-CFG’s effectiveness

  • Constrained MA-CFG improves both image quality and modal alignment, unlike vanilla CFG, whose alignment gains are negligible.MA-CFG alone significantly improves audio-visual alignment but slightly degrades image quality; restricting its interval to [0.5, 1] avoids that trade-off.Source: §10.2 · p. 13, Table 9 · p. 13

11. Limitations

UniAVGen’s current scope is strongest for speech-video generation but does not cover video-aligned ambient sounds. Multi-person audio generation is also constrained by the text encoder.

11. Limitations

  • UniAVGen lacks video-aligned ambient sound generation despite performing well in speech-video generation.The authors identify broader high-quality audio-video data collection as a future direction.Source: §11 · p. 13
  • Audio generation for multi-person scenarios remains constrained by the inflexible text encoder.The authors plan to improve the audio-branch text encoder with multimodal large language models.Source: §11 · p. 13

Related papers

Loading 2511.03334v2…