Native Audio-Visual Alignment for Generation

Longbin Ji; Guan Wang; Xuan Wei; Chenye Yang; Xiangrui Liu; Zhenyu Zhang; Shuohuan Wang; Yu Sun; Jingzhou He

Source-linked AI summary

Native Audio-Visual Alignment for Generation

Longbin Ji, Guan Wang, Xuan Wei, Chenye Yang, Xiangrui Liu, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Jingzhou He

arXiv:2605.30073v1May 28, 2026cs.CV

TL;DR

Existing joint audio-video generators struggle to model fine-grained synchronization while conditioning on semantic context. NAVA separates native audio-video alignment from context conditioning and outperforms representative baselines across synchronization, semantic consistency, visual quality, and timbre controllability.

Problem
Existing open-source methods provide limited fine-grained audio-video co-evolution because correspondence is established through late-stage interaction or mixed with contextual conditioning.
Method
NAVA aligns audio and video in a dedicated interaction space before externally conditioning joint denoising, using Align-then-Fuse MMDiT and Timbre-in-Context Conditioning.
Results
NAVA significantly outperforms representative baselines in audio-visual synchronization, semantic consistency, visual quality, and timbre controllability, using 6.3B parameters.
Takeaways & Limitations
Experiments indicate that native audio-visual alignment with decoupled context conditioning is a promising direction for joint audio-video generation.
Takeaways & Limitations
NAVA remains limited on long-tail and highly compositional audio events, including rare animal sounds, music, singing, and complex scene-sound mixtures.

Abstract

from arXiv · show

Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.

1 Introduction

NAVA frames joint audio-video generation as context-conditioned native audio-visual alignment, separating dedicated audio-video correspondence from external contextual conditioning. Its Align-then-Fuse MMDiT architecture and Timbre-in-Context Conditioning support modality-aware collaborative denoising and flexible content-timbre binding.

Introduction: Joint audio-video generation models temporal and semantic correspondences within one process, reducing error propagation and improving cross-modal coherence.This contrasts with cascaded pipelines that synthesize modalities sequentially.
Introduction: Dual-tower methods generate audio and video separately, introducing cross-modal interaction only through additional late-stage alignment modules.Posterior alignment weakens the joint evolution of audio and video.
Introduction: Fully unified tri-modal methods directly interact across context, video, and audio, but couple semantic control with low-level audio-visual synchronization.Semantic guidance, event correspondence, and temporal alignment are optimized in one representation space.
Introduction: NAVA first establishes audio-video correspondence in a dedicated alignment space, then uses external context to guide the aligned representation.This decouples context conditioning from native audio-video alignment.
Introduction: NAVA’s Align-then-Fuse MMDiT uses modality-aware layers for heterogeneous audio-video alignment and shared fusion layers for collaborative denoising.The architecture transitions from modality-aware alignment to modality-shared denoising.
Introduction: Timbre-in-Context Conditioning treats timbre cues as contextual conditions for specific speech spans, enabling flexible content-timbre binding without auxiliary speaker-control branches.The mechanism binds timbre cues to corresponding speech segments.

2 Method

NAVA separates native audio-video synchronization from external semantic and timbre conditioning. Its Align-then-Fuse MMDiT architecture aligns modalities before shared denoising, while Timbre-in-Context Conditioning binds reference timbres to speech spans.

Align-then-Fuse MMDiT: NAVA instantiates this design with Align-then-Fuse MMDiT: Hierarchical Alignment Layers establish correspondence before Unified Fusion Layers perform modality-shared denoising.Early modality-aware projections stabilize heterogeneous interaction, while later shared transformer blocks encourage co-evolution after alignment.
Motivation: Existing dual-tower methods establish audio-video correspondence only through late-stage interaction, making fine-grained synchronization dependent on posterior alignment.Each modality evolves largely in its own feature space before cross-modal correspondence is explicitly introduced.
Context-Conditioned Native Alignment: NAVA first applies joint self-attention to audio-video tokens in a dedicated synchronization space, then injects external context through cross-attention.This separates native correspondence learning from semantic and controllable guidance without treating context tokens as self-attention peers.
Timbre-in-Context Conditioning: Timbre-in-Context Conditioning represents reference timbre cues as context tokens and binds them to corresponding speech spans for controllable multi-speaker generation.The mechanism uses the existing context-guided cross-attention pathway rather than an auxiliary speaker-control branch or global timbre signal.
Training and Inference: NAVA uses progressive multi-task training and factorized inference guidance to support audio-video alignment, prompt adherence, and reference-timbre preservation.Training covers T2AV, TI2AV, T2A, T2V, and TIA2AV tasks, while inference separately controls textual, alignment, and timbre guidance.

3 Experiments

Experiments show that NAVA delivers strong audio-visual synchronization, video and audio quality, reference-timbre controllability, and perceptual performance across objective benchmarks, visualizations, and user studies. Ablations further validate explicit alignment and condition-factorized guidance.

Verse-Bench Results: With 6.3B parameters, NAVA achieves the highest Sync-C of 7.791, lowest Sync-D of 7.566, and best video quality score of 0.659 on Verse-Bench.These results support the Align-then-Fuse design’s ability to preserve visual generation while synchronizing audio and visual motion.
Verse-Bench Results: NAVA obtains an IB-Score of 0.313, lowest WER of 0.099, PQ of 6.861, and FD of 0.833, combining semantic consistency with competitive audio fidelity.LTX 2.3 achieves the highest IB-Score, while NAVA outperforms Ovi-1.1 and remains competitive with MoVA and Davinci.
Reference-Timbre Evaluation: On Seed-TTS, NAVA achieves the highest speaker similarity of 66.7 among audio-video models and a competitive WER of 4.20.NAVA substantially outperforms DreamID-Omni within the audio-video model category.
User Study: In T2AV user studies, NAVA’s overall-quality win rates reach 67.5%, 60.0%, and 80.0%, while alignment win rates reach 62.5%, 65.0%, and 72.5% against Ovi-1.1, LTX-2.3, and daVinci.The evaluation covers 250 cases across T2AV and TI2AV generation under the GSB protocol.
Ablation Studies: The UFL-only variant weakens Sync-C and IB scores, whereas combining HAL and UFL yields the best alignment and video quality with competitive audio quality.This demonstrates that fully shared generation without an explicit alignment stage is insufficient for fine-grained audio-video correspondence.
Ablation Studies: Alignment CFG increases Sync-C from 6.170 to 7.791, reduces Sync-D from 8.755 to 7.566, improves IB from 0.355 to 0.402, and lowers WER from 0.126 to 0.099.Video quality changes only slightly, while PQ improves from 6.658 to 6.860.

4 Related Work

Prior work spans video-to-audio generation, joint audio-video generation, unified multimodal modeling, and controllable audio-visual generation. These approaches improve cross-modal interaction and conditioning but face challenges in tight synchronization and flexible reference control.

Video-to-Audio Generation: Video-to-audio generation synthesizes acoustic content from video and often serves as a cascaded component for audio-visual content creation.Early methods use multimodal representation learning and cross-modal conditioning with Transformer architectures or visual-textual encoders.
Audio-Video Joint Generation: Audio-video joint generation synthesizes both modalities within a shared process, requiring tighter temporal and semantic coordination than video-to-audio generation.Early systems include MM-Diffusion, Javis-DiT, and Universe-1; recent open-source systems include UniAVGen, Ovi, LTX, and MoVA.
Unified Multimodal Modeling: Unified models such as Apollo and daVinci-MagiHuman enable direct multimodal interaction through shared attention or tri-modal spaces.Fully mixing semantic context with generation modalities can entangle high-level conditioning with low-level audio-video synchronization.
Controllable Audio-Visual Generation: Controllable audio-visual generation conditions on identity, reference audio, speaker style, or timbre in addition to synchronized audio and video.UniAVGen and DreamID-Omni use reference tokens or identity/timbre conditions, while many mechanisms apply global controls or auxiliary branches.

5 Conclusion

NAVA decouples audio-visual synchronization from context conditioning through dedicated alignment and external guidance, instantiated in an Align-then-Fuse MMDiT architecture with Timbre-in-Context Conditioning. Despite strong overall performance, it remains limited on long-tail and highly compositional audio events, motivating broader curated data and deeper audio-visual coupling.

Conclusion: NAVA establishes audio-video correspondence in a dedicated alignment space while using context as external guidance, decoupling synchronization from context conditioning.This formulation is instantiated as a Native Audio-Visual Alignment framework for joint audio-video generation.
Conclusion: The Align-then-Fuse MMDiT architecture bridges modality-aware alignment and unified audio-video denoising.NAVA also introduces Timbre-in-Context Conditioning for segment-level reference cues, though the supplied passage truncates the complete description.
Limitations and Future Work: NAVA remains limited on long-tail and highly compositional events, including rare animal sounds, music, singing, and complex mixtures of scene sounds.The passage identifies broader, meticulously curated audio-visual data as necessary for rare events and compositionally rich scenarios.
Limitations and Future Work: Deeper audio-visual coupling is identified as a highly promising direction for future work.This direction is presented alongside the need for broader and more carefully curated audio-visual data.

6 Appendix

The appendix describes NAVA’s scalable data construction, structured audio-visual annotation, quality filtering, and distributed training infrastructure. It combines heterogeneous media sources with modality-aware processing and asynchronous preparation for long multimodal sequences.

Data collection and preprocessing: NAVA constructs a heterogeneous audio-visual corpus from Koala-36M, TED-style speech videos, and raw movie/TV footage using large-scale segmentation, OCR filtering, subtitle removal, and deduplication.Video embeddings are extracted to remove redundant or near-duplicate clips.
Modality-aware tagging: Visual and acoustic metadata are assigned through VLM-based filtering, semantic tagging, YAMNet classification, and omni-modal categorization.Visual tags cover content types such as movies, documentaries, speeches, news, and interviews; audio categories include speech-related classes.
Caption annotation: Captions are generated hierarchically, with separate Qwen3-VL and Qwen3-Omni captions fused on the full dataset and Gemini-3-Pro captions produced for high-quality multi-speaker subsets.Fusion uses direct concatenation or Gemini-3-Flash rewriting on the full-scale dataset.
Quality filtering: Quality operators assess visual aesthetics, sharpness, brightness, and motion, audio aesthetics, and audio-visual alignment with SyncNet, SyncFormer, and ImageBind.The resulting corpus is diverse, deduplicated, high-quality, and richly annotated.
Corpus statistics: 15M clips remain for large-scale training after filtering, deduplication, and alignment checks, while 160K high-quality samples are retained for supervised fine-tuning.The raw collection contains approximately 20M audio clips and 100M video clips; Koala-36M contributes approximately 20% of the final training corpus, and average video duration is about 7 seconds.
Infrastructure and training cost: Training uses FSDP and asynchronous server-based preprocessing to shard model states, support long multimodal sequences, and reduce worker stalls from media and feature preparation.Dedicated data servers handle media I/O, audio-video extraction, VAE encoding, and prefetching.