Source-linked AI summary
OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
Jun Zhan, Chen Yang, Yitian Gong, Donghua Yu, Kuangwei Chen, Wenbo Zhang, Kexin Huang, Qi Luo, Zhe Xu, Ying Zhu, Jin Wang, Tengyue Zhang, Qi Chen, Cheng Chang, Songlin Wang, Junqi Dai, Jiasheng Ye, Xiaogui Yang, Tianyi Liang, Xiangyu Peng, Zhaoye Fei, Shimin Li, Qinyuan Cheng, Xie Chen, Xinchi Chen, Xipeng Qiu
TL;DR
Joint audio-video generation still lacks fine-grained cross-modal alignment because audio and video VAEs are typically trained separately. OmniVAE jointly trains aligned latent spaces with semantic distillation and segment-level contrastive learning, improving downstream representation learnability, generation quality, and synchronization.
Problem
Existing audio-video systems typically use independently trained VAEs, leaving latent spaces without explicit semantic or cross-modal alignment for fine-grained synchronization.
Method
OmniVAE jointly trains modality-specific audio and video latent spaces using semantic distillation and segment-level bidirectional contrastive alignment beyond reconstruction.
Results
The two objectives consistently improve latent-space learnability, downstream generation quality, and fine-grained audio-video synchronization, with contrastive variants achieving the strongest overall alignment.
Takeaways & Limitations
Embedding modality-specific semantics and cross-modal correspondence directly in the latents provides a stronger foundation for unified multimodal generation.
Takeaways & Limitations
Future work must test more effective semantic-learning methods and validate their benefits at larger scales.
Abstract
from arXiv · showhide
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
1 Introduction
The introduction identifies cross-modal latent misalignment as a central obstacle to synchronized audio-video generation and presents OmniVAE as a jointly trained VAE designed to embed semantic structure and correspondence directly into its latent spaces.
- 1 Introduction: Unified audio-video generation seeks a video and temporally synchronized, semantically coherent soundtrack from one text prompt.
- 1 Introduction: Independently trained modality-specific VAEs leave downstream generators to learn both modality distributions and audio-video correspondence, making fine-grained synchronization challenging.Reconstruction alone does not guarantee semantically structured or cross-modally aligned latent spaces.
- 1 Introduction: Prior work improves modality-specific semantic representations or demonstrates explicit synchronization, motivating a unified latent-space approach.
- 1 Introduction: OmniVAE jointly trains audio-video latent spaces to embed modality-specific semantics and cross-modal correspondence while retaining each modality’s reconstruction capability.
- 1 Introduction: OmniVAE is presented as an audio-video VAE designed explicitly for fine-grained semantic and temporal cross-modal latent alignment.The authors describe it as the first audio-video VAE designed explicitly for cross-modal latent alignment.
2 Related Work
Related work develops unified multimodal representations, modality-specific generative tokenizers, and joint audio-video generators. However, existing approaches either lack fine-grained temporal alignment or are not directly decodable for generative modeling, motivating OmniVAE’s aligned and reconstructive latent spaces.
- Multimodal Representation Learning: Multimodal unification organizes modalities in a common semantic coordinate system or explicitly aligned modality-specific spaces, without requiring shared encoders or identical latent distributions.
- Multimodal Representation Learning: Global contrastive learning and modality-agnostic codes capture high-level cross-modal semantics, but provide limited modeling of fine-grained temporal relationships.
- Generative Tokenizers: Existing discriminative representations are generally not optimized for signal reconstruction, preventing direct use as generative latent spaces.
- Generative Tokenizers: Video latent-diffusion systems use continuous VAEs for spatio-temporal compression, while audio systems use neural codecs and continuous VAEs such as DAC, Stable Audio VAE, and Qwen-Audio-VAE.
- Audio-Video Generation: Joint audio-video generation has progressed from coupled dual-U-Net and DiT frameworks to industrial systems and open-source models with synchronized audio.
3 OmniVAE
OmniVAE jointly trains separate audio and video VAE branches while adding training-only objectives for fine-grained cross-modal alignment and modality-specific semantic supervision. The resulting tokenizer supports downstream joint text-to-audio-video generation.
- Architecture: OmniVAE retains separate encoders and decoders for audio and video, with independent latent representations and no inference-time interaction between branches.The video branch uses 4× temporal and 8× spatial downsampling, while the audio branch encodes a 48 kHz waveform with temporal downsampling factor d_a.
- Cross-Modal Alignment: A segment-level audio-video contrastive objective aligns temporally corresponding latent segments in a shared embedding space using paired clips from the same source-video window.The objective contrasts matched segments against negatives at multiple levels to establish cross-modal correspondence in the latent representations.
- Cross-Modal Alignment: The negative-construction pipeline divides 8-second clips into S = 48 segments of 1/6 second and combines hard temporal distinctions with broad semantic diversity.Negatives are drawn beyond the anchor clip’s within-clip segments to retain source-video identity distinctions and broader semantic variation.
- Semantic Distillation: OmniVAE independently distills semantic features from frozen Qwen3-Omni visual and audio teachers into the video and audio latent spaces.Lightweight projectors align latent channels and temporal resolutions with teacher features, accounting for the video branch’s distinct first causal latent frame.
- Downstream Use: The trained OmniVAE is frozen as a tokenizer for a joint text-to-audio-video model trained with flow matching in the video and audio latent spaces.The downstream model first pretrains single-modality visual and audio priors, then couples them through lightweight cross-attention bridges.
4 Experiments
Experiments show that OmniVAE’s contrastive learning and semantic distillation improve cross-modal alignment and modality-specific generation while preserving reconstruction quality. Ablations further identify larger contrastive negative pools and loss-magnitude weighting as effective design choices.
- Reconstruction quality: Reconstruction experiments show that added objectives largely preserve video reconstruction quality across datasets and metrics.Video is evaluated on UCF-101 and Panda-70M; audio is evaluated on LibriSpeech, AudioSet, and MUSDB18.
- Audio-video alignment: Audio-video sync probing shows that contrastive learning provides the main gain in fine-grained temporal alignment, with OmniVAE performing best.The probe classifies 21 temporal offsets on VGGSound-Sparse and reports A@1, A@5, and A@1tol.
- Joint generation: In downstream text-to-audio-video generation, AVCLIP improves cross-modal alignment and semantic distillation strengthens modality-specific generation quality.Combining both objectives yields complementary gains and the strongest overall performance across the evaluated metrics.
- Contrastive design ablation: Higher frame rates, finer temporal segments, or longer clips substantially improve sync probing by enlarging the contrastive negative pool.Performance is relatively stable across segment lengths; the adopted setup uses 24 fps, 8-second clips, and 0.17-second segments.
- Loss-balancing ablation: Loss-magnitude weighting substantially outperforms alternative strategies on intra-video and overall segment retrieval while maintaining comparable reconstruction quality.Intra A@48 retrieves paired segments within a video, whereas Overall A@64 includes negatives from other videos in the batch.
5 Conclusion
OmniVAE jointly embeds modality-specific semantics and fine-grained audio-video correspondence in its latent representations, providing a stronger foundation for unified multimodal generation. Future work should improve semantic learning methods and validate them at larger scales.
- 5 Conclusion: OmniVAE jointly embeds modality-specific semantics and fine-grained cross-modal correspondence into audio-video latent representations.It augments standard reconstruction training with segment-level audio-video contrastive learning.
- 5 Conclusion: The findings highlight incorporating modality-specific semantic structure and cross-modal correspondence directly into learned representations as a stronger foundation for unified multimodal generation.
- 5 Conclusion: Future work should explore more effective modality-specific and cross-modal semantic learning methods and validate their benefits at larger scales.