Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Chetwin Low; Weimin Wang; Calder Katyal

Source-linked AI summary

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Chetwin Low, Weimin Wang, Calder Katyal

arXiv:2510.01284v1Sep 30, 2025cs.MMcs.CVcs.SDeess.AS

TL;DR

Existing audio–video generation commonly uses sequential or modality-specific pipelines, leaving unified one-pass generation underexplored. OVI jointly generates both modalities with symmetric DiT backbones, bidirectional cross-modal fusion, and scaled temporal embeddings, producing synchronized outputs while remaining competitive with dedicated audio models and open-source joint-generation baselines. Its current scope is short 5-second 720p/24 fps clips.

Problem
Unified one-pass audio–video generation at scale remains largely unexplored in open literature, while existing systems commonly handle one modality at a time or rely on post hoc alignment.
Method
OVI jointly trains architecturally matched audio and video DiT backbones with blockwise bidirectional cross-attention, scaled RoPE, shared T5 conditioning, and staged audio pretraining and fusion training.
Results
OVI is preferred over JavisDiT and UniVerse-1 across audio quality, video quality, and audio-video synchronization, while OVI-AUD performs comparably to dedicated state-of-the-art models on its respective metrics.
Takeaways & Limitations
The unified formulation produces coherent, synchronized audio–video outputs and provides a practical template for open, scalable joint generation.
Takeaways & Limitations
OVI is currently tuned to short 5-second 720p/24 fps clips, leaving minute-scale narratives, inter-shot transitions, and global story consistency out of scope.

Abstract

from arXiv · show

Audio-video generation has often relied on complex multi-stage architectures or sequential synthesis of sound and visuals. We introduce Ovi, a unified paradigm for audio-video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, Ovi achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw audio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled-RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips. All the demos, code and model weights are published at https://aaxwaz.github.io/Ovi

1 INTRODUCTION

OVI addresses the limited availability of unified, one-pass audio–video generation by jointly modeling both modalities. Its twin-DiT design, staged training, and cross-modal fusion target synchronized, high-quality 5-second clips.

1 INTRODUCTION: OVI produces audio and video in a single pass with two architecturally matched latent diffusion transformers.The branches exchange information through blockwise, bidirectional cross-modal attention.
1 INTRODUCTION: A single frozen T5 encoder conditions both modalities through a combined natural-language prompt.The shared conditioning scheme unifies semantic control across audio and video.
1 INTRODUCTION: The model uses aligned RoPE scaling and bidirectional fusion to learn synchronization between audio and video.The design is intended to avoid separate pipelines and post hoc alignment.
1 INTRODUCTION: OVI trains an audio tower from scratch on large-scale audio, then fine-tunes twin backbones on paired audio–video data.The audio stage targets speech and diverse sound effects while preserving unimodal fidelity during fusion training.
1 INTRODUCTION: The resulting system yields synchronized 5-second clips at 720×720 and 24 fps.The contribution statement reports strong synchronization without face masks or post hoc alignment.

2 RELATED WORK

Prior work largely generates one modality conditioned on the other, while unified audio–video generation remains comparatively new. Existing approaches use DiT-based latent generation but differ in conditioning, fusion, and synchronization mechanisms.

2 RELATED WORK: T2V, A2V, and V2A systems generally handle one modality at a time rather than learning audio and visuals jointly.Joint audio–video generation is treated as a distinct and newer subproblem.
2 RELATED WORK: T2V methods generate silent video from text, optionally using a reference image, with latent-space spatiotemporal DiTs.Sora and subsequent open-source implementations are described as major developments in this area.
2 RELATED WORK: A2V methods condition video generation on fixed, pre-generated audio and may add reference images, emotion features, or audio masks.These designs include HunyuanVideo-Avatar and long-form talking-head approaches.
2 RELATED WORK: V2A pipelines generate audio from video and text using latent DiTs, often with frame-level cross-modal attention or auxiliary alignment components.MMAudio performs joint attention inside one DiT but requires an auxiliary synchronization module.
2 RELATED WORK: Joint-generation systems such as Veo3, UniVerse-1, and related projects seek synchronized audio–video outputs with different backbone and fusion choices.Veo3 is identified as closed-source, while open-source replication efforts are described as limited.

3 DATA PROCESSING PIPELINE

OVI constructs multimodal training data through filtering, synchronization checks, captioning, and packing. It combines paired audio–video data with audio-only data to support alignment and audio modeling.

3 DATA PROCESSING PIPELINE: OVI curates paired audio–video and audio-only corpora for modality alignment, acoustic pretraining, and fine-tuning.The corpora include human and nonhuman data from diverse contexts.
3 DATA PROCESSING PIPELINE: The pipeline has four stages: splitting and filtering, sync detection, captioning, and packing.These stages target quality, diversity, and synchronization across modalities.
3 DATA PROCESSING PIPELINE: Scene detection isolates 121-frame clips at 24 fps, while RAFT, aesthetic prediction, and face detection filter motion, quality, and contextual composition.The target resolution is greater than 720×720 pixels before final resizing.
3 DATA PROCESSING PIPELINE: SyncNet retains speech clips with |offset| ≤3, confidence > 1.5, and minimum mean volume of −60 decibels.The filter is applied at million-video scale using scalar confidence and offset values.
3 DATA PROCESSING PIPELINE: An MLLM creates verbose captions interleaving visual events and audible speech, followed by rich audio descriptions.It receives seven evenly spaced frames and the entire audio track.
3 DATA PROCESSING PIPELINE: Packing removes video margins, resizes frames to 720×720, samples at 24 fps, and converts video and audio into byte arrays.Audio-only preprocessing uses up to 12-second clips for pretraining and exactly 5.04-second clips for video-compatible training.

4 METHOD

OVI uses symmetric audio and video DiT branches that exchange semantic and temporal information throughout the network. Staged audio training, scaled RoPE, paired sampling, and selective attention fine-tuning support synchronized generation.

4.1 ARCHITECTURE OVERVIEW: The audio and video branches share an identical DiT architecture, including transformer blocks, heads, head dimensions, and FFNs.The video branch starts from Wan2.2 5B, while the audio branch is trained from scratch.
4.1 ARCHITECTURE OVERVIEW: Each transformer block contains paired cross-attention layers that let audio attend to video and video attend to audio.This exchanges synchronization cues throughout the network without intermediate projection layers.
4.1 ARCHITECTURE OVERVIEW: Audio RoPE frequencies are scaled by 31/157 ≈0.197 because video latents span 31 frames while audio latents form 157 tokens.The scaling aligns temporal positions for cross-modal attention.
4.1 ARCHITECTURE OVERVIEW: A single frozen T5 encoder embeds a combined prompt used independently by the audio and video branches.Visual details inform audio generation, while acoustic details guide facial movements and video actions.
4.2.1 AUDIO MODEL TRAINING: Audio generation uses a pretrained 1D VAE to encode mel-spectrograms and BigVGAN to vocode decoded spectrograms into waveforms.The pipeline operates at 16 kHz.
4.2.1 AUDIO MODEL TRAINING: The audio tower is pretrained on variable-length speech and then fine-tuned on padded 5.04-second waveforms with added sound effects.Pretraining uses hundreds of thousands of hours and targets speaker traits such as pitch and emotion.
4.2.2 AUDIO–VIDEO MODEL TRAINING: Fusion training fine-tunes self-attention and cross-attention modules while freezing FFNs, leaving 5.7B of 11B parameters trainable.Paired latents and a shared timestep encourage correspondences such as lip-sync and action–sound alignment without explicit synchronization losses.

5 EXPERIMENTS

OVI is evaluated as a unified audio-video generator through cross-modal attention visualizations, audio benchmarks, human preference testing, and an audio-tower ablation. The results indicate relevant audio-visual grounding, competitive standalone audio generation, and consistent preference over selected joint-generation baselines.

5.1 CROSS-MODAL ATTENTION VISUALIZATIONS: Speech, drumming, and animal sounds attend to corresponding visual regions, indicating alignment between audio tokens and relevant scene cues.A2V heatmaps show attention concentrated on mouths, drums, and source body parts.
5.4 RESULTS: OVI is evaluated against joint audio-video baselines with a 50-participant blind pairwise preference study measuring quality and synchronization.The study reports pairwise win rates against competing models on Verse-Bench.
5.4 RESULTS: OVI is consistently preferred over JavisDiT and UniVerse-1 across audio quality, video quality, and audio-video synchronization.The comparison uses the Verse-Bench dataset and reports substantial preference margins.
5.2 AUDIO EVALUATION: OVI-AUD achieves performance comparable to dedicated state-of-the-art models on both text-to-audio and text-to-speech evaluation metrics.T2A evaluation uses FDPANNs, FDVGG, IS, and CLAP, while TTS evaluation reports WER.
5.5 ABLATION STUDY: Combining speech transcripts and audio descriptions into one T5 embedding preserves comparable WER while improving audio fidelity and alignment metrics.The unified embedding also lets both towers condition on the same text representation for joint generation.

6 LIMITATIONS AND CONCLUSION

OVI provides unified audio-video generation through symmetric twin backbones and blockwise bidirectional fusion, producing coherent synchronized outputs without heuristic synchronization modules. Its current scope is limited to short 5-second 720p/24 fps clips and constrained audio bandwidth.

6 LIMITATIONS AND CONCLUSION: OVI is tuned to short 5-second 720p/24 fps clips, leaving minute-scale narratives, inter-shot transitions, and global story consistency out of scope.The authors suggest chunk-wise causal audio and video approaches as one possible route to longer duration.
6 LIMITATIONS AND CONCLUSION: The fixed 16 kHz 1D-VAE audio path constrains bandwidth and spatial realism, potentially flattening high-fidelity music, spatial cues, and subtle timbre.The authors propose higher-bandwidth latents or post-processing bandwidth extension as possible improvements.
6 LIMITATIONS AND CONCLUSION: OVI treats audio and video as one generative object, jointly learning timing and semantics through symmetric backbones and blockwise bidirectional fusion.The framework supports synchronization without face masks or auxiliary synchronization modules.
6 LIMITATIONS AND CONCLUSION: OVI produces coherent, synchronized outputs and establishes a practical template for simple and scalable joint audio-video generation.The conclusion characterizes the unified formulation as competitive and effective.