PaperlayerView exact revision on arXiv

Source-linked AI summary

MOVA: Towards Scalable and Synchronized Video-Audio Generation

SII-OpenMOSS Team, :, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, Wenming Tu, Xiangyu Peng, Yang Gao, Yanru Huo, Ying Zhu, Yinze Luo, Yiyang Zhang, Yuerong Song, Zhe Xu, Zhiyu Zhang, Chenchen Yang, Cheng Chang, Chushu Zhou, Hanfu Chen, Hongnan Ma, Jiaxi Li, Jingqi Tong, Junxi Liu, Ke Chen, Shimin Li, Shiqi Jiang, Songlin Wang, Wei Jiang, Zhaoye Fei, Zhiyuan Ning, Chunguo Li, Chenhui Li, Ziwei He, Zengfeng Huang, Xie Chen, Xipeng Qiu

arXiv:2602.08794v2cs.CVcs.SD

TL;DR

Joint video-audio generation remains limited by modality-isolated pipelines and proprietary systems. MOVA combines dual video and audio towers with bidirectional cross-modal interaction, and scaling improves synchronization, especially for lip-synced speech.

  • Problem

    Video-audio generation remains underexplored because cascaded pipelines lack cross-modal interaction and leading synchronized systems are closed-source.

  • Method

    MOVA uses asymmetric pretrained video and audio towers with bidirectional cross-attention to generate synchronized video-audio latents.

  • Results

    Scaling video capacity and training data substantially improves synchronization, particularly lip synchronization, while smaller models show performance saturation.

  • Takeaways & Limitations

    Joint video-audio generation benefits from scaling model capacity and aligned training data, especially for fine-grained speech synchronization.

  • Takeaways & Limitations

    The limited audio-tower capacity degrades performance on singing, complex sound textures, music, and instrumental content.

Abstract

from arXiv · show

Audio is indispensable for real-world video, yet generation models have largely overlooked audio components. Current approaches to producing audio-visual content often rely on cascaded pipelines, which increase cost, accumulate errors, and degrade overall quality. While systems such as Veo 3 and Sora 2 emphasize the value of simultaneous generation, joint multimodal modeling introduces unique challenges in architecture, data, and training. Moreover, the closed-source nature of existing systems limits progress in the field. In this work, we introduce MOVA (MOSS Video and Audio), an open-source model capable of generating high-quality, synchronized audio-visual content, including realistic lip-synced speech, environment-aware sound effects, and content-aligned music. MOVA employs a Mixture-of-Experts (MoE) architecture, with a total of 32B parameters, of which 18B are active during inference. It supports IT2VA (Image-Text to Video-Audio) generation task. By releasing the model weights and code, we aim to advance research and foster a vibrant community of creators. The released codebase features comprehensive support for efficient inference, LoRA fine-tuning, and prompt enhancement.

1 Introduction

MOVA addresses the limitations of cascaded, largely closed-source video-audio generation through synchronized joint modeling. It combines an asymmetric dual-tower architecture, scalable bimodal data construction, and training that improves synchronization across modalities.

  • Motivation: Cascaded pipelines limit quality because audio and video are synthesized separately without modality interaction, while leading end-to-end systems remain closed-source [11–16].This leaves high-quality video-audio generation comparatively underexplored in the research community.
  • Challenges: Video-audio generation introduces three challenges: fine-grained audio-video data construction, mutually informative modality fusion, and differing native information density between modalities.These challenges make simultaneous bimodal generation more difficult than video-only generation.
  • Contributions: MOVA targets synchronized video-audio generation with multilingual lip-synced speech and precisely aligned environmental sounds.It also supports synchronized generation across diverse scenarios, including English and Chinese multi-speaker speech, physical sound effects, scene text, and 16:9 or 9:16 formats.
  • Architecture: MOVA couples pretrained video and audio generators through an asymmetric dual-tower architecture with bidirectional cross-attention for modality fusion.The structure combines an A14B video DiT, a 1.3B audio DiT, and a 2.6B bidirectional Bridge module.
  • Data and Training: MOVA uses a fine-grained audio-video captioning pipeline to produce high-quality bimodal training data at scale.Scaling video-audio training yields continuous improvements in synchronization performance across both modalities.

2 Model Architecture

MOVA jointly generates temporally aligned video–audio pairs in compact modality-specific latent spaces, coupling pretrained video and audio diffusion backbones through bidirectional hidden-state exchange. Aligned positional encoding places video and audio tokens on a shared temporal scale, while flow matching trains synchronized cross-modal dynamics.

  • Latent Representation: MOVA compresses video and 48 kHz mono audio into compact latent representations using pretrained Wan2.1 and DAC-style audio VAEs.All subsequent modules operate in these latent spaces.
  • Training Objective: Training uses flow matching to regress a cross-modal velocity field conditioned on text and optionally an input image, with separate video and audio loss weights.The objective balances video- and audio-velocity regression through λ_v and λ_a.
  • Backbones and Bridge: The architecture combines Wan2.2 I2V A14B and a 1.3B text-to-audio diffusion backbone through a lightweight dual-tower Bridge.The Bridge enables bidirectional information exchange while preserving the pretrained backbones.
  • Backbones and Bridge: At each interaction layer, cross-attention injects video hidden states into the audio DiT and audio hidden states into the video DiT.This hidden-state-level coupling exchanges information in both directions between the two DiT backbones.
  • Aligned RoPE: Aligned RoPE maps video and audio latent positions onto a shared temporal scale to prevent cross-attention from pairing tokens at mismatched physical times.Video indices are scaled by s = f_a/f_v, where f_v and f_a are the post-VAE latent frame rates.

3 Data Engineering

MOVA builds a three-stage data curation pipeline that converts heterogeneous raw videos into fixed-length, high-quality, synchronized clips with coherent multimodal captions. The process combines preprocessing, quality and alignment filtering, and separate video/audio annotation followed by cross-modal caption merging.

  • Data curation overview: A three-stage pipeline progressively preprocesses raw videos, filters for audio, video, and audiovisual quality, and annotates retained clips with coherent multimodal captions.The stages produce fixed-length clips, remove low-quality samples, and preserve strong audio-visual consistency and semantic labels.
  • Video preprocessing: The pipeline standardizes videos into 720p, 24fps, 8.05-second clips, using VAD and scene transitions to create speech/non-speech and single-/multi-scene segments.Each segment contains 193 frames, and speech boundaries are adjusted to avoid truncating ongoing speech.
  • Data collection: The curated corpus combines filtered public datasets [26] [27] [28] [29] [30] [31] [32] with in-house data spanning domains such as education, sports, beauty, news, and animation.This diversity is intended to support generalization across complex real-world scenarios.
  • Data filtering: Quality filtering evaluates audio quality, video quality, temporal synchronization, and semantic audiovisual alignment before retaining clips.Audiobox-aesthetics, DOVER, SynchFormer, and ImageBind assess these dimensions, respectively.
  • Multimodal annotation: Video and audio are annotated through distinct pipelines using MiMo-VL-7B-RL for video, Qwen3-Omni models [40] for speech and non-speech audio, and GPT-OSS-120B [41] for cross-modal caption merging.Caption merging checks visual-audio consistency and resolves potential conflicts between modalities.

4 Training Strategy

MOVA uses staged audio pretraining followed by synchronous end-to-end video-audio joint training, with frozen VAEs and bidirectional Bridge cross-attention connecting the towers. A three-phase curriculum, modality-specific noise schedules, and heterogeneous learning rates support scalable training and audio-video alignment.

  • Training pipeline: Training proceeds in two stages: a 1.3B text-to-audio DiT is pretrained on music, general sounds, and speech, then jointly trained with the A14B video tower.The video tower and video VAE initialize from Wan2.2, the audio VAE from HunyuanVideo-Foley [21], and both VAEs remain frozen; Bridge cross-attention connects the towers.
  • Audio tower pretraining: The audio tower reuses the Wan2.1-1.3B architecture but replaces 3D positional encoding with temporal 1D encoding, while preserving other components and depth.Its training data covers WavCaps and VGGSound [26] [42], JamendoMaxCaps [43], and in-house TTS, with duration-controlled fixed-length clips.
  • Audio tower evaluation: 10.54 IS exceeds AudioLDM2 (7.79), while FD openl3 reaches 72.25 versus AudioLDM2 (72.04) and TangoFLUX (80.47), with KL passt at 1.47 versus 1.66.On AudioCaps, CLAP remains 0.463; Table 2 reports the benchmark metrics and model settings.
  • Audio tower evaluation: CU = 5.56 and PQ = 6.20 are the best AudioBox results, although the method does not lead in CE or PC.These results surpass advanced baselines including AudioLDM2 and Tango2.
  • Optimization strategy: End-to-end optimization updates the Bridge and both towers from the first step, using η_br = 2 × 10^-5 versus η_b = 1 × 10^-5 to balance alignment and tower stability.A two-stage warm-start plateaued early, whereas the factor-of-two learning-rate difference accelerates Bridge convergence and reduces forgetting in pretrained towers.

5 Inference

MOVA uses dual classifier-free guidance to independently control text fidelity and cross-modal alignment, while a multi-stage prompt-enhancement workflow improves visually grounded video-audio generation from images or text.

  • Dual Classifier-Free Guidance: Dual CFG independently scales textual and Bridge-based cross-modal conditioning, providing flexible control over the alignment–quality trade-off via a principled Bayes-derived formulation [57].The general formulation uses three function evaluations per step (NFE=3), while two-branch special cases use NFE=2.
  • Dual Classifier-Free Guidance: Text-only CFG yields high semantic fidelity, such as ImageBind scores, but weaker temporal synchronization and higher DeSync.It keeps Bridge active in both branches and uses NFE=2.
  • Dual Classifier-Free Guidance: Text + modality CFG produces stronger synchronization, including lower DeSync and better lip-sync, by disabling Bridge injection in the unconditional branch.This two-branch scheme also uses NFE=2 and isolates the cross-modal alignment signal.
  • Prompt Enhancement Workflow: MOVA enhances user inputs through visual grounding and narrative synthesis to preserve the reference frame’s style, lighting, cinematography, and visual priors.Qwen3-VL [58] extracts structured descriptions covering style, cinematography, visual elements, and exact OCR text, while an LLM such as Gemini 2.5 Pro synthesizes the generation prompt using in-context learning [60].
  • Prompt Enhancement Workflow: The final generator uses the synthesized prompt and initial frame as dual conditioning, while an uninformative white image enables text-to-video-audio generation.The workflow is illustrated in Figure 5 and supports zero-shot video-audio synthesis from text prompts.

6 Evaluation

MOVA is evaluated on broad and dedicated video–audio benchmarks using objective measures of audio quality, cross-modal alignment, temporal consistency, and lip synchronization, alongside human preference studies. Across these evaluations, MOVA achieves strong audio quality, synchronization, lip-sync, speaker attribution, resolution robustness, and user preference, with dual CFG improving alignment-related metrics.

  • Audio Fidelity and Speech Quality: MOVA-360p achieves state-of-the-art IS 4.269 and DNSMOS 3.797, outperforming LTX-2 and Ovi in audio fidelity and speech quality.The cascaded WAN2.1 + MMAudio baseline reaches IS 4.036 but cannot generate intelligible speech, whereas MOVA maintains natural and clear speech at 720p.
  • Audio-Visual Alignment: MOVA-360p with dual CFG reaches DeSync 0.351 and IB-Score 0.315, surpassing LTX-2 and Ovi, while nearly matching the cascaded pipeline’s DeSync 0.260.The results indicate that dual CFG strengthens auditory-event binding to visual context in the unified architecture.
  • Lip-Sync Precision and Multi-Speaker Attribution: MOVA-360p with dual CFG achieves the best LSE-D 7.004 and LSE-C 7.800, while MOVA-720p reaches cpCER 0.149 versus 0.220 for LTX-2 and 0.436 for Ovi.These results demonstrate strong fine-grained lip synchronization and speaker identity consistency in multi-character scenes.
  • Scaling to High Resolution: MOVA-720p maintains DeSync 0.485 and IB-Score 0.277 versus 0.475 and 0.286 for MOVA-360p, showing limited degradation when scaling resolution.The comparison is summarized in Table 4.
  • Effect of Dual Classifier-Free Guidance: Increasing dual CFG from 1.0 to 4.0 consistently improves alignment metrics, with LSE-C peaking at 7.891 and DeSync falling to 0.365.The progression includes lower DeSync and LSE-D alongside higher IB-Score and LSE-C, as reported in Table 5.
  • Subjective Comparison, Ablation, and Emergent T2VA Capability: MOVA receives an ELO rating of 1113.8 and exceeds 50% win rates, including over 70% against Ovi and WAN + MMAudio; its T2VA variant reaches IS 4.370 and DeSync 0.441.In the internal ablation, prompt refinement raises MOVA-720p from ELO 982.9 to 1025.3, while dual CFG can slightly reduce preference through weaker relative text guidance.

7 Discussion

The discussion identifies conditioning-direction and diffusion-schedule tensions in synchronized audio-visual generation, explains why the adopted dual-CFG ordering is practically useful, and highlights limitations in audio capacity, multi-speaker synchronization, and computational scaling.

  • Conditioning Tension: Discrete event-driven scenes naturally favor Video→Audio conditioning because visual events anchor the timing and location of many sound events.The passage identifies impacts, collisions, cuts, and percussive gestures as examples of visually driven events with clear temporal onsets.
  • Conditioning Tension: Fixed audio and video noise schedules create a diffusion-conditioning tension because audio corruption is design-fixed while visual uncertainty varies with scene content.The schedules σ_v(t_v) and σ_a(t_a) are predefined and non-learnable, while effective visual uncertainty can change with object scale and visual dominance.
  • Dual-CFG Design: The adopted dual-CFG ordering reduces to standard text-only CFG when s_B=1 and interpolates between text-only and text-plus-modality guidance without changing sampling.This ordering first activates cross-modal information through c_B and then applies text guidance through c_T, while retaining fixed cross-modal injection.
  • Limitations: Key limitations include weak modeling of singing, complex textures, and music; unreliable multi-speaker attribution; and high sequence-length-driven compute and latency costs.A Wan2.1-1.3B audio backbone limits acoustically rich signals, annotation errors compound multi-speaker synchronization failures, and a 720p 8 s clip produces approximately 1.6 × 10^5 tokens.

8 Related Work

Prior work has advanced scalable video and audio generation, but most systems remain video-only or rely on cascaded pipelines. Joint audio-video models address these limitations through end-to-end architectures, though existing approaches face instability or restricted sound coverage.

  • Video Generation: Diffusion transformers [1] [68] support large-scale video synthesis, while open models such as Wan and HunyuanVideo [69] achieve near-photorealistic outputs.Recent extensions target long-horizon generation [73], controllable camera motion [74], and resolutions above 1080p, but most text-to-video systems omit audio.
  • Audio Generation and Cascaded Pipelines: Latent diffusion enables scalable text-to-audio generation [50] [75], while cascaded video-to-audio pipelines [24] [78] [79] remain prevalent for audiovisual content creation.Audio VAEs use compact latent representations: DAC [76] applies residual vector quantization, whereas Stable Audio [77] uses a stereo variational autoencoder with spectral losses.
  • Joint Audio-Video Generation: End-to-end joint generation [17] [18] [19] [20] [81] seeks to overcome cascaded limitations, but existing methods encounter adversarial-training instability or remain restricted to ambient sounds.MMDisCo [83] uses discriminator-guided cooperative diffusion, while MM-Diffusion and JavisDiT [81] use dual-stream cross-modal attention and are limited to ambient sounds.

9 Conclusions

MOVA is an open, scalable framework for synchronized joint video–audio generation, combining a dual-tower architecture with a bidirectional bridge and Aligned RoPE. The work addresses data, modeling, and scaling challenges while providing practical large-scale training optimizations and planned open releases.

  • Model and architecture: MOVA combines a 32B-parameter model with 18B active parameters, using asymmetric video and audio towers, a 2.6B bidirectional bridge, and Aligned RoPE for fine-grained temporal interaction.The architecture includes an A14B video backbone and a 1.3B audio backbone.
  • Data and training: MOVA curates over 100,000 hours of fine-grained audio–visual data and introduces training designs for stable large-scale multimodal diffusion, including decoupled timestep sampling.The data include sound, music, and speech annotations aligned to visual content, while modality-specific noise schedules are enabled by decoupled timestep sampling.
  • Systems optimization: Approximately 35% MFU was achieved in stable 1024-GPU training through Context Parallelism, FSDP2 strategies, and scheduled garbage collection.These system optimizations target practical large-scale training efficiency.
  • Release and impact: The authors plan to release MOVA’s model weights, training code, and inference pipelines as an open baseline for synchronized audio–video generation research.The release is intended to support future research and community development.

Appendix … A.6 Benchmark Details

The appendix documents reproducible training and benchmarking configurations, speech-window algorithms, filtering logic, multimodal captioning prompts, and benchmark scenarios for joint video–audio generation. It also provides implementation examples spanning visual, audio, speech, and integrated captions.

  • A.1 Training Hyperparameters: Table 7 specifies the three-phase training configuration, including parallelism, learning rates, noise schedules, data curation, and 1024-GPU deployment with data-parallel replicate size 64.These details are provided for reproducibility.
  • A.2 Ascend 910A2 Benchmark Details: The Ascend 910A2 appendix reports an 8-device microbenchmark with CP=4 and DP-shard=2, while cautioning that step times depend on the software and distributed-training stack.The benchmark is not presented as a complete estimate of training cost at scale.
  • A.3 Multi-shot and Single-shot Speech Window Generation: Multi-shot and single-shot speech-window generation use VAD segments, scene boundaries, and fixed 8.05-second windows constrained to avoid overlap and unnatural scene transitions.Multi-shot advances across speech segments, whereas single-shot searches consecutive scene intervals for windows fully contained within a scene.
  • A.4 Detailed Filtering Thresholds: Data-quality filtering uses thresholds across audio quality, video quality, and audio-visual alignment, with thresholds chosen empirically to balance corpus quality and diversity.Alignment retains videos meeting either IB-Score ≥0.2 or DeSync ≤0.5, while the speech subset requires positive EAT-contained-Speech and EAT-contained-Singing classifications.
  • A.5 Audio-visual Captioning Details: The annotation pipeline separates visual captioning, speech transcription, and non-speech audio captioning before GPT-OSS-120B [41] consistency checking and caption merging.It uses MiMo-VL-7B-RL for video captioning, Qwen3-Omni-Instruct [40] for speech transcription, and Qwen3-Omni-Captioner [40] for non-speech sounds.
  • Visual Description Prompt / Speech Transcription Prompt / Merge Caption Prompt: The caption prompts enforce visually verifiable descriptions, exact speech transcription, and coherent integration of visual context, dialogue, speaker dynamics, music, ambience, and other non-speech audio.Visual prompts prohibit audio-based inference; speech prompts preserve the original language and mark absent speech as null; merge prompts avoid repetition and omit empty fields.
  • Visual Description / Audio Description / Speech Description / Integrated Caption: Caption examples combine separate sound, music, and video descriptions into a total caption, illustrated with an HP printer scene, professional male narration, background electronic music, and an abruptly cut-off advertisement excerpt.The integrated example includes visual cartridge insertion, verbatim speech, and synchronized multimodal context.
Loading 2602.08794v2…