Source-linked AI summary
UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, Gang Yu
TL;DR
Joint audio-video generation lacks an open model with strong temporal coordination, motivating UniVerse-1. The paper stitches pretrained video and music experts and uses online annotation to support aligned training. It reports coordinated audio-visual generation and presents the approach as a viable efficient route based on pretrained unimodal foundations.
Problem
Open research lacks publicly detailed methods for simultaneous audio-video generation with temporal synchronization, especially for speech and lip movements.
Method
UniVerse-1 fuses pretrained WAN2.1 video and Ace-step music models through stitching of experts and uses online annotation for temporally and semantically aligned labels.
Results
The authors report that the methodology produces coordinated audio-visual content and offers a viable, efficient pathway by leveraging pre-existing unimodal foundations.
Takeaways & Limitations
The model, code, dataset, and Verse-Bench benchmark are publicly available to support reproducible research in joint audio-video generation.
Takeaways & Limitations
Training was restricted to the Wan2.1-1.3B video model, so performance is inherently limited by that base model’s capacity.
Abstract
from arXiv · showhide
We introduce UniVerse-1, a unified, Veo-3-like model capable of simultaneously generating coordinated audio and video. To enhance training efficiency, we bypass training from scratch and instead employ a stitching of experts (SoE) technique. This approach deeply fuses the corresponding blocks of pre-trained video and music generation experts models, thereby fully leveraging their foundational capabilities. To ensure accurate annotations and temporal alignment for both ambient sounds and speech with video content, we developed an online annotation pipeline that processes the required training data and generates labels during training process. This strategy circumvents the performance degradation often caused by misalignment text-based annotations. Through the synergy of these techniques, our model, after being finetuned on approximately 7,600 hours of audio-video data, produces results with well-coordinated audio-visuals for ambient sounds generation and strong alignment for speech generation. To systematically evaluate our proposed method, we introduce Verse-Bench, a new benchmark dataset. In an effort to advance research in audio-video generation and to close the performance gap with state-of-the-art models such as Veo3, we make our model and code publicly available. We hope this contribution will benefit the broader research community. Project page: https://dorniwang.github.io/UniVerse-1/.
1 Introduction
UniVerse-1 addresses the open-research gap in synchronized audio-video generation with an open-source unified model and a training strategy built around fused pretrained experts, online annotation, and a dedicated benchmark.
- Motivation: Video generation has advanced mainly as a visual task, while post-hoc audio generation cannot enforce reverse temporal alignment for speech and video.This limitation makes tasks such as lip-speech synchronization impossible for post-hoc systems.
- UniVerse-1: UniVerse-1 is an open-source model designed to simultaneously generate coordinated audio and video.It is presented as a Veo-3-like unified model intended to bridge closed-source systems and open research.
- Core methodology: The stitching-of-experts paradigm fuses pretrained WAN2.1 video and Ace-step music models through lightweight cross-modal MLP connectors.The connectors enable bidirectional interaction between modalities and leverage pretrained priors to accelerate convergence.
- Core methodology: An online annotation pipeline dynamically generates labels during training to maintain temporal and semantic alignment and mitigate static-annotation degradation.The pipeline targets the misalignment problem created when random clips are paired with captions for entire videos.
- Data and evaluation: The work contributes approximately 7,600 hours of aligned audio-video data and Verse-Bench, a benchmark spanning joint and unidirectional generation tasks.Verse-Bench contains 600 image-text prompt pairs across diverse sound categories and includes the Verse-Ted audio-to-video subset.
2 Related Works
Related work traces parallel progress in diffusion-based video and audio generation, followed by early joint audio-video systems whose limited datasets constrained development.
- Video diffusion models: Diffusion models transformed video generation, progressing from UNet-based systems toward latent video models using 3D VAEs and Diffusion Transformers.Across these models, training-data quality and scale are identified as central factors in generation quality.
- Audio diffusion models: Audio diffusion research similarly advanced from early high-fidelity generation toward latent diffusion architectures that improved efficiency and quality.
- Joint audio-video generation: Early joint audio-video diffusion systems used separate audio and video subnetworks but were typically trained on datasets smaller than 10 hours.The small data scale limited these initial approaches.
3 UniVerse-1
UniVerse-1 combines pretrained video and music-generation foundations with flow matching, curated multimodal data, and quality filtering to support unified audio-video synthesis.
- Model foundations: UniVerse-1 is built by stitching the 1.3B-parameter Wan2.1 video model with the 3.5B-parameter Ace-step music model.Wan2.1 uses a 3D VAE, text encoder, and DiT, while Ace-step combines audio compression, text, lyric, speaker, and DiT components.
- Training objective: Both pretrained models use Conditional Flow Matching to learn a velocity field transporting samples from a source noise distribution to the target data distribution.The target velocity is x1 − x0, and training minimizes an L2 prediction loss.
- Data curation: The curation pipeline removes videos without audio, low-quality clips, short segments, silent content, and speech clips lacking verified faces or lip synchronization.Speech clips are retained only when SyncNet confidence exceeds 2.0.
- Data curation: VGGSound and AudioSet clips are scene-detected or timestamp-segmented, with only clips longer than five seconds retained.
- Dataset: 7,685 hours of data are divided into verified speech-centric, general audio-video, and audio-focused subsets.The subsets contain 1,187, 3,074, and 3,422 hours, respectively.
3.3 Online Data Annotation
UniVerse-1 replaces fixed offline captions with an online annotation pipeline that dynamically aligns sampled audio-video clips and their textual annotations. The pipeline processes clips during training and produces synchronized speech, video, and ambient-audio labels.
- Motivation: Offline captions can become temporally and semantically misaligned with randomly sampled clips, especially when audio events must synchronize with video.This misalignment is particularly problematic for joint audio-video generation.
- Pipeline: The online pipeline runs concurrently with training, fetching raw video, processing clips in real time, and buffering synchronized data tuples.Training consumes the prepared tuples from a shared buffer.
- Pipeline: Each sampled segment undergoes multimodal captioning, text and video encoding, and audio conversion into a mel-spectrogram latent representation.The process begins by randomly extracting a fixed-length segment, such as five seconds, with corresponding audio and video.
- Multimodal annotation: Whisper supplies speech transcripts, while QWen2.5-Omni produces aligned speech, video, and ambient-audio annotations for each clip.The structured prompt incorporates the transcribed speech before generating the three annotation types.
- Outcome: The just-in-time process produces training instances whose audio and video latents are synchronized with semantically consistent textual annotations.The authors state that this eliminates the data-misalignment problem inherent in offline methods.
3.5 Independent Noise Sampling Strategy
The Independent Noise Sampling Strategy removes spurious correlations introduced when video and audio noises are drawn sequentially from one deterministic PRNG sequence. Separate independently seeded generators improve robustness to inference-time changes and mitigate audio-quality degradation.
- Problem: A shared deterministic PRNG sequentially samples video and audio noises, creating a spurious structural correlation that multimodal diffusion models can learn as a shortcut.The issue arises from the deterministic Linear Congruential method.
- Problem: Inference-time changes to video noise sampling, such as resolution or duration, alter subsequent audio noise and significantly degrade audio generation quality.The mismatch breaks the correlation learned during training.
- Solution: The Independent Noise Sampling Strategy uses separate, independently seeded PRNG instances for each modality, producing statistically independent noise vectors.This isolates noise generation between video and audio.
- Effect: Independent noise sampling makes the model robust to inference-time condition variations and mitigates the resulting performance degradation.The method specifically addresses the deterministic cross-modal correlation.
4 Experiments
The experiments evaluate UniVerse-1 across diverse audio-video benchmarks and six generation tasks, comparing it with specialized and joint-generation baselines. Results show strong identity preservation, competitive pitch correlation, and benefits from the proposed training strategies, while cross-model comparisons require caution.
- Setup: UniVerse-1 is trained with an effective batch size of 128 for 50k steps on a 7,600-hour audio-visual dataset using AdamW at 5e−6.Training uses FSDP across multiple nodes with gradient accumulation of 4.
- Benchmark: Verse-Bench contains 600 image-text prompt pairs spanning diverse sound categories and supports joint, audio-to-video, and other unidirectional generation tasks.Its subsets include image-text pairs, annotated online videos, and the Verse-Ted subset.
- Evaluation: The evaluation covers six generation tasks using video, audio, speech, synchronization, distributional, semantic, production-quality, and consistency metrics.Metrics include MS, AS, ID, FD, KL, CLAP-based scores, WER, and SyncNet confidence.
- Results: UniVerse-1 achieves ID: 0.89 for video identity preservation and PC: 2.49 for audio pitch correlation, while remaining behind leading audio-only models on audio quality.Joint-generation metrics should be interpreted alongside audio-text CLAP because comparison settings differ.
- Ablation Study: The ablation study finds that LQLS improves video quality and identity consistency, while INSS significantly enhances audio generation quality.These findings support the reported contributions of the two strategies.
5 Limitation and Future Work
The study is an initial exploration constrained by computational resources and limited to training on the Wan2.1-1.3B video model. Future work targets larger video foundations and more extensive data curation.
- Limitation: Training was restricted to the Wan2.1-1.3B video model because of computational-resource constraints, limiting performance by the capacity of this base model.The authors identify larger video foundation models as a future direction.
- Future Work: Future work will scale experiments to larger video foundation models and conduct more extensive, refined data curation.The stated objective is to advance open-source audio-video synthesis toward proprietary state-of-the-art performance.
6 Conclusion
UniVerse-1 integrates video and music generation foundations through stitching of experts to enable joint audio-video synthesis. The curated dataset and Verse-Bench support evaluation and reproducible future research.
- Contribution: UniVerse-1 performs joint audio-video synthesis by deeply integrating a video foundation model and a music generation model through stitching of experts.The framework is fine-tuned on a curated dataset.
- Resources: The work introduces Verse-Bench as a comprehensive benchmark and releases the model and code to support comparative research and further innovation.The conclusion presents these resources as part of the paper’s reproducibility and research contribution.
- Conclusion: The authors report that leveraging pre-existing unimodal foundations offers a viable and efficient pathway for building sophisticated multimodal generative models.This is the paper’s stated methodological conclusion.
A.1 Statistical Results
This section presents the Verse-Bench category catalog and its statistical distributions across two datasets and their combination.
- Figures 4–6 visualize statistical results for the two sets and their distributions.Figure 4 covers Sets 1 and 2, while Figures 5 and 6 separately cover Set 1 and Set 2.
- Verse-Bench’s category list is provided in Tables 3, 4, and 5.
- Table 3 reports detailed audio classification statistics for Set 1.
- Table 4 reports detailed audio classification statistics for Set 2.
- Table 5 reports combined audio classification statistics for Set 1 and Set 2.