Source-linked AI summary
Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model
SII-GAIR, Sand. ai, :, Ethan Chern, Hansi Teng, Hanwen Sun, Hao Wang, Hong Pan, Hongyu Jia, Jiadi Su, Jin Li, Junjie Yu, Lijie Liu, Lingzhi Li, Lyumanshan Ye, Min Hu, Qiangang Wang, Quanwei Qi, Steffi Chern, Tao Bu, Taoran Wang, Teren Xu, Tianning Zhang, Tiantian Mi, Weixian Xu, Wenqiang Zhang, Wentai Zhang, Xianping Yi, Xiaojie Cai, Xiaoyang Kang, Yan Ma, Yixiu Liu, Yunbo Zhang, Yunpeng Huang, Yutong Lin, Zewei Tao, Zhaoliang Liu, Zheng Zhang, Zhiyao Cen, Zhixuan Yu, Zhongshu Wang, Zhulin Hu, Zijin Zhou, Zinan Guo, Yue Cao, Pengfei Liu
TL;DR
Open-source audio-video generation still faces a challenge in combining strong quality, multilingual support, inference efficiency, and architectural simplicity. daVinci-MagiHuman addresses this with a single-stream Transformer for text, video, and audio, achieving strong human-centric and multilingual generation with fast inference, including 5-second 256p video in 2 seconds on one H100 GPU.
Problem
Open-source audio-video generation remains limited in combining strong generation quality, multilingual support, inference efficiency, and a simple, scalable architecture.
Method
daVinci-MagiHuman uses a single-stream Transformer that processes text, video, and audio through a shared backbone and unified self-attention stack.
Results
The model delivers strong human-centric and multilingual audio-video generation while its distilled system generates a 5-second 256p video in 2 seconds on a single H100 GPU.
Takeaways & Limitations
daVinci-MagiHuman provides an open-source foundation combining architectural simplicity, human-centric quality, multilingual capability, and fast inference.
Abstract
from arXiv · showhide
We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation. daVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer that processes text, video, and audio within a unified token sequence via self-attention only. This single-stream design avoids the complexity of multi-stream or cross-attention architectures while remaining easy to optimize with standard training and inference infrastructure. The model is particularly strong in human-centric scenarios, producing expressive facial performance, natural speech-expression coordination, realistic body motion, and precise audio-video synchronization. It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French. For efficient inference, we combine the single-stream backbone with model distillation, latent-space super-resolution, and a Turbo VAE decoder, enabling generation of a 5-second 256p video in 2 seconds on a single H100 GPU. In automatic evaluation, daVinci-MagiHuman achieves the highest visual quality and text alignment among leading open models, along with the lowest word error rate (14.60%) for speech intelligibility. In pairwise human evaluation, it achieves win rates of 80.0% against Ovi 1.1 and 60.9% against LTX 2.3 over 2000 comparisons. We open-source the complete model stack, including the base model, the distilled model, the super-resolution model, and the inference codebase.
1. Introduction
daVinci-MagiHuman is an open-source audio-video generator that combines a simple single-stream architecture with strong human-centric, multilingual generation and efficient inference. The complete model stack is released to support practical and extensible research.
- Open-source progress remains limited in combining strong generation quality, multilingual support, inference efficiency, and a simple scalable architecture.
- 2 seconds enables generation of a 5-second 256p video on a single H100 GPU through the single-stream backbone, latent-space super-resolution, and inference-level optimization.
- The complete model stack—including the base model, distilled model, super-resolution model, and inference codebase—is fully open-sourced.
- A single-stream Transformer processes text, video, and audio within shared weights, avoiding heavily specialized multi-stream architectures while remaining easy to optimize.
- The model is particularly strong at expressive human generation, including emotion, speech-expression coordination, facial performance, body motion, and audio-video synchronization.
- Strong spoken audio-video generation is supported across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French, plus additional languages.
2 Methodology
The methodology uses a unified single-stream Transformer to jointly denoise video and audio while reducing architectural and infrastructure complexity. Its design combines shared multimodal processing with boundary-specific layers, timestep-free denoising, and per-head gating.
- Separate multimodal pathways and fusion modules create architectural, implementation, and optimization complexity.
- A single-stream Transformer represents text, video, and audio tokens in one shared backbone and processes them with unified self-attention layers.
- The 15B-parameter, 40-layer backbone jointly denoises video and audio at every step.
- Sandwich Architecture Layout: The sandwich layout uses modality-specific projections and RMSNorm parameters in the first and last 4 layers, while the middle 32 layers share Transformer parameters.
- Timestep-Free Denoising: Timestep-free denoising infers the denoising state directly from noisy video and audio latents without a dedicated timestep pathway.
- Per-Head Gating: Per-head gating applies a learned sigmoid gate to each attention head before output projection, improving numerical stability and representability with minimal overhead.
3. Evaluation
The evaluation compares daVinci-MagiHuman with Ovi 1.1 and LTX 2.3 across quality, human preference, and inference efficiency. Results show strong automatic and human-evaluated performance alongside a fast distilled inference pipeline.
- Inference Efficiency: Latent-space super-resolution generates low-resolution video and audio latents first, then refines them at higher resolution to reduce computation.
- Inference Efficiency: Eight denoising steps without CFG reduce inference cost while maintaining strong generation quality.Latency measurements in Section 3 use the distilled model unless otherwise specified.
- The evaluation compares daVinci-MagiHuman with Ovi 1.1 and LTX 2.3 across automatic quality, human preference, and inference efficiency.
- Quantitative Quality Benchmark: 14.60% WER is the lowest reported speech-intelligibility error rate, versus 40.45% for Ovi 1.1 and 19.23% for LTX 2.3.daVinci-MagiHuman also achieves the best visual quality and text alignment scores, while LTX 2.3 performs best on physical consistency.
- Human Evaluation: 80.0% against Ovi 1.1 and 60.9% against LTX 2.3 are daVinci-MagiHuman’s pairwise human-evaluation win rates over 2,000 comparisons.
- Inference Efficiency: The base stage uses a distilled model at 256p, while super-resolution and Turbo VAE decoding provide higher-resolution outputs on one H100 GPU.The reported pipeline latency is 38 seconds for a 5-second video at the highest described output setting.
A Authors
The paper lists its authors alphabetically, excluding the project leaders, and separately identifies Yue Cao and Pengfei Liu as project leaders.
- The authors are listed in alphabetical order, excluding the project leaders.
- Yue Cao and Pengfei Liu are identified as project leaders.