PaperlayerView exact revision on arXiv

Source-linked AI summary

Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model

SII-GAIR, Sand. ai, :, Ethan Chern, Hansi Teng, Hanwen Sun, Hao Wang, Hong Pan, Hongyu Jia, Jiadi Su, Jin Li, Junjie Yu, Lijie Liu, Lingzhi Li, Lyumanshan Ye, Min Hu, Qiangang Wang, Quanwei Qi, Steffi Chern, Tao Bu, Taoran Wang, Teren Xu, Tianning Zhang, Tiantian Mi, Weixian Xu, Wenqiang Zhang, Wentai Zhang, Xianping Yi, Xiaojie Cai, Xiaoyang Kang, Yan Ma, Yixiu Liu, Yunbo Zhang, Yunpeng Huang, Yutong Lin, Zewei Tao, Zhaoliang Liu, Zheng Zhang, Zhiyao Cen, Zhixuan Yu, Zhongshu Wang, Zhulin Hu, Zijin Zhou, Zinan Guo, Yue Cao, Pengfei Liu

arXiv:2603.21986v1cs.CV

TL;DR

Open-source models for synchronized audio-video generation remain limited, particularly in combining generation quality, multilingual support, inference efficiency, and a simple scalable architecture. daVinci-MagiHuman uses a single-stream Transformer for text, video, and audio, and its distilled model generates a 5-second 256p video in 2 seconds on one H100 GPU.

  • Problem

    Open-source progress on synchronized audio-video generation remains limited, especially for models combining generation quality, multilingual support, inference efficiency, and a simple scalable architecture.

    Source: §1 · p. 2

  • Method

    The model uses a single-stream Transformer to process text, video, and audio tokens in a shared backbone with unified self-attention layers.

    Source: §2 · p. 3

  • Results

    2 seconds: the distilled model generates a 5-second 256p video on one H100 GPU; a 5-second 1080p video takes 38 seconds.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents daVinci-MagiHuman as a strong open-source foundation for audio-video generation.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation. daVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer that processes text, video, and audio within a unified token sequence via self-attention only. This single-stream design avoids the complexity of multi-stream or cross-attention architectures while remaining easy to optimize with standard training and inference infrastructure. The model is particularly strong in human-centric scenarios, producing expressive facial performance, natural speech-expression coordination, realistic body motion, and precise audio-video synchronization. It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French. For efficient inference, we combine the single-stream backbone with model distillation, latent-space super-resolution, and a Turbo VAE decoder, enabling generation of a 5-second 256p video in 2 seconds on a single H100 GPU. In automatic evaluation, daVinci-MagiHuman achieves the highest visual quality and text alignment among leading open models, along with the lowest word error rate (14.60%) for speech intelligibility. In pairwise human evaluation, it achieves win rates of 80.0% against Ovi 1.1 and 60.9% against LTX 2.3 over 2000 comparisons. We open-source the complete model stack, including the base model, the distilled model, the super-resolution model, and the inference codebase.

1. Introduction

Against limited open-source progress in synchronized audio-video generation, daVinci-MagiHuman combines a simple single-stream architecture with human-centric, multilingual generation and fast inference.

1 Introduction

  • A shared-weight single-stream Transformer processes text, video, and audio while avoiding specialized multi-stream designs and remaining easy to optimize.The architecture is designed to work jointly with training and inference infrastructure.Source: §1 · p. 2
  • The model performs strongly in expressive human generation and coordinates voice, facial expression, body motion, and audio-video synchronization across multiple languages.Supported languages include Mandarin and Cantonese Chinese, English, Japanese, Korean, German, and French, with additional languages also supported.Source: §1 · p. 2
  • In 2 seconds for 5-second 256p and 38 seconds for 5-second 1080p, the distilled model generates video on one H100 GPU.Latent-space super-resolution reduces the computation required for high-resolution generation.Source: §1 · p. 2
  • The release includes the complete model stack: base and distilled models, the super-resolution model, and the inference codebase.The authors present it as a practical, extensible foundation for future audio-video generation research.Source: §1 · p. 2

2 Methodology

daVinci-MagiHuman uses a shared single-stream Transformer to jointly denoise video and audio alongside text, aiming to simplify multimodal modeling and make the system easier to optimize. Its design combines modality-specific boundary layers with shared middle layers, timestep-free denoising, and per-head attention gating.

Single-Stream Transformer

  • Dual-stream video models commonly process text and video in separate branches, while audio-video systems often add separate modality pathways and fusion blocks (Wan et al., 2025; Kong et al., 2024; HaCohen et al., 2026; Low et al., 2025).The paper says these designs increase architectural complexity and create irregular computation patterns that make implementation and optimization harder.Source: §2 · pp. 2–3
  • The 15B-parameter, 40-layer backbone jointly denoises video and audio from a unified sequence of text tokens, a reference-image latent, and noisy video and audio tokens, using self-attention only.A latent-space super-resolution stage can further refine video at higher output resolutions.Source: Figure 2 · p. 3, §2 · p. 3
  • The shared-backbone design represents text, video, and audio tokens together and uses unified self-attention layers, reducing engineering complexity and easing joint optimization of the model and infrastructure.It is presented as an alternative to maintaining separate pathways for each modality.Source: §2 · p. 3

Sandwich Architecture Layout

  • The first and last 4 layers use modality-specific projections and RMSNorm parameters, while the middle 32 layers share the main Transformer parameters across modalities.This sandwich layout keeps modality-sensitive processing near the boundaries while placing most computation in a common representation space.Source: §2 · p. 3, Figure 2 · p. 3

Timestep-Free Denoising

Per-Head Gating

  • Each attention block uses a learned sigmoid gate for every head to modulate its attention output before the output projection.The architecture figure specifies per-head gating throughout the Transformer blocks.Source: §2 · p. 3, Figure 2 · p. 3

3. Evaluation

daVinci-MagiHuman leads the reported visual-quality and text-alignment results and speech intelligibility, is preferred in pairwise human evaluation, and uses a staged inference pipeline to produce high-resolution video efficiently. The evaluation also describes shared conditioning and inference techniques used in that pipeline.

3. Evaluation

  • Denoising video and audio tokens, text, and optional image conditions share one latent/token space and model rather than task-specific conditioning branches.This unified interface supports multiple conditioning and generation settings without dedicated conditioning modules.Source: §3 · p. 4
  • Latent-space super-resolution refines low-resolution video and audio latents jointly, updating video while reusing noised base-stage audio as auxiliary input.The video latent is upsampled with trilinear interpolation, noised, and refined using 5 additional denoising steps; the 1080p model uses local attention in many layers to control attention cost. Keeping audio as input helps preserve lip synchronization when the base video is coarse.Source: §3 · p. 4
  • DMD-2 distillation enables generation in 8 denoising steps without CFG, and MagiCompiler provides around 1.2× speedup on H100.The inference stack also replaces the original video decoder with a lightweight re-trained Turbo VAE decoder to reduce decoding overhead; the Wan2.2 VAE is used for encoding.Source: §3 · p. 4
  • daVinci-MagiHuman achieves the best visual quality and text alignment, and lowest WER (14.60%) versus Ovi 1.1 (40.45%) and LTX 2.3 (19.23%).LTX 2.3 performs best on physical consistency, while daVinci-MagiHuman remains competitive on that metric.Source: Table 1 · p. 4, §3 · p. 5
  • daVinci-MagiHuman wins 80.0% of comparisons against Ovi 1.1 and 60.9% against LTX 2.3 across 2,000 pairwise judgments.The evaluation used 10 raters and assessed overall audio-video quality, synchronization, and naturalness.Source: §3 · p. 5, Figure 3 · p. 5
  • The full pipeline generates a 5-second 1080p video in 38.4 seconds on a single H100 GPU.The base stage runs at 256p using the distilled model; super-resolution and decoding account for the added cost at higher output resolutions.Source: §3 · p. 5, Table 2 · p. 5

Related papers

Loading 2603.21986v1…