PaperlayerView exact revision on arXiv

Source-linked AI summary

Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model

Team Seedance, Heyi Chen, Siyan Chen, Xin Chen, Yanfei Chen, Ying Chen, Zhuo Chen, Feng Cheng, Tianheng Cheng, Xinqi Cheng, Xuyan Chi, Jian Cong, Jing Cui, Qinpeng Cui, Qide Dong, Junliang Fan, Jing Fang, Zetao Fang, Chengjian Feng, Han Feng, Mingyuan Gao, Yu Gao, Dong Guo, Qiushan Guo, Boyang Hao, Qingkai Hao, Bibo He, Qian He, Tuyen Hoang, Ruoqing Hu, Xi Hu, Weilin Huang, Zhaoyang Huang, Zhongyi Huang, Donglei Ji, Siqi Jiang, Wei Jiang, Yunpu Jiang, Zhuo Jiang, Ashley Kim, Jianan Kong, Zhichao Lai, Shanshan Lao, Yichong Leng, Ai Li, Feiya Li, Gen Li, Huixia Li, JiaShi Li, Liang Li, Ming Li, Shanshan Li, Tao Li, Xian Li, Xiaojie Li, Xiaoyang Li, Xingxing Li, Yameng Li, Yifu Li, Yiying Li, Chao Liang, Han Liang, Jianzhong Liang, Ying Liang, Zhiqiang Liang, Wang Liao, Yalin Liao, Heng Lin, Kengyu Lin, Shanchuan Lin, Xi Lin, Zhijie Lin, Feng Ling, Fangfang Liu, Gaohong Liu, Jiawei Liu, Jie Liu, Jihao Liu, Shouda Liu, Shu Liu, Sichao Liu, Songwei Liu, Xin Liu, Xue Liu, Yibo Liu, Zikun Liu, Zuxi Liu, Junlin Lyu, Lecheng Lyu, Qian Lyu, Han Mu, Xiaonan Nie, Jingzhe Ning, Xitong Pan, Yanghua Peng, Lianke Qin, Xueqiong Qu, Yuxi Ren, Kai Shen, Guang Shi, Lei Shi, Yan Song, Yinglong Song, Fan Sun, Li Sun, Renfei Sun, Yan Sun, Zeyu Sun, Wenjing Tang, Yaxue Tang, Zirui Tao, Feng Wang, Furui Wang, Jinran Wang, Junkai Wang, Ke Wang, Kexin Wang, Qingyi Wang, Rui Wang, Sen Wang, Shuai Wang, Tingru Wang, Weichen Wang, Xin Wang, Yanhui Wang, Yue Wang, Yuping Wang, Yuxuan Wang, Ziyu Wang, Guoqiang Wei, Wanru Wei, Di Wu, Guohong Wu, Hanjie Wu, Jian Wu, Jie Wu, Ruolan Wu, Xinglong Wu, Yonghui Wu, Ruiqi Xia, Liang Xiang, Fei Xiao, XueFeng Xiao, Pan Xie, Shuangyi Xie, Shuang Xu, Jinlan Xue, Shen Yan, Bangbang Yang, Ceyuan Yang, Jiaqi Yang, Runkai Yang, Tao Yang, Yang Yang, Yihang Yang, ZhiXian Yang, Ziyan Yang, Songting Yao, Yifan Yao, Zilyu Ye, Bowen Yu, Jian Yu, Chujie Yuan, Linxiao Yuan, Sichun Zeng, Weihong Zeng, Xuejiao Zeng, Yan Zeng, Chuntao Zhang, Heng Zhang, Jingjie Zhang, Kuo Zhang, Liang Zhang, Liying Zhang, Manlin Zhang, Ting Zhang, Weida Zhang, Xiaohe Zhang, Xinyan Zhang, Yan Zhang, Yuan Zhang, Zixiang Zhang, Fengxuan Zhao, Huating Zhao, Yang Zhao, Hao Zheng, Jianbin Zheng, Xiaozheng Zheng, Yangyang Zheng, Yijie Zheng, Jiexin Zhou, Jiahui Zhu, Kuan Zhu, Shenhan Zhu, Wenjia Zhu, Benhui Zou, Feilong Zuo

arXiv:2512.13507v3cs.CV

TL;DR

Producing integrated video-audio works calls for precise synchronization and emotional alignment. Seedance 1.5 pro uses a unified multimodal generation model and reports end-to-end acceleration exceeding 10× while preserving performance, alongside high audio-visual consistency and multilingual and dialect lip-sync.

  • Problem

    As video generation moves toward multimodal integration, the challenge is generating audio-visual outputs with precise synchronization and emotional alignment for production-ready works.

    Source: §2 · p. 3

  • Method

    The model combines a unified MMDiT-based joint video-audio architecture and multi-task pretraining with a curated multimodal data framework.

    Source: §1 · p. 2

  • Results

    Exceeding 10× end-to-end acceleration while preserving model performance is reported alongside high audio-visual consistency and multilingual and dialect lip-sync.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The model supports text- and image-conditioned joint audio-video generation as well as unimodal video generation.

    Source: §1 · p. 2

    Mastery of specific vocal styles across opera sub-genres is still evolving.

    Source: §2.2 · p. 8

Abstract

from arXiv · show

Recent strides in video generation have paved the way for unified audio-visual generation. In this work, we present Seedance 1.5 pro, a foundational model engineered specifically for native, joint audio-video generation. Leveraging a dual-branch Diffusion Transformer architecture, the model integrates a cross-modal joint module with a specialized multi-stage data pipeline, achieving exceptional audio-visual synchronization and superior generation quality. To ensure practical utility, we implement meticulous post-training optimizations, including Supervised Fine-Tuning (SFT) on high-quality datasets and Reinforcement Learning from Human Feedback (RLHF) with multi-dimensional reward models. Furthermore, we introduce an acceleration framework that boosts inference speed by over 10X. Seedance 1.5 pro distinguishes itself through precise multilingual and dialect lip-syncing, dynamic cinematic camera control, and enhanced narrative coherence, positioning it as a robust engine for professional-grade content creation. Seedance 1.5 pro is now accessible on Volcano Engine at https://console.volcengine.com/ark/region:ark+cn-beijing/experience/vision?type=GenVideo.

1 Introduction

Seedance 1.5 pro is presented as a foundational model for native joint video-audio generation across text- and image-guided tasks. Its contributions span multimodal data curation, a unified architecture, post-training, inference acceleration, and synchronized, expressive video generation.

Unified multimodal joint generation architecture

  • Seedance 1.5 pro supports native joint video-audio synthesis for text- and image-guided tasks, with a unified MMDiT-based architecture designed for cross-modal synchronization and semantic consistency.Multi-task pre-training supports T2VA, I2VA, T2V, and I2V.Source: §1 · p. 2

Comprehensive audio-visual data framework

  • The model uses a multi-stage data-curation pipeline, professional-grade audio and video captioning, and scalable infrastructure to prioritize coherence, expressive motion, and curriculum-based scheduling.The framework is engineered for large-scale multimodal data processing.Source: §1 · p. 2

Meticulous post-training optimization

  • Nearly 3× faster training resulted from infrastructure optimizations to an audio-video RLHF pipeline following high-quality-dataset SFT.A multidimensional reward model targets motion quality, visual aesthetics, and audio fidelity in T2V and I2V.Source: §1 · p. 2

Efficient inference acceleration

  • Over 10× end-to-end inference acceleration was achieved through multi-stage distillation, quantization, and parallelism while preserving model performance.Distillation reduces the number of function evaluations required during generation.Source: §1 · p. 2

Audio-visual synchronization and multilingual support

  • Seedance 1.5 pro improves lip-motion, intonation, and performance-rhythm alignment, with native multilingual and dialect support that captures vocal prosody and emotional tension.The introduction describes natural performance in stylized narratives such as comedy and animation.Source: §1 · p. 3

Cinematic control and narrative coherence

  • The model offers autonomous cinematic camera control, including long takes and dolly zooms, and strengthens narrative-context analysis and coordination across audio-visual segments.It also supports cinematic transitions and professional color grading.Source: §1 · p. 3

2 Evaluation · Appendix

The evaluation framework treats audio–visual synchronization and emotional alignment as central to production-ready generation. Seedance 1.5 pro shows advantages in video and audio comparisons and in Chinese-language, cinematic, and theatrical storytelling scenarios.

2.1 Comprehensive Evaluation

  • The evaluation combines expert human assessment with a multidimensional benchmark built from real-world prompts and expanded for industry scenarios and audio.SeedVideoBench-1.5 was developed with professional film directors and experts from film production, cinematography, and design.Source: §2.1 · p. 3

2.1.1 SeedVideoBench 1.5

  • SeedVideoBench 1.5 evaluates video subjects, motion, interactions, and camera movement alongside application scenarios and fine-grained voice, vocal-attribute, and non-speech audio labels.For I2V, audio synthesis is conditioned on visual cues in the reference image to support semantic consistency.Source: §2.1.1 · pp. 3–4

2.1.2 Video Evaluation Metrics

  • Video evaluation adds vividness across action, camera movement, atmosphere, and emotion, and judges prompt following by underlying intent rather than keyword matching.The vividness criteria include expressive action and cinematic camera movement; narrative tasks permit intent-aligned creative flexibility while preserving the core request.Source: §2.1.2 · p. 4

2.1.3 Audio Evaluation Metrics

  • Audio evaluation measures instruction fidelity, acoustic quality, audio–visual synchronization, and expressiveness, including speech-to-lip timing and emotional contribution to narrative coherence.It also considers sound-effect alignment with visible events, vocal and non-vocal artifacts, and the emotional inflection of speech and background music.Source: §2.1.3 · p. 5

2.1.4 Human Evaluation

  • Seedance 1.5 pro improves over Seedance 1.0 Pro, leads T2V instruction following, and is competitive in visual aesthetics, motion dynamics, and I2V.Source: Figure 3 · p. 5, §2.1.4 · pp. 5–6, Figure 4 · p. 6
  • Seedance 1.5 pro excels in Chinese vocal generation and lip–audio synchronization, surpassing Veo 3.1 and Kling 2.6 on synchronization while maintaining controlled expressiveness relative to Sora 2.Chinese dialogue, dialects, and monologues show articulatory clarity with few syllable-dropping or mispronunciation artifacts; Sora 2 has stronger emotional expressiveness, while Seedance maintains alignment without over-exaggeration.Source: Figure 5 · p. 6, §2.1.4 · pp. 6–7, Figure 6 · p. 7

2.2 Potential Advantages in Application Scenarios

  • The model supports Chinese film, micro-drama, and theatrical storytelling with dialect-rich dialogue, continuous-shot lip synchronization, complex camera operations, and genre-aligned scene development.Examples include Sichuanese, Taiwan Mandarin, Cantonese, and Shanghainese, as well as orbital, arc, and tracking shots with visual continuity.Source: §2.2 · pp. 7–8
  • In opera-inspired scenes, the model captures operatic speech cadence and Eastern performance gestures, though mastery of specific vocal styles remains in development; close-ups preserve facial micro-expression and character continuity.Source: §2.2 · pp. 7–8

Related papers

Loading 2512.13507v3…