Source-linked AI summary
Wan-S2V: Audio-Driven Cinematic Video Generation
Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, Ke Sun, Linrui Tian, Guangyuan Wang, Qi Wang, Zhongjian Wang, Jiayu Xiao, Sheng Xu, Bang Zhang, Peng Zhang, Xindi Zhang, Zhe Zhang, Jingren Zhou, Lian Zhuo
TL;DR
Existing audio-driven character animation methods mainly address speech and singing, while complex film and television scenarios require richer motion and cinematic control. Wan-S2V combines text-guided global motion with audio-driven local detail, and experiments report stronger expressiveness, realism, identity consistency, and motion consistency than existing approaches.
Problem
Existing audio-driven models primarily target speech, singing, single-scene, or single-character videos rather than complex film and television scenarios.
Method
Wan-S2V combines text for global video dynamics with audio for fine-grained expressions and local actions, using aligned audio injection and motion-frame token reduction.
Results
Qualitative and quantitative comparisons report that Wan-S2V surpasses existing approaches in expressiveness, realism, identity consistency, motion diversity, and multi-clip motion consistency.
Takeaways & Limitations
Wan-S2V extends audio-driven human video generation toward more expressive and consistent film and television scenarios, including long-video generation.
Abstract
from arXiv · showhide
Current state-of-the-art (SOTA) methods for audio-driven character animation demonstrate promising performance for scenarios primarily involving speech and singing. However, they often fall short in more complex film and television productions, which demand sophisticated elements such as nuanced character interactions, realistic body movements, and dynamic camera work. To address this long-standing challenge of achieving film-level character animation, we propose an audio-driven model, which we refere to as Wan-S2V, built upon Wan. Our model achieves significantly enhanced expressiveness and fidelity in cinematic contexts compared to existing approaches. We conducted extensive experiments, benchmarking our method against cutting-edge models such as Hunyuan-Avatar and Omnihuman. The experimental results consistently demonstrate that our approach significantly outperforms these existing solutions. Additionally, we explore the versatility of our method through its applications in long-form video generation and precise video lip-sync editing.
1 INTRODUCTION
Wan-S2V extends audio-driven video generation beyond single-scene or single-character settings toward complex film and television scenarios. It combines text-guided global motion control with audio-driven fine-grained character motion and uses long-video stabilization, diverse training data, and extensive validation.
- Current audio-driven models mainly target single-scene human videos or single-character driving, leaving complex production scenarios insufficiently addressed.
- Text specifies global dynamics such as camera movement, character trajectories, and interactions, while audio controls expressions and localized actions.
- Motion-frame token reduction addresses long-video consistency by allowing more motion frames without the same computational burden.
- Wan-S2V enables natural character movements in challenging scenes through text-guided global motion and audio-driven fine-grained local motion.
- The model is trained and validated using a large-scale dataset spanning human speech, singing, and complex film and television character videos.
2 DATA PROCESSING PIPELINE
The data pipeline builds a large human-centric video pool through automated screening and manual curation, then refines it using pose tracking, quality assessment, and dense captions. These stages target human activities, usable motion, video quality, and detailed generation conditions.
- The pipeline seeks videos containing one or more human characters engaged in specific activities.
- Automated screening uses coarse caption filtering, while a specialized captioning pipeline addresses captions that miss nuanced character activities.
- Manual curation adds high-quality videos featuring speaking, singing, dancing, and other intentional complex human activities.
- Pose tracking with VitPose and DWPose supplies a multimodal control signal and supports fine-grained dataset screening.
- Video quality is assessed through clarity and motion-stability analysis alongside additional quality metrics.
- Dense captions describe camera angles and physical appearance while avoiding subjective evaluations and emotional interpretations.
3 MODEL ARCHITECTURE
Wan-S2V generates audio-synchronized video from a reference image, audio, and prompt by denoising latent video representations. Its architecture encodes temporally aligned audio features and injects them into segmented visual-token processing.
- Given a reference image, audio, and prompt, the model generates synchronized video while preserving the reference image’s content.
- Training encodes target RGB frames into latent space, adds noise at a continuous timestep, and applies Flow Matching.
- The model predicts velocity from the noisy latent representation, then inference recovers the clean latent under reference, motion-frame, audio, and prompt conditions.
- Reference, target, and optional motion frames are encoded into visual tokens, with motion frames providing previous information for clip continuity.
- Wav2Vec features combine rhythmic, emotional, and lexical cues before causal temporal compression produces audio features aligned with latent video frames.
- Audio Blocks attend between each audio feature segment and its corresponding visual-token segment instead of using full 3D attention.
4 IMPLEMENTATION
The implementation uses hybrid parallelism and variable-length resolution training to support large Wan-S2V models and diverse video outputs. FSDP shards parameters, Context Parallelism accelerates computation, and token limits control resolution processing.
- Parallel Strategy: Wan-S2V uses hybrid-parallel training with FSDP and Context Parallelism for large-scale, full-parameter training.
- Parallel Strategy: FSDP shards parameters across 8 GPUs, enabling training of the Wan-S2V-14B model with 80GB of memory per GPU.
- Parallel Strategy: Context Parallelism combines RingAttention and Ulysses, reducing iteration time from ∼100 seconds to ∼12 seconds on 8 GPUs.
- Variable-Length Training: Variable-length resolution training uses post-patchify token count and resizes or crops videos exceeding the maximum limit M.
5 EXPERIMENTS
Experiments compare Wan-S2V with existing audio-driven video generation methods using qualitative and quantitative evaluations, including long-term consistency and item identity preservation. The results show stronger visual quality, motion dynamics, identity consistency, and long-video coherence than the compared approaches.
- Experimental Setup: Wan-S2V-14B is built on Wan-14B and evaluated against existing audio-driven video generation models.The comparisons include Hunyuan-Avatar, OmniHuman, and other open-source audio-animation methods on quantitative and qualitative evaluations.
- Overall Results: Wan-S2V surpasses current approaches in expressiveness and realism across quantitative metrics and visual results.The reported advantage covers generated-content realism and expressiveness in comprehensive comparisons.
- Qualitative Comparison: Wan-S2V preserves identity during highly dynamic motion and generates a wider range of motion than Hunyuan-Avatar and OmniHuman.Hunyuan-Avatar shows facial distortions and identity shifts, while OmniHuman often remains close to the reference image’s static pose.
- Long-Term Consistency: Wan-S2V maintains both the direction and speed of motion across multiple clips, whereas OmniHuman fails to preserve the motion trend.The comparison uses a scene involving a train moving coherently across consecutive clips.
- Long-Term Consistency: FramePack encodes more motion frames to preserve long-term temporal dependencies and maintain item identity across consecutive video clips.A generated paper matches the previous clip’s appearance, while removing FramePack can cause substantial object-appearance drift.
- Quantitative Evaluation: Wan-S2V improves frame quality using FID, SSIM, and PSNR, lowers FVD for video quality, and achieves clearer hand shapes and more diverse hand motions.EMO2 records the highest HKC and HKV, while HY-Avatar’s poker-face expressions produce a higher EFID.
6 CONCLUSION
The conclusion presents Wan-S2V as an advance for audio-driven human video generation in film and television scenarios. It attributes expressiveness and consistency to combining text-based global motion control with audio-driven fine-grained expressions, while identifying unresolved cinematic challenges.
- Conclusion: Wan-S2V combines text for global motion control with audio for fine-grained character expressions to generate more expressive and consistent videos.The approach addresses audio-driven human video generation in complex film and television scenarios.
- Limitations: Nuanced multi-person interactions and precise camera control driven solely by audio remain formidable challenges.The conclusion identifies these as unresolved complexities in film and television generation.
7 CONTRIBUTORS
The paper lists its contributors alphabetically by last name and provides the complete contributor list.
- Contributors: All contributors are listed in alphabetical order by their last names.The paper then provides the full list of named contributors.
- Contributors: The contributor list includes Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, and the remaining named authors.The complete list contains 23 contributors.