PaperlayerView exact revision on arXiv

Source-linked AI summary

Wan-S2V: Audio-Driven Cinematic Video Generation

Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, Ke Sun, Linrui Tian, Guangyuan Wang, Qi Wang, Zhongjian Wang, Jiayu Xiao, Sheng Xu, Bang Zhang, Peng Zhang, Xindi Zhang, Zhe Zhang, Jingren Zhou, Lian Zhuo

arXiv:2508.18621v1cs.CV

TL;DR

Audio-driven generation has often focused on single-scene or single-character videos, while film and television call for more complex character motion. Wan-S2V pairs text-guided global motion with audio-driven local motion, and the authors report stronger expressiveness and realism than current approaches.

  • Problem

    Existing audio-driven models primarily target single-scene or single-character videos, while film and television scenarios call for more complex generation.

    Source: §1 · p. 1

  • Method

    Wan-S2V combines text-guided global motion control with audio-driven fine-grained local character motion for complex scenes.

    Source: §1 · pp. 1–2

  • Results

    Wan-S2V surpasses current approaches in expressiveness and realism of generated content.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The authors aim to make high-quality audio-driven video synthesis more accessible and practical through an approach spanning data, training, and optimized inference.

    Source: §6 · p. 9

    Nuanced multi-person interactions and precise camera control driven solely by audio remain formidable challenges.

    Source: §6 · p. 9

Abstract

from arXiv · show

Current state-of-the-art (SOTA) methods for audio-driven character animation demonstrate promising performance for scenarios primarily involving speech and singing. However, they often fall short in more complex film and television productions, which demand sophisticated elements such as nuanced character interactions, realistic body movements, and dynamic camera work. To address this long-standing challenge of achieving film-level character animation, we propose an audio-driven model, which we refere to as Wan-S2V, built upon Wan. Our model achieves significantly enhanced expressiveness and fidelity in cinematic contexts compared to existing approaches. We conducted extensive experiments, benchmarking our method against cutting-edge models such as Hunyuan-Avatar and Omnihuman. The experimental results consistently demonstrate that our approach significantly outperforms these existing solutions. Additionally, we explore the versatility of our method through its applications in long-form video generation and precise video lip-sync editing.

1 INTRODUCTION

Wan-S2V extends audio-driven video generation toward complex film and television scenes by combining text control of global cinematic motion and interactions with audio control of fine-grained character actions. The introduction also presents training-data and training strategies, plus motion-frame token compression for more stable long-video generation.

1 INTRODUCTION

  • Its training combines film and television audiovisual data with talking and singing character videos, then uses staged full-dataset pretraining and high-quality-data fine-tuning.The stated training strategy includes pretraining audio-processing modules and hybrid FSDP with Context Parallel to accelerate large-scale full-parameter training.Source: §1 · pp. 1–2
  • Wan-S2V addresses long-video instability by reducing Motion Frame token counts at different times, enabling more Motion Frames to support stable generation.The introduction identifies the tradeoff between computational complexity and using enough Motion Frames to maintain long-term consistency.Source: §1 · p. 2
  • Wan-S2V targets complex scenes with text-guided global motion and audio-driven fine-grained character movement, including expressions and localized actions.The method builds on Wan while aiming to preserve text-control fidelity alongside audio-driven capabilities.Source: §1 · pp. 1–2
  • The authors describe their training data as large-scale and diverse and report using extensive experiments to validate the model’s effectiveness.The described data includes solo speech and singing videos as well as complex characters in film and television dramas.Source: §1 · p. 2

2 DATA PROCESSING PIPELINE

The pipeline builds a large human-centric video pool through automated screening and manual curation, then applies pose-based filtering, quality checks, audio-visual alignment checks, and detailed captioning.

Data Collection

  • Automated screening of open-source datasets and manual selection of videos with complex human activities yielded an initial pool of millions of human-centric samples.Dataset captions were coarse-grained and often missed nuanced motion, motivating a specialized captioning pipeline.Source: §2 · p. 2

Pose Tracking and Fine-grained Filtering

  • VitPose tracking (Xu et al. (2022)), converted to DWPose (Yang et al. (2023)), provides optional action-aligned control and supports fine-grained dataset filtering.Filtering excludes videos with negligible character presence or without consistently visible faces, and Light-ASD (Liao et al. (2023)) removes clips with unsynchronized or absent active speakers.Source: §2 · pp. 2–3

Video Quality

  • Video quality screening evaluates clarity with Dover (Wu et al. (2023)), motion stability with UniMatch (Xu et al. (2023)), face/hand sharpness, aesthetics, and subtitle occlusion.These checks filter excessive subject/background movement, blurred faces or hands, low aesthetic quality, and subtitles that occlude faces or hands.Source: §2 · p. 3

Dense Video Caption

  • QwenVL2.5-72B (Bai et al. (2025)) generates detailed captions describing camera angles, appearance and specific actions, and background features while avoiding subjective evaluations and emotional interpretations.The captions are intended to help align generation with the input prompt.Source: §2 · pp. 3–4

3 MODEL ARCHITECTURE

Wan-S2V generates audio-synchronized video from a reference image, audio, and prompt by denoising multi-frame latent noise, with aligned audio features and optional motion history conditioning generation.

3 MODEL ARCHITECTURE

  • Given a reference image, audio, and video-content prompt, Wan-S2V denoises multi-frame latent noise into synchronized video while preserving the reference image’s content.At inference, denoising is conditioned on the reference frame, motion frames, audio, and prompt.Source: §3 · p. 4
  • Training uses Flow Matching to predict velocity ϵ − x0 from noisy latents, while inference recovers x0 from xt under reference-frame, motion, audio, and prompt conditioning.Target RGB frames are encoded by a 3D VAE before noise is added at a continuous timestep; Flow Matching is introduced by Lipman et al. (2023).Source: §3 · p. 4
  • A 3D VAE encodes reference, target, and motion frames into visual tokens; optional motion history supports clip continuity, with Frame Pack compressing earlier frames.Frame Pack is introduced by Zhang & Agrawala (2025), and motion-frame conditioning follows Tian et al. (2024).Source: §3 · p. 4
  • Wav2Vec features fused across layers capture rhythmic, emotional, and lexical cues, then causal temporal convolutions compress them into features aligned with video latent frames.The weighted-average layers combine shallow rhythmic and emotional cues with deeper lexical content; Wav2Vec is by Schneider et al. (2019), and the layer weighting follows Tian et al. (2024).Source: §3 · p. 4
  • Audio Blocks segment visual tokens over time and attend between each audio feature and its corresponding visual segment, reducing computation versus full 3D attention while maintaining synchronization.Audio features are passed into each Audio Block, where latent visual tokens are divided along the temporal dimension.Source: §3 · p. 4

4 IMPLEMENTATION

Wan-S2V is trained from a pre-trained Wan model using hybrid parallelism and variable-length video training, followed by successive audio, speech, film-and-television, and high-quality fine-tuning stages.

4 IMPLEMENTATION

  • Training proceeds from audio encoder training to speech-video training, film-and-television plus speech-video training, and high-quality SFT.The model is based on a pre-trained Wan model.Source: §4 · p. 4

4.1 PARALLEL STRATEGY

  • FSDP shards parameters across 8 GPUs in one node, enabling Wan-S2V-14B training with 80GB of memory per GPU.The hybrid parallel strategy combines FSDP Zhao et al. (2023) with Context Parallelism.Source: §4.1 · pp. 4–5
  • Context Parallelism combines RingAttention and Ulysses, reducing an iteration from ∼100 to ∼12 seconds on 8 GPUs.The setup supports models exceeding 16B parameters and video training up to 48 frames at 1024 × 768 resolution.Source: §4.1 · pp. 4–5
  • Variable-length resolution training caps post-patchify token counts at M by resizing or cropping videos that exceed the limit; videos already under M are used unchanged.Source: §4 · p. 4, §4.1 · p. 5

5 EXPERIMENTS

Wan-S2V is built on Wan and is evaluated through qualitative and quantitative comparisons. The reported results highlight improved video quality and motion consistency, with EMO2 leading on hand-keypoint metrics.

5 EXPERIMENTS

  • The Wan-14B model is evaluated on EMTD solo-talking videos against open-source audio-animation and audio-driven DiT models, including EchoMimicV2, MimicMotion, EMO2, FantasyTalking, and Hunyuan-Avatar.EMTD is proposed by Meng et al. (2024); the comparison also includes EchoMimicV2 by Meng et al. (2024), MimicMotion by Zhang et al. (2024), EMO2 by Tian et al. (2025a), FantasyTalking by Wang et al. (2025), and Hunyuan-Avatar.Source: §5 · p. 6, §5.2 · p. 6

5.1 QUALITATIVE EVALUATION

  • Compared with Hunyuan-Avatar and Ominihuman, Wan-S2V better preserves identity during large movements and generates a wider range of motion.Hunyuan-Avatar shows facial distortion and identity shifts, while Ominihuman’s motion often remains close to the reference pose.Source: §5.1 · p. 6, Figure 4 · p. 7
  • With FramePack, Wan-S2V preserves motion direction and speed across clips and maintains item identity, unlike generation without FramePack.FramePack encodes more motion frames; the example shows a paper matching across consecutive clips, whereas its appearance may drift without FramePack.Source: §5.1 · p. 6, Figure 5 · p. 8, Figure 6 · p. 8

5.2 QUANTITATIVE EVALUATION

  • Wan-S2V improves frame metrics (FID, SSIM, PSNR), FVD, and hand metrics HKC/HKV; EMO2 achieves the highest HKC and HKV.HY-Avatar has higher EFID, associated in the passage with its poker-face expressions.Source: Table 1 · p. 9, §5.2 · p. 9

6 CONCLUSION

Wan-S2V advances audio-driven human video generation for film and television by combining text for global motion control with audio for fine-grained character expression. The paper also outlines remaining challenges and future directions for the Vida research series.

6 CONCLUSION

  • Text enables global motion control while audio drives fine-grained character expressions, supporting more expressive and consistent video generation.The approach spans data, training, and optimized inference, with the aim of making high-quality audio-driven video synthesis more accessible and practical.Source: §6 · p. 9
  • Nuanced multi-person interactions and precise camera control driven solely by audio remain formidable challenges.These limitations qualify the approach's progress in complex film and television scenarios.Source: §6 · p. 9
  • Wan-S2V is the first work in the Vida research series, which envisions future work on advanced character control and dynamic dancing generation.The series aims to foster continued research and development in human-centric video synthesis.Source: §6 · p. 9

Related papers

Loading 2508.18621v1…