Source-linked AI summary
WanSong v1.0 Technical Report
Binghui Chen, Pandeng Li, Yu Liu, Jingren Zhou
TL;DR
Long-form music generation still faces efficiency, quality, and controllability challenges, especially in autoregressive or multi-stage systems. WanSong addresses these challenges with an end-to-end pure-diffusion model that generates multilingual songs and separated vocal and background-music stems, achieving strong objective-metric performance across its benchmark.
Problem
Existing music-generation methods face efficiency and perceptual-consistency challenges for long-form audio, while many rely on autoregressive or multi-stage pipelines.
Method
WanSong is an end-to-end pure-diffusion model that treats audio as continuous tokens and independently models vocals and background music for separated stem output.
Results
WanSong achieves strong performance across objective metrics on its benchmark, with efficient inference enabled through step distillation.
Takeaways & Limitations
WanSong demonstrates a practical diffusion-based direction for high-fidelity, multilingual, long-duration generation with editable vocal and background-music outputs.
Abstract
from arXiv · showhide
Music generation foundation models have recently attracted significant industry attention. However, achieving efficient generation and high-fidelity long-form audio while supporting controllability remains challenging. To address these needs, we present \textbf{WanSong}, a simple yet powerful approach for long-form, commercial-grade song generation. Unlike autoregressive (AR) and cascaded multi-stage pipelines (\eg, AR followed by diffusion), \textbf{WanSong} is a pure diffusion-based model that directly generates high-fidelity, multilingual songs up to 5 minutes and outputs dual stems (vocals and background music) in a single run. In addition, our diffusion framework enables faster inference through step-distillation, and offers an efficient pathway for fine-tuning and customization to support downstream editing tasks.
1 Introduction
WanSong v1.0 addresses the efficiency and long-form quality challenges of AR-dominant music generation with an end-to-end, single-stage pure diffusion framework. It treats audio as continuous tokens and supports controllable, multilingual, high-fidelity, long-duration song generation with independently modeled vocal and BGM stems.
- Motivation: AR-based music generation can challenge inference efficiency and consistent perceptual quality for long-form audio.
- Approach: WanSong replaces multi-stage AR-dominant pipelines with an end-to-end, single-stage pure diffusion model that directly models audio as continuous tokens.
- Architecture: The framework concatenates text and audio tokens into one unified sequence and feeds them into a refined hybrid-MMDit backbone conditioned on an LLM text captioner.
- Capabilities: WanSong v1.0 is presented as a commercial-grade text-to-music model supporting multilingual, high-fidelity, long-duration song generation.
- Dual-stem modeling: WanSong independently models vocals and BGM to prevent cross-interference and directly output separated stems for more convenient post-production editing.
2 Data
WanSong builds its training data through a five-stage curation pipeline covering collection, pre-processing, filtering, captioning, and sampling. The resulting training set contains more than 6 million hours of multilingual song data.
- Data curation pipeline: The data curation pipeline comprises collection, pre-processing, filtering, captioning, and sampling to produce high-quality, balanced, and diverse training data.The pipeline is designed because training-data quality and diversity strongly determine generative-model performance.
- Data collection: The collected corpus spans multiple languages, musical styles, vocal songs, instrumental music, and a wide range of song durations.This collection strategy targets broad coverage across languages, musical content, and duration.
- Data pre-processing: Each raw audio file undergoes classification, quality assessment, structural parsing, VAD, language classification, and track separation, yielding a candidate pool with diverse annotations.The multi-stage pre-processing pipeline addresses noise in the raw collected audio dataset.
- Data filtering and sampling: Filtering jointly considers audio category and quality to construct a high-quality, balanced sub-dataset, while fine-grained sampling balances categories throughout training.The labeling taxonomy also supports flexible, adaptive scheduling across training phases.
- Captioning and dataset scale: More than 6 million hours of multilingual song data comprise the resulting training set.Captioning annotates musical style, vocals, instruments, mode and key, language, reverb, emotions, lyrics, and more.
3 Method
WanSong uses a continuous stereo audio VAE, a hybrid shared-parameter diffusion transformer, and dual-stem output modeling to generate detailed, controllable audio representations. Training proceeds through staged flow matching and human-preference alignment across musicality, lyric accuracy, and prompt alignment.
- Audio VAE: The continuous 1-D VAE compresses 44.1 kHz stereo audio into a 64-channel latent stream at approximately 43.1 Hz, using a downsampling factor of 1024.The representation retains more time-frequency detail than Stable Audio 2 by halving the compression ratio, supporting sharp transients and high-fidelity source separation.
- Audio VAE: The VAE is trained adversarially with multi-resolution STFT magnitude, discriminator feature-matching, hinge adversarial, and KL-divergence losses.The magnitude loss operates on both Mid-Side and Left-Right channels, while the KL term uses a standard normal prior.
- Diffusion Backbone: Text tokens and dual-stem audio tokens are concatenated and packed across samples before entering a hybrid transformer with fully shared AdaLN parameters.The architecture processes modalities without separate parameters in every layer and is designed to improve token throughput and utilization.
- Dual-Stem Modeling: The model independently produces vocal and background-music tokens through joint-learning-in-layer dual-stem modeling to balance phoneme accuracy against background-music restoration.Larger CFG improves phoneme accuracy but suppresses background music, whereas smaller CFG restores background music better but degrades vocal accuracy.
- Training Strategy: Pre-training uses 90-second training, 300-second training, and supervised fine-tuning stages within a flow-matching framework for audio denoising.Training samples audio latents, Gaussian noise, and timesteps from a logit-normal distribution, then predicts the ground-truth velocity with an MSE objective balancing vocal and background-music losses.
- Preference Alignment: RLHF aligns generation with human preferences for musicality, lyric accuracy, and prompt alignment using dimension-specific reward models, DPO, and ReFL.Human annotations compare positive and negative samples for each dimension, providing reward supervision for subsequent reinforcement learning.
4 Evaluation
WanSong is evaluated through VAE reconstruction fidelity on music and speech benchmarks, alongside multilingual long-form song-generation metrics and compression-ratio ablations. Results show gains from architectural and domain-balancing choices, while lower compression substantially improves fidelity.
- VAE Reconstruction Evaluation: The VAE evaluation uses 200 held-out Wan-song music clips and 2,000 randomly sampled SeedTTS speech clips, reporting STFT distance, mel-spectrogram distance, and scale-invariant SDR.STFT and mel-spectrogram distances are lower-is-better metrics, while SI-SDR is higher-is-better.
- VAE Reconstruction Evaluation: At compression 2048, WanSong outperforms Stable Audio 2 across all metrics on both datasets, including −19% STFT and +0.28 dB SI-SDR on music.The reported speech gains are −22% STFT and +1.72 dB SI-SDR.
- VAE Reconstruction Evaluation: Reducing compression from 2048 to 1024 produces +2.86 dB SI-SDR on music and +4.27 dB on SeedTTS, the largest overall fidelity improvement.The results attribute this improvement to the finer latent grid capturing sharp, fine-grained transient details.
- Song Generation Evaluation: The generation benchmark contains 200 approximately four-minute samples spanning Chinese, English, Japanese, and Korean across more than ten Level-1 musical genres.Evaluation uses Pronunciation Error Rate, Songeval, SongBench, and Muq text alignment.
- Compression-Ratio Ablation: In the 90s compression ablation, compression 2048 with patch size 1 yields 19.2% PER and Quality score 4 of 2.1, while the 1024-plus-patch-2 setup yields 20.6% PER.The experiments compare compression 2048, compression 1024 with patch size 2, and compression 1024 on the 90s benchmark.
5 Related Work
Prior song-generation systems target coherent vocals and background music from lyrics and style prompts, using multi-stage autoregressive, token-based, or diffusion pipelines. Recent work also explores chain-of-thought reasoning, while several approaches remain complex or dependent on discrete tokens and multi-stage generation.
- Song-generation systems aim to produce coherent vocals and background music from lyrics and style prompts.
- MusicLM uses a multi-stage sequence-to-sequence pipeline, while SongGen, Yue, and LeVo train autoregressive models on discrete tokens.
- SongCreator employs a U-Net diffusion model but still requires discrete tokens and a multi-stage generation pipeline, which can be complex.
- MusicCot explores chain-of-thought to enhance the reasoning ability of songs.
6 Conclusion
WanSong v1.0 presents an end-to-end pure diffusion framework for long-form, commercial-grade music generation, combining strong fidelity with efficient inference. Its dual-stem modeling, RLHF alignment, and benchmark results support practical multilingual generation with editable vocal and BGM outputs.
- Framework: WanSong v1.0 uses a single-stage, end-to-end pure diffusion pipeline that models audio as continuous tokens for long-form, commercial-grade music generation.The framework differs from autoregressive and cascaded AR+diffusion systems.
- Framework: Step distillation enables efficient inference while preserving strong audio fidelity.
- Dual-stem modeling: A dual-stem token scheme with joint learning within each block reduces vocal–BGM mutual interference and directly outputs demixed vocals and BGM stems.The design addresses vocal/BGM imbalance and CFG’s inability to preserve vocal accuracy and BGM strength simultaneously, simplifying post-production and editing.
- Preference alignment: RLHF aligns generation with human preferences across musicality, lyric accuracy, and prompt alignment.
- Evaluation: WanSong achieves strong objective-metric performance on the WanSong benchmark, validating its diffusion framework and dual-stem modeling strategy.
- Overall contribution: WanSong v1.0 delivers high fidelity, multilingual capability, long-duration generation, and editable stem outputs without complicated, gimmicky components.