Source-linked AI summary
Qwen-Music Technical Report
Jin Xu, Kangdi Wang, Ruibin Yuan, Shun Lei, Xiong Wang, Xize Cheng, Xueyao Zhang, Yang Zhang, Yiheng Chen, Yongqi Wang, Yue Wang, Zhifang Guo, Zihan Liu, Zijian Lin, Dake Guo, Hangrui Hu, Lei Xie, Linhan Ma, Wei Xue, Wenxiang Guo, Xinfa Zhu, Xipin Wei, Yangze Li, Yuanjun Lv, Yuxuan Wang, Yunfei Chu, Zhiyong Wu
TL;DR
Song generation must jointly model lyrics, melody, vocals, instrumentation, and long-range structure. Qwen-Music combines semantic composition, explicit melody planning, and acoustic rendering for text-to-music and cover generation, achieving the best results on 13 of 16 objective text-to-music metrics and strong human preference results.
Problem
Song generation requires jointly modeling lyrics, melody, rhythm, vocal performance, instrumentation, and musical structure over minutes.
Method
Qwen-Music unifies semantic composition, Melody-CoT melody planning, and generative stereo rendering for text-to-music and reference-audio-based cover generation.
Results
13 of 16 text-to-music metrics show Qwen-Music achieving the best result, alongside strong preference-test performance against leading proprietary systems.
Takeaways & Limitations
Qwen-Music demonstrates a unified, scalable, and controllable approach spanning long-range musical semantics and waveform-level acoustic realization.
Takeaways & Limitations
Future work is needed for more flexible long-context structure modeling, finer expressive singing control, and more efficient high-quality rendering.
Abstract
from arXiv · showhide
In this report, we introduce Qwen-Music, a powerful music generation model capable of producing highly musical and high-fidelity songs with complete vocal singing. Qwen-Music supports two core tasks: Text to Music Generation, which create entirely new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs with different styles and vocal characteristics. Architecturally, Qwen-Music integrates three core components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. Qwen-Music-Tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information for LLM prediction. Based on these tokens, Qwen-Music-LLM performs autoregressive music semantic modeling, with a key novelty being a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving creativity, musicality, structural coherence, and reference-audio-based melody cloning. To overcome the fidelity limitations of discrete semantic tokens, Qwen-Music-Render performs generative stereo rendering, enriching acoustic details and producing high-fidelity stereo waveforms. Finally, we train Qwen-Music-LLM on more than 5 million hours of multilingual music data covering hundreds of languages. We first apply quality-aware pre-training curriculum, then use progressive post-training, comprising supervised initialization, offline DPO, and online GSPO, to further improve musicality and instruction-following ability. Across 600 Chinese and English prompts, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover song generation, Qwen-Music preserves reference melodies more accurately than leading proprietary systems.
1 Introduction
Qwen-Music unifies semantic song composition and acoustic rendering for controllable text-to-music and reference-audio-based cover generation. Its explicit melody planning, generative rendering, and staged training support coherent, high-fidelity songs and strong preference results against proprietary systems.
- Architecture: The system separates long-horizon semantic composition from waveform-level acoustic rendering through compact Music Semantic Tokens and a dedicated rendering stage.Music is represented as a 25 Hz single-codebook stream, while rendering reconstructs acoustic detail through semantic-conditioned acoustic latents, spectrogram decoding, and refinement.
- Unified framework: Qwen-Music supports text-to-music and reference-audio-based cover generation, controlling genre, mood, instrumentation, vocal timbre, and vocal gender.The unified framework accepts text descriptions, lyrics, musical attributes, and optional reference audio.
- Melody planning and control: Melody-CoT plans vocal melodies before full-song generation, while reference melody tokens condition cover generation to preserve source melodies under new styles and vocal attributes.Melody serves as an explicit intermediate representation connecting textual intent or reference audio to musical realization.
- Training: Quality-graded pre-training and multi-stage preference alignment combine supervised learning, offline preference optimization, and online policy optimization to improve musicality, instruction following, and controllability.The curriculum progressively uses heterogeneous music data at different quality levels.
- Evaluation: 59.1% against MiniMax Music 2.5+, 66.7% against MiniMax Music 2.6, 58.3% against Mureka V8, and 55.4% against Suno V5 were Qwen-Music’s professional-rater blind A/B win rates.Against Suno V5.5, Qwen-Music remained comparable with a slight preference advantage of 50.3%.
2 Architecture
Qwen-Music uses a staged inference pipeline that rewrites user requests into structured musical conditions, generates Music Semantic Tokens, and renders the result. Reference-song melody tokens can optionally guide the LLM for melody cloning.
- Inference pipeline: The pipeline first rewrites a natural-language request into structured textual conditions containing musical tags and generated lyrics.Tags can describe genre, singer characteristics, and instrumental arrangement.
- Inference pipeline: Qwen-Music-LLM generates Music Semantic Tokens from the rewritten textual condition.
- Inference pipeline: For melody cloning, the LLM can optionally use melody tokens extracted from a reference song.
- Inference pipeline: Qwen-Music-Render takes the rewritten textual condition and generated tokens to produce the final output.
2.1 Overview
Qwen-Music transforms user requests into complete high-fidelity stereo songs through structured conditioning, semantic-token generation, and rendering. Its pipeline combines a 25 Hz tokenizer, an autoregressive LLM with melody-aware generation, and a waveform renderer.
- Inference pipeline: The inference pipeline rewrites natural-language requests into structured musical conditions and lyrics before autoregressively generating Music Semantic Tokens.Conditions may include genre, singer characteristics, instrumental arrangement, and other musical tags.
- Qwen-Music-Tokenizer: Qwen-Music-Tokenizer converts raw audio into a 25 Hz single-stream representation with a Conformer-based encoder and multi-stage training.Its recipe combines self-supervised learning, causal adaptation, multi-task supervision, and vector quantization.
- Qwen-Music-LLM: Qwen-Music-LLM generates semantic tokens from text, lyrics, musical attributes, and optional melody tokens, using Melody-CoT planning for original songs and reference melodies for covers.Cover generation follows target style and vocal attributes while conditioning on melody tokens extracted from a reference song.
- Qwen-Music-Render: Qwen-Music-Render converts generated Music Semantic Tokens into high-resolution audio waveforms.The overview identifies rendering as the final stage after semantic-token generation.
2.2 Qwen-Music-Tokenizer
Qwen-Music-Tokenizer converts raw music waveforms into a single 25 Hz stream of Music Semantic Tokens using a shared Conformer backbone trained through four stages. Multi-task supervision and vector quantization encourage the codes to preserve lyrics, melody, harmony, and spectral information for downstream autoregressive modeling.
- Architecture and training recipe: The tokenizer maps raw music waveforms into a single stream of 25 Hz Music Semantic Tokens through four training stages.The stages are BestRQ pretraining, causal adaptation, multi-task supervised fine-tuning, and VQ tokenizer training.
- Architecture and training recipe: A 24-layer, 0.6B-parameter Conformer backbone processes mono 24 kHz audio after convolutional frame-rate reduction to 25 Hz.The shared backbone uses log-Mel spectrogram features and represents one frame every 40 ms.
- Multi-task supervision: Multi-task supervision trains lyrics transcription, Mel reconstruction, and chroma prediction so discrete codes retain lyrical content, melody, harmony, and other musical semantics.The three heads are a multilingual-subword CTC head, a 100 Hz 128-bin Mel head, and a 12-bin chroma head.
- Vector quantization: A single VQ bottleneck with 32768 entries quantizes an intermediate Conformer representation, while upper layers and supervision heads operate on the quantized stream.The quantizer uses a cosine metric, straight-through estimation, and a commitment loss.
- Vector quantization: 99%+ codebook utilization keeps essentially the full quantizer active, with each 40 ms frame mapped to one integer token at inference.Inference discards post-VQ layers and heads, retaining only the frontend, pre-insertion Conformer layers, and quantizer.
2.3 Qwen-Music-LLM
Qwen-Music-LLM autoregressively models Music Semantic Tokens from textual conditions, with Melody-CoT explicitly planning melodies before full-song generation. Its unified melody-token interface supports both text-to-music generation and reference-melody-conditioned cover generation.
- Architecture: Qwen-Music-LLM uses a 3B dense Qwen3.5-Omni initialization to autoregressively predict Music Semantic Tokens capturing composition, vocal phrasing, accompaniment, and section-level development.The model operates in the discrete semantic-token space produced by Qwen-Music-Tokenizer.
- Melody-CoT: Melody-CoT explicitly plans melody before full-mixture token generation, separating compositional planning from arrangement and improving control under high-level text prompts.The mechanism addresses the challenge that text usually specifies style and lyrics without determining melodic contour or compositional structure.
- Melody-CoT: Melody-CoT provides one interface for text-to-music and cover generation: text prompts can optionally produce an intermediate melody plan, while covers prefix extracted reference melody tokens with target tags and lyrics.The model then generates new full-mixture Music Semantic Tokens conditioned on these inputs.
- Melody Tokenizer: The Melody Tokenizer extracts vocal pitch at 50 Hz, downsamples it by 8× with median pooling to 6.25 Hz, and maps relative pitch offsets into a 256-entry token vocabulary.This suppresses local ornaments and vibrato while retaining the compositional melody; relative MIDI offsets reduce leakage of absolute pitch range, key, and singer-dependent timbre.
- Training: Training applies next-token loss to both Melody-CoT and final Music Semantic Token regions while mixing direct, section-level, and unique-section-level sequence patterns.Section-level patterns omit non-vocal sections, and unique-section-level patterns sample one representative segment per repeated section type to reduce over-conditioning.
2.4 Qwen-Music-Render
Qwen-Music-Render converts semantic-token outputs and textual conditions into high-fidelity 48 kHz stereo waveforms through a three-stage neural rendering pipeline. Its Spec-VAE and frequency-aware refinement mechanisms target detailed spectral reconstruction under 192× compression.
- Rendering pipeline: Qwen-Music-Render uses a semantic-conditioned DiT to predict acoustic latents, which Spec-VAE decodes into spectrograms before Band-Mode Refiner correction and inverse-STFT synthesis.The renderer conditions on Music Semantic Tokens, rewritten text, and lyrics, producing 48 kHz stereo waveforms.
- Architecture: The 1.3 B DiT backbone contains 32 Transformer blocks, hidden dimension d = 1024, and 24 attention heads.Each block combines self-attention, cross-attention to auxiliary conditioning, and feed-forward layers with RoPE and AdaLN modulation.
- Spec-VAE: Spec-VAE compresses 48 kHz stereo complex spectrograms into 128-dimensional latents at 25 Hz with a total compression ratio of 192×.Stereo channels use delayed fusion in the encoder and early splitting in the decoder; a Band-Mode Refiner corrects residual spectral artifacts.
- Spec-SnakeBeta: Spec-SnakeBeta parameterizes activation parameters along the frequency axis, encouraging stronger periodic modulation at higher frequencies while adapting most strongly in the low-to-mid frequency range.This frequency-aware prior exploits the fixed physical frequency represented by each STFT bin and is used by the Spec-VAE decoder.
- Band-Mode Refiner: The Band-Mode Refiner applies phase-only correction at low frequencies, joint magnitude-and-phase correction in the mid-frequency band, and magnitude-only correction at high frequencies.It is implemented as a lightweight ConvNeXt-1D module designed to improve spectral fidelity.
- Training: Training proceeds through reconstruction-only Spec-VAE pretraining, waveform-domain adversarial training, and refiner training with waveform and spectral discriminators.The staged recipe stabilizes training under the 192× compression setting and progressively improves fine-grained acoustic detail.
3 Training
Qwen-Music-LLM is trained with a quality-aware pre-training curriculum and a progressive post-training pipeline. The process moves from broad musical coverage toward higher-quality generation, then aligns musicality, controllability, and instruction following through supervised and preference-based optimization.
- Training overview: Qwen-Music-LLM training combines quality-aware pre-training with progressive post-training.The pipeline targets the backbone LLM and separates pre-training from subsequent alignment.
- Quality-aware pre-training: A quality-graded curriculum progressively schedules heterogeneous music data, preserving broad musical coverage before concentrating on higher-quality generation.Instead of discarding lower-quality data, the corpus is organized into quality levels and trained progressively.
- Quality-aware pre-training: An internal MOS-based reward model, trained on professionally human-rated data, ranks samples within each genre into seven quality buckets, Q1–Q7.The predicted scores estimate and rank general musical quality for curriculum construction.
- Quality-aware pre-training: The three-stage curriculum progresses from Q3–Q6 general pre-training, through Q2 annealing, to Q1 high-quality refinement.These stages transition from broad coverage toward more structured, stable, musical, controllable, and instruction-following generation while maintaining balanced genre and language distributions.
- Progressive post-training: Post-training proceeds through supervised fine-tuning, offline DPO, and online GSPO to improve musicality, instruction following, controllability, and audio quality.Rewards include the internal musicality MOS predictor and instruction-following models, while GSPO enables on-policy exploration and sequence-level optimization.
4 Evaluation
Qwen-Music is evaluated on text-to-music and cover song generation using objective metrics, professional blind preference tests, genre-wise analysis, and external rankings. It achieves strong musicality, audio quality, controllability, lyric intelligibility, and reference-melody preservation, while its rendering refiner improves reconstruction quality.
- Subjective Evaluation: Qwen-Music wins blind A/B tests against MiniMax Music 2.5+ at 59.1%, MiniMax Music 2.6 at 66.7%, Mureka V8 at 58.3%, and Suno V5 at 55.4%.Against Suno V5.5, it holds a slight preference advantage of 50.3% versus 49.7%.
- Objective Evaluation: Qwen-Music achieves the best result in 13 of 16 evaluated SongBench, SongEval, and AudioBox-Aesthetic dimensions.It ranks first in six of seven SongBench dimensions and all five SongEval dimensions.
- Controllability and Intelligibility: Qwen-Music scores 8.44 on average across five tag-following dimensions and achieves a PER of 6.10, the second-lowest among all systems.It performs best on vocal gender following and second on mood and vocal timbre control.
- Cover Song Generation: On cover-song evaluation, section-level Melody-CoT achieves the lowest Melody MAE on AI-generated references, while Qwen-Music outperforms MiniMax Cover across most objective dimensions on real-world songs.Unique-section-level conditioning substantially improves tag following over section-level conditioning on the AI-generated reference set.
- Rendering Reconstruction: The full Spec-VAE + Refiner decoder achieves STFT Distance 0.870, STFTlog1p 0.075, Mel Distance 0.461, MELlog1p 0.089, Spectral Pan Error 0.264, and CCPC 0.973.Compared with Spec-VAE alone, the refiner improves all reported metrics, including Mel Distance from 0.572 to 0.461.
5 Conclusion
Qwen-Music unifies text-to-music and reference-melody-based cover-song generation by connecting semantic composition with high-fidelity audio synthesis. Its scalable training and evaluation results support controllable, musically strong generation, while future work targets longer-context structure, expressive control, and rendering efficiency.
- Qwen-Music unifies text-to-music generation and reference-melody-based cover-song generation in one large-scale system.
- 25 Hz single-codebook Music Semantic Tokens, Melody-CoT modeling, and generative stereo rendering connect semantic composition with high-fidelity audio synthesis.
- More than 5 million hours of multilingual music data support quality-graded pre-training, followed by supervised learning, offline DPO, and online GSPO post-training.
- Evaluations show competitive or superior text-to-music performance in musicality, audio quality, controllability, and lyric intelligibility, alongside reference-melody preservation and stylistic adaptation for covers.
- Future work will address flexible long-context musical structure modeling, finer-grained expressive singing and performance control, and more efficient high-quality rendering architectures.
6 Authors
This section lists the paper’s authors and notes that they are presented in alphabetical order, with an asterisk marking the corresponding author.
- The listed authors include Shun Lei, Xiong Wang, Xize Cheng, Xueyao Zhang, Yang Zhang, and Yiheng Chen.
- Additional authors are Dake Guo, Hangrui Hu, Lei Xie, Linhan Ma, Wei Xue, and Wenxiang Guo.
- The author list also includes Xinfa Zhu, Xipin Wei, Yangze Li, Yongqi Wang, Yue Wang, and Zhifang Guo.
- The superscript 2 indicates alphabetical order, while an asterisk denotes the corresponding author.
- It further lists Yuanjun Lv, Yuxuan Wang, and Yunfei Chu.