Source-linked AI summary
DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion
Ziqian Ning, Huakang Chen, Yuepeng Jiang, Chunbo Hao, Guobin Ma, Shuai Wang, Jixun Yao, Lei Xie
TL;DR
Existing song-generation systems often separate vocals and accompaniment, use complex pipelines, lack open implementations, or struggle with scalable full-length synthesis. DiffRhythm uses an end-to-end latent diffusion design with simple conditioning and sentence-level alignment, generating complete vocal-and-accompaniment songs up to 4m45s in 10 seconds. The authors report high musical quality and lyrical intelligibility, release code and pretrained models, and identify editing and text-based style control as unexplored capabilities.
Problem
Existing approaches often generate tracks separately or use complex multi-stage architectures, while few academic systems provide open, documented end-to-end full-song generation.
Method
DiffRhythm combines latent diffusion with a VAE, DiT conditioning on lyrics and style, and sentence-level lyrics alignment for end-to-end song generation.
Results
DiffRhythm generates complete stereo songs up to 4m45s in 10 seconds with vocals and accompaniment, while preserving high musical quality and lyrical intelligibility.
Takeaways & Limitations
The simple non-autoregressive design, released code, and pretrained models support scalable and reproducible end-to-end song-generation research.
Takeaways & Limitations
Specific-segment editing and text-based style control remain unexplored; the current system uses short audio clips as style references.
Abstract
from arXiv · showhide
Recent advancements in music generation have garnered significant attention, yet existing approaches face critical limitations. Some current generative models can only synthesize either the vocal track or the accompaniment track. While some models can generate combined vocal and accompaniment, they typically rely on meticulously designed multi-stage cascading architectures and intricate data pipelines, hindering scalability. Additionally, most systems are restricted to generating short musical segments rather than full-length songs. Furthermore, widely used language model-based methods suffer from slow inference speeds. To address these challenges, we propose DiffRhythm, the first latent diffusion-based song generation model capable of synthesizing complete songs with both vocal and accompaniment for durations of up to 4m45s in only ten seconds, maintaining high musicality and intelligibility. Despite its remarkable capabilities, DiffRhythm is designed to be simple and elegant: it eliminates the need for complex data preparation, employs a straightforward model structure, and requires only lyrics and a style prompt during inference. Additionally, its non-autoregressive structure ensures fast inference speeds. This simplicity guarantees the scalability of DiffRhythm. Moreover, we release the complete training code along with the pre-trained model on large-scale data to promote reproducibility and further research.
1 Introduction
DiffRhythm addresses limitations in holistic song generation with an end-to-end diffusion model for full songs containing vocals and accompaniment. Its simple, scalable design adds sentence-level lyrics alignment, a high-fidelity VAE, and publicly released resources.
- Existing systems often generate vocal and accompaniment tracks independently, producing disjointed musical experiences.
- Commercial end-to-end platforms provide limited open-source implementation and technical documentation.
- Its simple architecture and data pipeline are designed for scalability, while its non-autoregressive approach enables fast generation.
- DiffRhythm generates full songs with both vocal and accompaniment using an end-to-end diffusion-based model.The model is described as capable of generating full songs and durations up to four minutes.
- Sentence-level lyrics alignment improves vocal intelligibility while requiring only sentence-start annotations and minimal supervision.
- The VAE targets high-fidelity music reconstruction and robustness to MP3 compression artifacts, and the training resources are publicly available.
2 Related Work
Related work spans singing-voice, symbolic music, accompaniment, and song generation, with common approaches separating vocal and accompaniment production. DiffRhythm’s architecture is presented as an alternative full-song generation pipeline.
- Singing-voice models generate vocals from lyrics, scores, durations, reference voices, or textual descriptions.
- Symbolic music and songwriting systems generate melodies, lyrics, or song representations, while rap systems focus on lyrics generation.
- Accompaniment models generate instrumental music conditioned on vocals or provide controllable accompaniment generation.
- A common song-generation methodology first generates vocals from lyrics and then predicts accompanying music.
3 DiffRhythm
DiffRhythm is a full-diffusion model for end-to-end generation of full-length songs with vocals and accompaniment. It combines latent audio compression, diffusion-transformer generation, and sentence-level lyric alignment to support scalable long-form synthesis.
- Scalability: The model uses a non-autoregressive diffusion design and efficiency techniques intended to support scalable generation over long sequences.The implementation uses LLaMA decoder layers, FlashAttention2, and gradient checkpointing for long-context computation.
- Overview: DiffRhythm is designed for end-to-end generation of full-length songs containing both vocals and accompaniment.The model is described as the first full-diffusion-based approach specifically designed for end-to-end song generation.
- Variational Autoencoder: The VAE compresses arbitrary-length stereo waveforms into compact latent representations, reducing the computational burden of long-form diffusion modeling.The encoder and decoder operate fully convolutionally, with z = E(y) and reconstructed audio ˆy = D(E(y)).
- Diffusion Transformer: The diffusion transformer generates lyrics-conditioned latent songs using style prompts, diffusion timesteps, and phoneme embeddings as conditioning features.These features are combined with noised latent representations and supplied to the DiT.
- Lyrics-to-Latent Alignment: Sentence-level lyric alignment maps phoneme sequences into latent-frame positions corresponding to lyric timestamps, improving intelligibility with limited supervision.Lyrics are converted through G2P, placed into a latent-aligned sequence, and indexed using the latent frame rate.
4 Experimental Setup
The experiments use large-scale multilingual music data, long-form training configurations, waveform-reconstruction tests, objective song-generation metrics, and listening evaluations. The setup also evaluates reconstruction under both lossless and MP3-compressed inputs.
- Training Data: DiffRhythm was trained on approximately 1 million songs totaling 60,000 hours, with Chinese, English, and instrumental music in a 3:6:1 ratio.Tracks averaged 3.8 minutes, and a rule-based pipeline filtered low-quality lyrics.
- Model Configuration: The VAE uses adapted Stable Audio 2 weights, 157M parameters, 2.5M decoder-training iterations, and 64-dimensional latents at a 21.5 Hz frame rate.The audio is processed at 44.1 kHz with a compression factor of f = 2048.
- Model Configuration: Diffusion training uses 32 Euler ODE steps, CFG scale 4, and two sequence lengths spanning approximately 95 seconds to 4m45s.The base model uses Lmax = 2048 before fine-tuning with Lmax = 6144.
- Evaluation Metrics: Objective evaluation uses STOI, PESQ, MCD, PER, FAD, and realtime factor, while subjective tests collect 1-to-5 ratings from 30 listeners for musicality, quality, and intelligibility.FireRedASR is used for singing-vocal recognition, with PER selected instead of WER or CER.
5 Evaluation Results
DiffRhythm achieves strong reconstruction and song-generation quality while retaining substantial inference-speed advantages. Evaluation also identifies sentence-level alignment as essential for intelligible lyrics-to-vocal correspondence, while mixed vocal-accompaniment evaluation complicates PER interpretation.
- Waveform Reconstruction: 3.8% and 12.3% relative improvements in STOI and PESQ over the best baseline occur under lossless input, with comparable MCD.With lossy MP3 inputs, the proposed VAE maintains robust performance with only minimal degradation across all three metrics.
- Waveform Reconstruction: The VAE restores missing high-frequency components, mid-frequency spectral continuity, and vocal formant structures degraded by MP3 compression.Spectral visualization contrasts these restorations with vague harmonics from an open-source baseline.
- Song Generation: DiffRhythm achieves superior quality and intelligibility to SongLM, with comparable musicality and an 18.2% relative reduction in PER.SongLM retains slightly better FAD and musicality scores, indicating remaining room for long-term acoustic consistency and melodic expression.
- Song Generation: Both DiffRhythm variants maintain RTF below 0.04, achieving an approximately 50× speedup over SongLM.The full-length variant has marginally degraded PER and FAD relative to the base version, likely because longer sequences increase modeling complexity.
- Ablation: Removing sentence-level alignment catastrophically degrades intelligibility and audio quality while preserving basic musical structure.The ablation supports sentence-level alignment as essential for semantic correspondence between lyrics and vocals.
- Evaluation Caveat: PER may remain relatively high because ASR evaluates mixed vocal-and-accompaniment audio without source separation.Accompaniment may interfere with recognition, limiting direct interpretation of PER across systems.
6 Conclusion
The paper presents DiffRhythm as a full-diffusion model for complete stereo songs with vocals and accompaniment. Its simple, non-autoregressive design supports scalable generation, rapid inference, and open end-to-end research.
- Conclusion: DiffRhythm generates complete stereo songs with vocals and accompaniment up to 4m45s in 10 seconds.The system preserves high musical quality and lyrical intelligibility while using non-autoregressive inference.
- Conclusion: The design removes complex multi-stage cascading modeling and laborious data preprocessing, facilitating scalability.Code and pretrained models are released to support open end-to-end research in song generation.
7 Limitations
Two capabilities remain unexplored: editing or extending selected song segments and finer-grained text-based stylistic control.
- Unexplored Capabilities: Song editing through inpainting and continuation through outpainting have not been investigated.The authors suggest random masking of latent representations during training as a possible route to these capabilities.
- Unexplored Capabilities: The framework uses short audio clips as style references rather than natural-language conditioning for finer-grained stylistic control.Text conditioning could remove the need for audio references and improve flexibility, according to the authors.