Source-linked AI summary
YuE: Scaling Open Foundation Models for Long-Form Music Generation
Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, Xinrun Du, Zhen Ye, Tianyu Zheng, Zhengxuan Jiang, Yinghao Ma, Minghao Liu, Zeyue Tian, Ziya Zhou, Liumeng Xue, Xingwei Qu, Yizhi Li, Shangda Wu, Tianhao Shen, Ziyang Ma, Jun Zhan, Chunhui Wang, Yatian Wang, Xiaowei Chi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Shansong Liu, Lingrui Mei, Peng Li, Junjie Wang, Jianwei Yu, Guojian Pang, Xu Li, Zihao Wang, Xiaohuan Zhou, Lijun Yu, Emmanouil Benetos, Yong Chen, Chenghua Lin, Xie Chen, Gus Xia, Zhaoxiang Zhang, Chao Zhang, Wenhu Chen, Xinyu Zhou, Xipeng Qiu, Roger Dannenberg, Jiaheng Liu, Jian Yang, Wenhao Huang, Wei Xue, Xu Tan, Yike Guo
TL;DR
Long-form lyrics-to-song generation requires models to maintain alignment and coherent musical structure over minutes-long contexts, a challenge not solved by simply extending context length. YuE addresses this with an open foundation-model approach combining specialized prediction, conditioning, training, and in-context learning techniques; it matches or exceeds several commercial systems and performs competitively on music-understanding benchmarks.
Problem
Full-song modeling must handle minutes-long contexts while maintaining effective lyrics following, but simply scaling language-model context length is non-trivial and computationally demanding.
Method
YuE combines large-scale data, track-decoupled next-token prediction, segment-wise conditioning, multitask multiphase training, and redesigned in-context learning for controllable song generation.
Results
YuE matches or exceeds several commercial systems in musicality, controllability, and cross-lingual lyrics following, while achieving competitive results on standard music-understanding benchmarks.
Takeaways & Limitations
YuE demonstrates the promise of open, large-scale music models for controllable high-quality song generation and broader music-aware AI research.
Takeaways & Limitations
Lyrics-following can fail when generated instrumental preludes delay singing onset, causing drift from the prepended lyrics condition.
Abstract
from arXiv · showhide
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1) track-decoupled next-token prediction to overcome dense mixture signals, (2) structural progressive conditioning for long-context lyrical alignment, and (3) a multitask, multiphase pre-training recipe to converge and generalize. In addition, we redesign the in-context learning technique for music generation, enabling versatile style transfer (e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE's learned representations can perform well on music understanding tasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model, music generation
1. Introduction
YuE is introduced as an open foundation-model family for long-form lyrics-to-song generation, targeting coherence, lyrical alignment, and expressive vocals. Its design combines track-decoupled modeling, structural conditioning, redesigned in-context learning, and large-scale multitask training.
- Motivation: Lyrics-to-song generation requires full-song vocal and accompaniment synthesis despite long-range dependencies, polyphonic signal complexity, and singing-specific linguistic distortion.Existing audio-generation methods commonly produce short instrumental outputs, while proprietary full-song systems remain undisclosed.
- Training and results: Large-scale multitask and multiphase pre-training helps YuE generalize from scarce conditional lyrics-to-song data while strengthening multilingual and control-signal handling.The final training phase removes speech and unconditional music data, adds stronger controls, and uses approximately 20K hours of selected high-quality data.
- Framework: YuE uses an autoregressive framework with audio and text tokenizers plus separate Stage-1 music-language-modeling and Stage-2 residual-modeling language models.Stage 1 models semantic-rich audio tokens, while Stage 2 refines them with additional codebooks.
- Track-decoupled prediction: Dual-NTP splits each frame into vocal and accompaniment tokens, addressing the difficulty of representing both signals with one token and improving lyric intelligibility in dense mixtures.The approach jointly contextualizes both tracks while allowing their explicit segregation during modeling and processing.
- Structural conditioning: Structural Progressive Conditioning interleaves annotated song segments containing structure labels, lyrics, and audio to support long-context lyrics following and song-level structure control.Songs are automatically segmented into sections such as intro, verse, chorus, bridge, and outro, typically shorter than 30 seconds.
- Music in-context learning: YuE redesigns music in-context learning with single-track and dual-track reference conditioning, supporting style transfer, voice cloning, and bidirectional content creation.Reference audio may contain accompaniment, vocals, or a full mixture, and dual-track references interleave separated vocal and accompaniment tokens.
4. Training and Inference Strategies
YuE uses multitask learning and multiphase training to address scarce conditional lyrics-to-song data, balance lyric control against singing quality, and extend long-context generation. Its inference strategies combine forced decoding, classifier-free guidance, and dual-track in-context learning.
- 4.1.1. Multitask Learning: Multitask pre-training transfers knowledge from auxiliary tasks because paired lyrics-audio data are scarce and unconditional music dominates the dataset.The setup decomposes lyrics-to-song generation into capabilities and includes speech, music-generation, and lyrics-to-song tasks.
- 4.1.1. Multitask Learning: TTS data establish linguistic-to-vocal alignment, but excessive exposure biases generation toward speech and rap while degrading singing genres.Insufficient TTS training instead leads to poor lyric adherence, so the proportion of TTS data must be balanced.
- 4.1.1. Multitask Learning: Music-generation training uses tagged unconditional recordings and dual-track vocal-instrumental data to improve musicality and support style control.Tracks are arranged as tag-conditioned NTP or Dual-NTP objectives, with text instructions distinguishing the prediction modes.
- 4.1.2. Multiphase Training: The four training phases warm up on high-quality English and Chinese data, add multilingual in-the-wild data, extend context to 16384, and inject stronger controls during annealing.Phase 3 adds 750B tokens for long-context dependencies, while Phase 4 uses approximately 20K hours of selected high-quality data and 40B tokens.
- 4.3. Test-time Strategies: Inference constrains token vocabularies by stage and codebook, while classifier-free guidance combines conditional and unconditional log-probabilities.For in-context learning, chorus references improve musicality and stability, and dual-track conditioning is enabled by default for better audio quality.
5. Experiments
The experiments evaluate YuE with large-scale data, closed-source baselines, human A/B tests, and automatic audio metrics. Human evaluation finds YuE competitive with several proprietary systems but still behind Suno V4.
- 5.1. Data Setup: The training data include 70k hours of conditional speech and 650K hours of in-the-wild music recordings.After filtering, 10% of music data has corresponding lyrics; Stage-1 also contains 13B speech tokens, over 200B unconditional music tokens, and 28B CoT music tokens.
- 5.2. Evaluation Protocol: YuE is benchmarked against Suno V4, Udio, Hailuo, and Tiangong because no other academic or open-source system provides usable long-form song generation.The comparison reflects a black-box evaluation conducted in January 2025.
- 5.3. Results: YuE outperforms Hailuo, matches Tiangong and Udio, and trails Suno V4 in average human preference and musicality.Its musicality win–loss ratios are balanced against Tiangong and Udio, decisive against Hailuo, and unfavorable against Suno V4.
- 5.2. Evaluation Protocol: The evaluation covers musicality, vocal and accompaniment quality, arrangement, melody, structure, lyric following, and controllability criteria.The listed criteria include Overall Musicality, VocalQual, AccompQual, MusicArr, MelodicAttrac, SongStruct, LyricFollow, GenCtrl, and InstrCtrl.
- 5.3. Results: Automatic evaluation reports KL divergence, Frechet Audio Distance, and Audiobox-Aesthetic dimensions for generated-audio distributions, quality, realism, and perceived aesthetics.The cited aesthetic dimensions include Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness.
6. Main Results
YuE is competitive with proprietary systems in human-rated musicality and control, while showing particular strengths in musical structure, vocal range, duration, and several automatic metrics. Evaluation also exposes weaknesses in acoustic fidelity and limitations in some automatic metrics.
- Overall Comparison with Proprietary Systems: YuE outperforms Hailuo, matches Tiangong and Udio, but trails Suno V4 in average human preference and musicality.Suno V4 remains the state-of-the-art system in the reported comparison.
- Detailed Comparison with Proprietary Systems: YuE excels in music structure and arrangement but shows deficiencies in vocal and accompaniment acoustic quality.The paper attributes these deficiencies to limitations of the current audio tokenization method.
- Detailed Comparison with Proprietary Systems: YuE performs strongest among other models in genre adherence, instrument/vocal consistency, and emotion control.Control evaluation also includes tempo/rhythm control and lyrics following, using human preference win rate because classifiers and transcription systems are limited.
- Vocal Agility and Duration: YuE demonstrates one of the widest vocal ranges, with a medium of approximately 27 semitones, close to top-performing closed-source systems.Higher vocal-range values indicate greater vocal expressiveness.
- Vocal Agility and Duration: YuE produces the longest audio and a substantially wider duration range than the other evaluated systems.This supports its ability to generate full-length songs beyond typical short AI-generated clips.
- Model Based Evaluation: YuE achieves the best KL divergence at 0.372, while Udio achieves the lowest FAD at 1.222 and YuE remains competitive at 1.624.The paper cautions that distribution-based metrics can suffer from sample-size bias and may miss subtle acoustic fidelity.
- Model Based Evaluation: YuE attains the highest CLaMP 3 alignment score at 0.240 but a lower CLAP score of 0.118, revealing disagreement between automatic alignment metrics.CLaMP 3 correlates better than CLAP with human controllability evaluations, including LyricFollow and InstrCtrl.
- Correlation Between Automatic Metrics and Human Evaluation: Vocal Range correlates above 0.85 with both musicality and average human preference, while content-based metrics correlate more strongly with acoustic quality.CE correlations include 0.66 with VocalQual and 0.56 with AccompQual.
7. Fine-tuning To More Languages
Fine-tuning enables YuE to support Chinese, Korean, and Japanese lyrics, with strongest reported lyrics-following performance in Japanese. Results remain uneven across languages and indicate a need for culturally specific training.
- Multilingual Adaptation: Fine-tuning within a 40B-token budget adapts YuE to Chinese, Korean, and Japanese.The fine-tuning procedure re-anneals from the last constant-learning-rate checkpoint using enhanced target-language data.
- Lyrics Following Across Languages: YuE achieves the highest Japanese lyrics-following performance at 70%.This is the best-performing result reported for Japanese lyrics following.
- Lyrics Following Across Languages: YuE ranks second in Chinese lyrics following at 60%, behind Suno at 73%, and third in Korean lyrics following at 55%.The results show stronger performance in Japanese than in the other evaluated languages.
- Musicality Across Languages: YuE places second in Chinese musicality at 62% and Korean musicality at 55%.The paper links the Chinese musicality gap relative to Suno to the need for more culturally specific training.
8. Analysis and Ablations
The ablations examine tokenization, source-separation priors, Dual-NTP, conditioning strategies, model scaling, and inference settings for lyrics following. Results show improved reconstruction robustness, lower training loss, stronger long-range alignment, and higher human preferences with the proposed choices.
- Source Separation Prior and Dual-NTP: Dual-NTP lowers training loss by approximately 0.4 versus standard NTP under identical 0.5B-model budgets and 20B consumed tokens.The method separately models vocal and accompaniment tracks, supporting robustness when accompaniment dominates.
- Source Separation Prior and Dual-NTP: At approximately -8.0 dB VAR, mixture reconstruction shows 20%+ ΔWER, whereas vocal-track reconstruction remains below 10% ΔWER in the worst case.Higher VAR indicates greater vocal prominence relative to accompaniment.
- Lyrics-following Capabilities with CoT: CoT achieves consistently superior WER across 30–150 seconds, while scaling from 0.5B to 7B reduces WER from approximately 70% to around 20%.WER is estimated with a fine-tuned Whisper model at 30-second intervals.
- Lyrics-following Capabilities with CoT: Vanilla, Curriculum, and ABF show substantially worse WER because instrumental preludes delay singing relative to the prepended lyrics condition.This failure complicates accurate lyric alignment over time.
- Scaling and Test-time Tricks: Human preference increases with model scale across musicality and lyrics-following, and ICL+CFG reaches a 0.79 musicality win rate versus 0.21 for CoT.ICL alone achieves a 0.63 win rate in the same A/B testing comparison.
9. Representation Quality
YuE is evaluated on MARBLE in an auxiliary unconditional single-track setting using only codebook-0 representations. Despite these restrictions, the evaluation tests several music-understanding tasks.
- MARBLE Evaluation: YuE’s MARBLE evaluation covers GTZAN genre classification, GS key recognition, MTG top-50 tagging, and EMO emotion regression.The model uses its Stage-1 LM in an unconditional single-track setting, and this auxiliary task is disabled halfway through training.
- Evaluation Setting: The MARBLE setup uses only discrete codes from codebook-0, providing substantially less information than dedicated representation-learning models.YuE was designed primarily as a generative model rather than explicitly for representation learning.
10. Emergent Abilities
YuE exhibits emergent vocal, stylistic, multilingual, and compositional behaviors beyond basic lyrics-to-song generation. These include expressive singing techniques, spontaneous musical continuation, cross-cultural pattern mixing, voice cloning, style transfer, and code switching.
- Advanced Vocal Techniques: YuE spontaneously generates advanced vocal techniques including vibrato, glissando, bel canto, death growl, belting, riffs and runs, and regional folk styles.The paper attributes these capabilities to capturing subtle vocal-performance nuances with Dual-NTP.
- Spontaneous Performance: YuE produces spontaneous musical behaviors such as scat singing, multipart a cappella harmonies, and contextually appropriate instrumental solos.These behaviors appear in jazz, a cappella, and folk examples.
- World Music and Pattern Mixing: YuE mixes global styles, generating combinations such as Chinese gangsta rap with Japanese shamisen instrumentation and scales.It also blends Chinese opera, Shanbei folk singing, and traditional Chinese vocals cohesively.
- Voice Cloning: YuE performs high-fidelity voice cloning, reproducing Billie Eilish and Faye Wong voices while generating new lyrics and melodies.The cloned voices retain signature timbral, breathy, and emotional characteristics.
- Style Transfer: YuE transforms Japanese female J-pop into English male rap while preserving the original city-pop accompaniment and lyrical-melodic structure.The framework supports cross-lingual and genre adaptation across vocal characteristics.
- Code Switching: YuE naturally code-switches among languages or dialects while preserving linguistic and stylistic consistency within a vocal performance.
11. Memorization Effect
The memorization study tests whether YuE reproduces training material under ICL, while the unsuccessful-attempts discussion identifies representation and training limitations. Results indicate lower generated-reference similarity than duplication-oriented baselines, but acoustic-token and unconditional-pretraining choices remain problematic.
- Memorization Evaluation: The memorization evaluation uses 1,200 training references and corresponding ICL generations, examining top-1% ByteCover2 similarity against GTZAN and Covers80 baselines.Frequent high-similarity pairs would suggest substantial memorization.
- Memorization Results: Ref-Gen similarity is significantly lower than Covers80 and moderate relative to GTZAN, although short repetitive motifs can still recur.The comparison is designed to contextualize possible memorization against genre-level similarity and known melodic duplicates.
- Unsuccessful Attempts: Acoustic-token language models show convergence difficulties and higher losses, which the authors associate with lossy, weakly semantic representations and reconstruction-focused shortcuts.The authors report only marginal gains from scaling such models substantially.
- Unsuccessful Attempts: At 7B parameters, unconditional pre-training becomes counterproductive because fine-tuning fails to establish effective text-to-vocal cross-modal alignment.The authors hypothesize that generic priors overshadow task-specific conditional mappings.
- Unsuccessful Attempts: Early ICL activation causes reference-audio overreliance and poor musicality, followed by invalid outputs after reference removal; the problem worsens with scaling.
- Conclusion: YuE combines large-scale data, Dual-NTP, segment-wise conditioning, and redesigned ICL to generate coherent full-length songs with expressive vocals and detailed structure.The conclusion positions the model as an open foundation-model family for long-form lyrics-to-song generation.
- Future Work: Future extensions include improving acoustic fidelity and mixing, adding musical knowledge and deeper prosodic or emotional controls, and expanding multilingual coverage.The paper also identifies applications in education, accessibility, therapy, and community-driven open research.
A. Subjective Evaluation
The subjective evaluation used pairwise comparisons of outputs from multiple music-generation models. Annotators judged overall musicality and related musical qualities after listening to both samples.
- Annotators compared two musical pieces generated by different models and selected which was superior or equivalent.Each test unit required complete playback followed by a binary comparative judgment.
- Figure 16 presents the platform used for the subjective evaluation experiment.
- Overall musicality assessed artistic value, professionalism, melody smoothness, harmonic quality, and rhythmic precision.
- Arrangement quality covered layer clarity, instrumental coordination, accompaniment balance, frequency distribution, and variation across sections.
- Melodic memorability evaluation considered ease of memorization, singability, catchiness, and emotional resonance.
A.3. Conditional Evaluation Dimension and Definitions
The conditional evaluation included lyrics following and tempo-and-rhythm dimensions. These dimensions assessed whether generated vocals and musical timing matched prompt requirements.
- Lyrics following measured accurate lyric delivery, pronunciation clarity, rhythmic alignment with the beat, and multilingual lyric naturalness.
- Tempo and rhythm assessed consistency with the requested BPM and adherence to specified rhythmic patterns.
B. Qwen2Audio-Instruct Tagging Prompt
Qwen2Audio-Instruct prompts were used to describe audio genre, instruments, mood, and vocal characteristics in structured JSON outputs.
- The music-tagging prompt requests JSON lists for Music_genre, Instrument, and Mood.The example represents multiple instruments as a list of strings.
- The vocal-tagging prompt requests JSON lists for gender, age, and vocal_timbre.The example allows multiple vocal-timbre descriptors.
C. Multilingual Subjective Evaluation
Figure 17 compares YuE with other systems across languages using lyrics following and musicality as evaluation dimensions.
- Figure 17 evaluates YuE versus other systems across different languages on lyrics following and musicality.
D. 15 English Prompts From GPT
The prompts provide English song lyrics organized into conventional sections such as verses, choruses, bridges, and outros. Their themes span ambition, romance, heartbreak, introspection, and perseverance.
- Themes: Several prompts focus on ambition, persistence, and personal achievement despite obstacles or criticism.These lyrics emphasize pursuing dreams, overcoming setbacks, and building an independent path.
- Prompt structures: The prompts repeatedly use verse, chorus, bridge, and outro structures to organize complete English songs.Several examples pair multiple verses with recurring choruses and transitional bridges or outros.
- Themes: Romantic commitment and intimacy form a major recurring theme across the prompts.The lyrics describe devotion, closeness, reconciliation, and promises to remain together.
- Themes: Other prompts portray heartbreak, loneliness, uncertainty, and searching for direction.These examples include lost love, solitary city scenes, emotional confusion, and unresolved longing.
- Themes: The collection also includes lyrics about escaping worries through shared dreams and rewriting a relationship’s history.These prompts frame partnership as a source of refuge, growth, and renewed possibility.