PaperlayerView exact revision on arXiv

Source-linked AI summary

YuE: Scaling Open Foundation Models for Long-Form Music Generation

Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, Xinrun Du, Zhen Ye, Tianyu Zheng, Zhengxuan Jiang, Yinghao Ma, Minghao Liu, Zeyue Tian, Ziya Zhou, Liumeng Xue, Xingwei Qu, Yizhi Li, Shangda Wu, Tianhao Shen, Ziyang Ma, Jun Zhan, Chunhui Wang, Yatian Wang, Xiaowei Chi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Shansong Liu, Lingrui Mei, Peng Li, Junjie Wang, Jianwei Yu, Guojian Pang, Xu Li, Zihao Wang, Xiaohuan Zhou, Lijun Yu, Emmanouil Benetos, Yong Chen, Chenghua Lin, Xie Chen, Gus Xia, Zhaoxiang Zhang, Chao Zhang, Wenhu Chen, Xinyu Zhou, Xipeng Qiu, Roger Dannenberg, Jiaheng Liu, Jian Yang, Wenhao Huang, Wei Xue, Xu Tan, Yike Guo

arXiv:2503.08638v2eess.AScs.AIcs.MMcs.SD

TL;DR

Long-form lyrics-to-song generation requires minutes-long contexts, beyond the under-30-second contexts typical of TTS and TTM systems. YuE combines track-decoupled prediction, segment-wise conditioning, redesigned in-context learning, and multitask, multiphase training; it matches or exceeds several commercial systems and achieves competitive music-understanding results.

  • Problem

    Full-song lyrics-to-song generation remains challenging because typical TTS and TTM systems handle under 30 seconds, while available music data are mostly unconditional.

    Source: §3.2.2 · p. 8, §4.1.1 · p. 12

  • Method

    YuE combines track-decoupled prediction, segment-wise conditioning, redesigned in-context learning, and multitask, multiphase training for long-form lyrics-to-song generation.

    Source: §1 · p. 5, §13 · p. 27

  • Results

    YuE matches or exceeds several commercial systems in musicality, controllability, and cross-lingual lyrics following, and achieves competitive music-understanding results on standard benchmarks.

    Source: §13 · p. 27

  • Takeaways & Limitations

    The findings support the promise of open, large-scale music models for controllable, high-quality song generation and broader music-aware AI research.

    Source: §13 · p. 27

    The model-based evaluation uses CLaMP3 because CLAP may have limited exposure to singing and musical content.

    Source: §6.2.3 · p. 19

Abstract

from arXiv · show

We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1) track-decoupled next-token prediction to overcome dense mixture signals, (2) structural progressive conditioning for long-context lyrical alignment, and (3) a multitask, multiphase pre-training recipe to converge and generalize. In addition, we redesign the in-context learning technique for music generation, enabling versatile style transfer (e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE's learned representations can perform well on music understanding tasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model, music generation

1. Introduction · 2. Related Work and Prelimenaries · 3. YuE

These sections introduce YuE, an open foundation-model family for long-form lyrics-to-song generation, and describe its modeling approach, including track-decoupled prediction, structural conditioning, music in-context learning, and a two-stage autoregressive architecture with text–audio tokenization and reconstruction.

1. Introduction

  • YuE targets the difficulty of open, coherent full-song lyrics-to-song generation, with models built on LLaMA2 and trained on trillions of tokens to generate songs up to five minutes long.The paper presents track-decoupled prediction, structural progressive conditioning, redesigned music in-context learning, and multitask multiphase pre-training as contributions; it also reports strong performance in musicality, vocal agility, multilingual lyrics following, and music understanding on MARBLE.Source: §1 · pp. 4–5

2. Related Work and Prelimenaries

  • Prior academic song systems struggle to sustain coherent, high-quality songs beyond short clips, while proprietary systems offer song-level generation without disclosed technical details.The related-work discussion also contrasts vocal-only singing synthesis with YuE’s joint generation of semantically meaningful vocals and accompaniment, and notes that semantic information fused into audio tokens improves convergence on complex in-the-wild data.Source: §2 · p. 5

3.1. Overview

  • YuE uses two LLaMA2-based language models: Stage 1 predicts lyrics-conditioned semantic-rich audio tokens, and Stage 2 models residual tokens before vocoder upsampling to 44.1 kHz.A semantic-acoustic fused tokenizer produces discrete audio tokens; a text tokenizer handles text inputs.Source: §3.1 · p. 6

3.2. Stage-1: Music Language Modeling

  • Stage-1 music language modeling conditions generation on diverse inputs, including lyrics, tags, structure, and reference audio, using three core techniques.These are track-decoupled next-token prediction, structural progressive generation, and music in-context learning.Source: §3.2 · p. 6

3.2.1. Track-Decoupled Next-Token Prediction

  • Standard next-token prediction struggles to model vocals and accompaniment together because the accompaniment can overshadow vocals and degrade lyric intelligibility.Dual-NTP emits separate vocal and accompaniment tokens at each time step; the authors report lower training loss, robust lyric adherence in challenging genres, joint track contextualization, and independent track processing.Source: §3.2.1 · pp. 6–8

3.2.2. Structural Progressive Conditioning

  • Simply extending context does not ensure long-song lyric following: RoPE conditioning reportedly degrades around 3K tokens and fails beyond 6K, even with 16K-token pre-trained contexts.Structural progressive conditioning pairs lyrics and audio within automatically segmented song sections, using section labels such as verse and chorus; a song averages 14 segments, most shorter than 30 seconds.Source: §3.2.2 · pp. 8–9

3.2.3. Music In-Context Learning

  • Music continuation-based in-context learning can require reference lyrics, restrict generation to one direction, and entangle reference content with the generated music.YuE supports single-track or interleaved dual-track references, prepending a sampled 20–40-second segment to structured training data; because early ICL data can induce copying and disrupt lyric control, the method delays a small ICL-data dose until annealing.Source: §3.2.3 · pp. 9–10

3.3. Stage-2: Residual Modeling

  • Stage 2 jointly predicts all eight codebooks in one autoregressive framework, including Stage 1’s codebook-0 tokens, to model both high-level structure and residual audio detail.The training target includes codebooks 0–7 and maintains a strictly time-aligned factorization.Source: §3.3 · pp. 10–11
  • Stage-2 places all codebook-0 tokens first, then predicts repeated eight-token blocks for codebooks 0–7 per frame, letting residuals attend to the complete semantic outline.The extended sequence is trained with standard teacher forcing.Source: §3.3 · p. 11
  • At inference, Stage-1 codebook-0 tokens are clamped, leaving codebooks 1–7 as the free autoregressive outputs and preserving sequence alignment.The model is a 1B-parameter Transformer with an 8K-token context window, trained on consecutive 6-second single-track segments.Source: §3.3 · p. 11

3.4. Tokenization and Audio Reconstruction

  • Vanilla text-to-speech and text-to-music methods perform poorly on song generation, where musicality and song-level lyric following are key challenges.Source: §3.4 · p. 11
  • YuE uses LLaMA’s 32,000-token BPE vocabulary for instructions, genres, lyrics, structural annotations, and segment boundaries, alongside X-Codec [Ye et al., 2024] audio tokens.It uses the first 8 of X-Codec’s 12 RVQ layers; codebook-0 captures semantic information including melody and vocal content.Source: §3.4 · pp. 11–12
  • Special tokens mark document and audio boundaries, identify generation stages, and indicate tokenizer types.The stage indicators <stage_1> and <stage_2> identify Stage 1 and Stage 2 tokens, respectively.Source: §3.4 · p. 12
  • A lightweight Vocos-adapted vocoder upsamples reconstructed audio from 16 kHz to 44.1 kHz by predicting higher-frequency components.Training uses random codebook dropout and a small amount of Gaussian noise to improve robustness.Source: §3.4 · p. 12

4. Training and Inference Strategies

YuE combines auxiliary-task training, a staged curriculum, and test-time controls to support lyric-aligned, long-form music generation. Its training balances speech and music objectives, extends context and language coverage, and injects stronger controls late in training.

4.1.1. Multitask Learning

  • Multitask pre-training transfers capabilities from speech, tagged music, and lyrics-to-song data to address scarce conditional data and strengthen lyric control.Text-to-speech data supports lyric following, but too much biases outputs toward rap and can degrade singing genres, while too little weakens lyric adherence. Music tracks receive open-vocabulary genre, instrument, and mood tags, and 40% are separated into vocal-instrumental tracks; lyrics-audio pairs are heuristically filtered to about 10% of matched tracks, with segment-level alignment and TTS training helping mitigate remaining mismatches.Source: §4.1.1 · pp. 12–13

4.1.2. Multiphase Training

  • Stage-1 training progresses from English and Chinese warm-up through multilingual data and longer context to a final phase that injects stronger controls.The first phase uses 280B tokens; the second reaches 1T tokens while maintaining a 2:1 old-to-new data ratio. Context then extends to 16,384 tokens, with single-track unconditional data removed. Final annealing uses about 20K hours of high-quality data and adds reference audio, gender, and vocal-timbre controls; BPM control is removed because its coupling with lyric length degrades lyric following. Only 40B tokens, about 2% of the compute budget, enabled the introduced controls.Source: §4.1.2 · pp. 13–14

4.2. Stage-2 Pre-training.

  • Stage-2 pre-training uses a 1B-parameter model over 2T tokens, incorporating speech, demixed music, and mixed music data.The model is trained with context length 8,192 and a linear warm-up followed by cosine annealing, reaching a maximum learning rate of 3 × 10^-4.Source: §4.2 · p. 14

4.3. Test-time Strategies

  • Chorus-based in-context learning improves musicality and stability, while dual-track references provide better audio quality than single-track references.Dual-track ICL is enabled by default unless otherwise specified.Source: §4.3 · p. 14

5. Experiments

The experiments evaluate YuE against closed-source song-generation systems using human judgments and automatic metrics, with training built on large speech and music datasets and scaled token budgets.

5.1. Data & Training Setup

  • Training used 70k hours of conditional speech data and 650K hours of mined music, scaling the 7B model to 1.75T tokens before 40B-token annealing.Stage 1 included 13B conditional speech tokens, over 200B unconditional music tokens, and 28B CoT music tokens; annealing used a 40B-token ICL dataset.Source: §5.1 · pp. 14–15

5.2. Evaluation Protocol

  • Human evaluation used 40 blinded evaluators to compare systems on musicality, vocal and accompaniment quality, lyrics following, song structure, and controllability.The main experiments evaluated 42 full-length English songs; multilingual prompts included 10 Chinese and 10 Japanese/Korean prompts, with proficient language speakers assessing non-English samples.Source: §5.2 · pp. 15–16
  • Human evaluation found YuE matched Tiangong and Udio and outperformed Hailuo; Figure 6 reports overall preference and musicality win-tie-loss.The figure compares YuE with four proprietary systems, but its caption names only these three outcomes.Source: Figure 6 · p. 16
  • Automatic evaluation measured audio-feature distribution differences, audio quality and realism, aesthetics, text-audio alignment, vocal agility, and generation duration.Metrics included KL divergence, Frechet Audio Distance, Audiobox-Aesthetic, CLAP and CLaMP 3 scores, and song-level vocal range and flexibility.Source: §5.2 · p. 16

6. Main Results

YuE performs competitively against proprietary systems in human evaluations, with strengths in musical structure, control, vocal agility, and song duration. Automatic evaluations and their correlations with human judgments show that different metrics capture different aspects of generated music.

6.1.1. Overall Comparison with Proprietary Systems.

  • YuE outperforms Hailuo, is comparable to Tiangong and Udio, and trails Suno V4 in average human preference and musicality.Suno V4 remains the highest-quality system in these comparisons, while YuE shows balanced musicality win–loss ratios against Tiangong and Udio.Source: §6.1.1 · p. 17

6.1.2. Detailed Comparison with Proprietary Systems.

  • YuE excels in music structure, arrangement, genre adherence, instrument/vocal consistency, and emotion, but trails Suno V4 overall.It shows deficiencies in vocal and accompaniment acoustic quality, and moderate emotion and tempo control; the authors point to tokenization and noisy lyric-alignment and tempo-tagging pseudo-labels as areas needing improvement.Source: §6.1.2 · p. 17, Figure 7 · p. 17

6.2.1. Vocal Agility

  • YuE’s median song-level vocal range is about 27 semitones, among the widest and close to Suno V4.Hailuo and Tiangong have median ranges around 20 semitones; higher values indicate greater vocal expressiveness.Source: §6.2.1 · p. 18, Figure 8 · p. 18

6.2.2. Duration

  • YuE produces the longest audio and a wider duration range than other systems, supporting full-length song generation.Suno V4 and Tiangong also generate relatively long audio, while Hailuo Music has the most restricted durations.Source: §6.2.2 · pp. 18–19, Figure 9 · p. 18

6.2.3. Model Based Evaluation

  • YuE leads KL (0.372) and CLaMP 3 alignment (0.240), remains competitive on FAD (1.624), and earns competitive aesthetics scores.Udio has the lowest FAD (1.222); YuE’s content-based scores are PQ (7.894), PC (6.280), CE (7.115), and CU (7.543), while its CLAP score is 0.118. The authors caution that automatic metrics do not always align with human perception.Source: §6.2.3 · p. 19, Table 3 · p. 19

6.2.4. Correlation Between Automatic Metrics and Human Evaluation

  • Vocal Range correlates above 0.85 with both Musicality and Average human preference, while KL and FAD correlate weakly with acoustic-quality preferences.The authors recommend Vocal Range as a practical proxy for musicality; they attribute the weak KL/FAD correlations with accompaniment and vocal quality partly to out-of-distribution evaluation and limited sample sizes.Source: §6.2.4 · p. 20, Table 4 · p. 20, Table 6 · p. 20
  • CLaMP 3 correlates better than CLAP with controllability judgments, including LyricFollow (0.42 vs. -0.25) and InstrCtrl (0.44 vs. -0.07).CLaMP 3 genre-following scores also relate to lyric-following despite not explicitly using lyrics; the authors note CLAP’s weaker correlations may reflect limited singing and music exposure during pretraining.Source: §6.2.4 · p. 20, Table 5 · p. 20
  • CE correlates with VocalQual (0.66) and AccompQual (0.56), but less with musicality-related measures; PC and PQ correlations are weaker or inconsistent.CE’s reported correlations include SongStruct (0.33), VAComp (0.35), MelAttrac (0.30), and MusicArr (0.31).Source: §6.2.4 · p. 21, Table 7 · p. 21

7. Fine-tuning To More Languages

Fine-tuning YuE across Chinese, Korean, and Japanese shows multilingual adaptability, with lyrics-following and musicality results varying by language and a remaining need for culturally specific training.

7. Fine-tuning To More Languages

  • Musicality rates were 62% in Chinese and 55% in Korean, both second-best; YuE’s gap to Suno in Chinese highlights a need for culturally specific training.The paper describes these results as evidence of cross-lingual transfer of musical features.Source: §7 · p. 21, Table 8 · p. 21
  • YuE was fine-tuned to Chinese, Korean, and Japanese within a 40B-token budget by re-annealing from the last constant-learning-rate checkpoint using enhanced target-language data.The results suggest potential for further improvement through targeted fine-tuning.Source: §7 · p. 21
  • Lyrics-following rates were 70% in Japanese (highest), 60% in Chinese (second to Suno’s 73%), and 55% in Korean (third).Table 8 reports human preference rates for lyrics-following and musicality across languages.Source: Table 8 · p. 21, §7 · p. 21

8. Analysis and Ablations

YuE’s analyses find that in-context conditioning and CFG improve musicality, while source-separation priors, CoT lyric conditioning, and scaling support lyric adherence and generation quality. The tokenizer comparison favors semantic-acoustic tokens for LM convergence, with X-Codec avoiding a reconstruction alignment flaw identified in Semanticodec.

8.1. Comparison of Audio Tokenizers

  • Semantic-acoustic tokenizers enabled 0.5B LM convergence, while acoustic-only tokens often failed; X-Codec also avoided Semanticodec’s alignment flaw.Acoustic tokenizers had decent reconstruction but difficult LM convergence; Semanticodec and X-Codec converged better through stable SSL-encoder clustering, though reconstruction quality was only fair. Semanticodec’s AudioMAE patch misalignment could propagate through reconstruction, unlike X-Codec’s Hubert-derived semantics.Source: §8.1 · p. 22, Table 9 · p. 22

8.2. Impact of Source Separation Prior and Dual-NTP

  • A 20%+ ΔWER appeared around -8.0 dB VAR for reconstructed mixtures, whereas vocal tracks’ worst ΔWER was 10%- around that VAR.The analysis reports that vocal tracks maintained low WER and smaller reconstruction-related increases, indicating resilience to VAR degradation and information loss.Source: §8.2 · pp. 22–23, Figure 10 · p. 23
  • Dual-NTP achieved approximately 0.4 lower training loss than standard NTP under identical 0.5B model, data, and 20B-token compute budgets.The authors interpret this ablation as evidence of Dual-NTP’s efficiency and robustness for song modeling.Source: §8.2 · pp. 22–23, Figure 11 · p. 23

8.3. Ablation Analysis of Lyrics-following Capabilities with CoT

  • WER fell from approximately 70% at 0.5B to around 20% at 7B, while CoT performed best across 30–150 seconds.WER was estimated with a fine-tuned Whisper model every 30 seconds; Vanilla, Curriculum, and ABF performed substantially worse, with Vanilla and Curriculum often generating instrumental preludes that delayed singing.Source: §8.3 · pp. 23–24, Figure 12 · p. 23

8.4. Effect of Scaling

  • Human evaluation showed improving musicality and lyric-following preferences across 0.5B, 2B, and 7B checkpoints, with the 7B model highest.The 0.5B and 2B models used 500B tokens and 16K context, while 7B used 1.75T tokens and the full training dataset.Source: §8.4 · p. 24, Figure 13 · p. 24

8.5. Analysis of Test-time Tricks

  • ICL+CFG outperformed ICL and CoT in musicality preference, which the authors link to stronger prompt-guided conditioning.ICL restricts decoded tokens to a musically favorable subspace guided by human-generated music, and CFG amplifies the text condition’s influence on next-token logits.Source: §8.5 · p. 24, Figure 14 · p. 24
  • 0.79 was the highest musicality win rate, achieved by ICL+CFG versus 0.63 for ICL and 0.21 for CoT.The authors attribute ICL’s advantage to conditioning on human-generated music prompts; CFG further amplifies the text condition.Source: Figure 14 · p. 24, §8.5 · p. 24

9. Representation Quality

Despite being a generative model evaluated with a limited, auxiliary representation-learning setup, YuE performs strongly on MARBLE: it leads on GS key recognition and remains competitive across several other tasks.

9. Representation Quality

  • Acc=67.0% on GS key recognition gives YuE state-of-the-art performance on the task.The evaluation uses YuE’s Stage-1 LM in an unconditional single-track setting.Source: §9 · p. 25
  • YuE remains competitive on GTZAN genre classification, MTG tagging, and EMO emotion regression.These results support the passage’s assessment of YuE’s general-purpose representation quality and learned musical skills.Source: §9 · p. 25
  • The evaluation uses an auxiliary task disabled halfway through training and only codebook-0 discrete codes, reducing available information versus dedicated representation models.YuE was designed as a generative model rather than explicitly for representation learning.Source: §9 · p. 25

10. Emergent Abilities

Scaling improves generation quality and unlocks emergent capabilities, including expressive vocal techniques, spontaneous performance behaviors, global style mixing, voice cloning, and multilingual adaptation.

Advanced Vocal Techniques

  • YuE spontaneously produces advanced techniques including vibrato, glissando, bel canto, death growl, riffs and runs, Beijing Opera, and Shanbei folk vocals.The authors attribute these capabilities to the Dual-NTP approach capturing subtle nuances in vocal performance.Source: §10 · p. 25

Spontaneous Performance

  • YuE generates expressive performance behaviors, including scat singing after lyrics run out, multipart a cappella harmonies, and contextually appropriate instrumental solos.Examples include harmonica interludes during vocal pauses in folk music.Source: §10 · p. 25

World Music & Pattern Mixing

  • YuE captures long-tail global styles and fuses them, such as Chinese gangsta rap with Japanese shamisen and combinations of Chinese vocal traditions.It blends Chinese opera, Shanbei folk singing, and traditional Chinese vocals within a cohesive performance.Source: §10 · p. 25

Voice Cloning

  • At inference time, YuE clones distinct vocal identities, reproducing Billie Eilish and Faye Wong’s timbral qualities, breathiness, and emotional nuances in new songs.The examples use new lyrics and melodies, with reference data provided only at inference.Source: §10 · p. 26

Style Transfer

  • YuE transforms a Japanese female J-pop vocal into English male rap while preserving the original city pop accompaniment and adapting prosody, phrasing, and expressiveness.The model supports cross-lingual and genre adaptation while preserving the original lyrical and melodic structure.Source: §10 · p. 26

Code Switching

  • YuE smoothly switches among languages or dialects within a vocal performance while preserving linguistic and stylistic consistency.Source: §10 · p. 26

11. Memorization Effect … 14. Ethics and Responsibility

YuE’s in-context generation shows no evidence of extensive copying, while its scaling experiments reveal training choices that hindered convergence, alignment, or musicality. The paper concludes that YuE supports open, controllable song generation and calls for transparent, culturally responsible use.

11. Memorization Effect

  • The memorization test used ByteCover2 cosine similarities for 1,200 training-reference and generated-song pairs, analyzing the top 1% against GTZAN and Covers80 baselines.ByteCover2 is described as a retrieval model optimized for melody-sensitive similarity across entire songs.Source: §11 · p. 26

12. Unsuccessful Attempts

  • Acoustic-token LMs converged poorly and had higher losses than semantic-enhanced-token LMs, which the authors attribute to limited-capacity, reconstruction-focused representations that encourage copying shortcuts.Even substantial scaling produced only marginal improvements [Hansen-Estruch et al., 2025, Xin et al., 2024, Parker et al., 2024].Source: §12 · pp. 26–27

11. Memorization Effect

  • Ref-Gen similarity was significantly below Covers80 and moderate relative to GTZAN, indicating that YuE’s ICL mode does not extensively copy training songs.The authors note occasional short repetitive motifs, particularly percussive loops.Source: §11 · p. 26, Figure 15 · p. 27

12. Unsuccessful Attempts

  • Unconditional pre-training helped sub-billion models learn basic mappings, but at 7B parameters fine-tuning failed to establish effective cross-modal alignment for lyrics-to-song.The authors hypothesize that overly generic priors create “catastrophic inertia” that impedes adaptation.Source: §12 · p. 27
  • Early ICL-data activation led to overreliance on reference audio, reduced musicality, and later invalid outputs such as silence or noise, with recovery becoming harder as models scaled.The authors conclude that ICL activation timing must be managed carefully to avoid overfitting and preserve creativity.Source: §12 · p. 27

13. Conclusion and Future Work

  • YuE combines large-scale data, track-decoupled prediction, segment-wise conditioning, and redesigned ICL to generate coherent full-length songs, matching or exceeding several commercial systems and achieving competitive music-understanding results.The authors identify acoustic fidelity, musical knowledge, prosodic and emotional controls, and multilingual and cross-cultural expansion as future directions; they also note potential uses in education, accessibility, and therapy.Source: §13 · pp. 27–28

14. Ethics and Responsibility

  • The authors advocate AI-generated or AI-assisted labels and describe culturally diverse training data and an ethics committee approval as supporting transparency and responsible cultural representation.Approval under protocol HREP-2023-0230 remains effective until 30-Jan-2027.Source: §14 · p. 28

A. Subjective Evaluation · A.1. Evaluation Methods · A.2. Evaluation Dimensions and Definitions

The subjective evaluation used blinded, randomized pairwise judgments across seven dimensions, assessing both the quality of individual musical elements and the coherence of the song as a whole.

A.1. Evaluation Methods

  • Annotators compared two pieces from different models after full playback, choosing whether A or B was superior or whether they were equivalent.Each dimension required a preference judgment, with instructions to minimize equivalence selections; presentation order was randomized under a double-blind protocol.Source: §A.1 · p. 35

A.2. Evaluation Dimensions and Definitions

  • Overall musicality covered professional-level artistic value, including melody smoothness, harmonic complexity, rhythmic precision, and arrangement creativity.The criterion asked whether the work approached the creative level of professional musicians or composers.Source: §A.2 · p. 35
  • Vocal quality assessed pitch and rhythmic stability, naturalness and richness of timbre, clarity, distortion, and expressive nuance.Criteria included breath control, articulation, and emotional conveyance.Source: §A.2 · p. 35
  • Accompaniment quality focused on realistic instrumental timbres and expressive detail, such as guitar-plucking subtlety and percussion dynamics.Source: §A.2 · p. 35
  • Arrangement complexity evaluated layering, instrumental interplay, balance, frequency range, element diversity, creativity, and variation or emotional progression across sections.Criteria included arrangement-layer clarity, low-frequency fullness, and high-frequency brightness.Source: §A.2 · pp. 35–36
  • Melodic memorability and catchiness measured how memorable and singable a melody was, its emotional resonance, and the presence of memorable hooks.Repeated hooks or memorable elements, especially in the chorus, were considered.Source: §A.2 · p. 36
  • Vocal-accompaniment matching assessed stylistic and musical compatibility in modality, harmony, and rhythm, including the absence of dissonance or conflict.Source: §A.2 · p. 36
  • Song structure clarity assessed sectional distinction, transition naturalness, and structural completeness across verses, choruses, and interludes.Source: §A.2 · p. 36

A.3. Conditional Evaluation Dimension and Definitions

The conditional evaluation rubric defines how to assess lyric performance, multilingual transitions, and whether generated music follows prompts for musical characteristics and expression.

A.3. Conditional Evaluation Dimension and Definitions

  • The rubric evaluates lyric fidelity and multilingual switching through accuracy, intelligibility, rhythmic alignment, fluency, pronunciation, and natural transitions.Source: §A.3 · p. 36
  • It also assesses genre and instrument or vocal configuration matching, alongside alignment between the prompt’s emotional description and the music.Source: §A.3 · p. 36
  • Tempo and rhythm are evaluated by checking whether the generated music matches the specified BPM and required rhythmic patterns.Source: §A.3 · pp. 36–37

B. Qwen2Audio-Instruct Tagging Prompt

The Qwen2Audio-Instruct tagging prompt requests audio descriptions as valid JSON, covering musical features and vocal characteristics.

B. Qwen2Audio-Instruct Tagging Prompt

  • The prompt asks for JSON lists for Music_genre, Instrument, and Mood, illustrated with jazz, saxophone and piano, and relaxed examples.Each key should contain a list of strings when it has multiple entries.Source: B. Qwen2Audio-Instruct Tagging Prompt · p. 37
  • A separate vocal-characteristics prompt requests JSON fields for gender, age, and vocal_timbre, with examples including female, adult, bright, and airy.Multiple entries for a key are represented as a list of strings.Source: B. Qwen2Audio-Instruct Tagging Prompt · p. 37

C. Multilingual Subjective Evaluation

The section presents a multilingual subjective comparison of YuE and other systems on lyrics following and musicality.

C. Multilingual Subjective Evaluation

  • Figure 17 compares YuE with other systems across languages on lyrics following and musicality.The supplied caption does not report scores or identify a winner.Source: Figure 17 · p. 38

D. 15 English Prompts From GPT

This section presents English song-lyric examples organized with labeled sections and recurring choruses, exploring themes including perseverance, searching for direction, romantic devotion, longing, and intimacy.

D. 15 English Prompts From GPT

  • The highlighted lyrics build a motivational rap around ambition, persistence, and overcoming setbacks, using recurring choruses and a bridge.The verses describe pursuing success and making family proud; the choruses and bridge reinforce determination.Source: D. 15 English Prompts From GPT · p. 39
  • A city-set narrative follows a solitary man searching for meaning, with a repeated chorus about feeling lost and a bridge suspended between darkness and light.The verses move through neon-lit streets and unanswered questions; the outro continues the image of chasing shadows.Source: D. 15 English Prompts From GPT · p. 40
  • Other lyrics portray romantic attachment through a speaker promising to remain by a loved one’s side, another recalling love that slipped away, and an angel’s hoped-for return and reciprocated love.Source: D. 15 English Prompts From GPT · pp. 41, 43
  • Further examples shift to intimate togetherness and a playful confession of love, with repeated choruses and a bridge in the confession song.One lyric asks a partner to stay close, while another builds toward finally speaking unspoken feelings.Source: D. 15 English Prompts From GPT · pp. 42, 48
  • One lyric portrays persistent devotion, with a narrator promising to follow and stand by a beloved who may leave.Its chorus repeats “I won’t back down,” while the verse and bridge reinforce the narrator’s determination.Source: D. 15 English Prompts From GPT · p. 51
  • The excerpts label verses, choruses, a bridge, and an outro, with the chorus refrain returning across the song.In one example, the repeated chorus is followed by a shortened outro.Source: D. 15 English Prompts From GPT · p. 52
  • Another lyric centers on intimate romantic closeness, asking a partner to stay and describing their presence as comforting.Verses and a bridge build around a recurring chorus about holding close and staying together.Source: D. 15 English Prompts From GPT · p. 52

Related papers

Loading 2503.08638v2…