MusicRL: Aligning Music Generation to Human Preferences

1. Introduction

• Sequence-level reinforcement learning can optimize rewards for sampled outputs, an approach the authors propose applying to music generation.The motivation draws on prior work optimizing sequence-level metrics in language generation.Source: §1 · p. 2
• Musicality is difficult to capture with automatic proxies, which only partially align with human preferences.Existing musicality approaches rely on complex music-theory rules and may be restricted to particular domains.Source: §1 · p. 2
• Large-scale user preference data may help bridge the gap between selected raters and the population interacting with a model.The paper notes that music preferences can be affected by this population gap.Source: §1 · pp. 2–3
• MusicRL-RU wins 62% against MusicRL-U and 66% against MusicRL-R, while MusicRL-R and MusicRL-U each outperform MusicLM.MusicRL-R wins 83% against MusicLM; MusicRL-U wins 74%.Source: §1 · p. 3
• MusicRL-R uses text-adherence and acoustic-quality rewards, while MusicRL-U uses a reward model trained on user preferences.The authors report that MusicRL-R is preferred over MusicLM 83% of the time and MusicRL-U 74% of the time.Source: §1 · p. 3

2. Related Work

• Stochastic inference can produce uncertainty in text adherence and quality, while reranking many sampled candidates increases inference cost and requires well-defined score functions.Prior work addresses uncertainty by sampling multiple sequences, scoring them, and returning the best candidate.Source: §2 · p. 3
• MusicRL finetunes MusicLM with reinforcement learning using rewards derived from automatic metrics, small-scale human ratings, and large-scale user feedback.The authors describe it as the first music-generation system to show benefits from feedback from hundreds of thousands of users.Source: §2 · p. 3
• Music-theory reward rules can be incomplete or contradictory, require balancing multiple rewards, and only approximate human musical preferences.Earlier approaches also use simple repetition patterns or rules such as staying in key and avoiding excessive note repetition.Source: §2 · pp. 3–4
• MusicRL’s training loop generates audio from music captions, scores the samples with a reward function, and uses reinforcement learning to maximize reward.Figure 2 depicts this process.Source: Figure 2 · p. 4
• Earlier learned-reward approaches can be limited to restricted domains such as monophonic piano or to symbolic generation, unlike MusicRL’s raw-audio preference learning across genres and styles.The paper names lo-fi hip-hop, orchestral symphonies, and modal jazz as examples.Source: §2 · p. 4

3.1. MusicLM

• MusicLM’s semantic and coarse acoustic stages are combined as interleaved tokens for RL fine-tuning, while Soundstorm handles parallel fine acoustic generation.The semantic tokens support long-term structure, and acoustic tokens support synthesis quality.Source: §3.1 · pp. 4–5

3.2. RL finetuning procedure

• RL updates the autoregressive policy to maximize reward using KL-regularized REINFORCE, with a learned value baseline to reduce policy-gradient variance.The baseline and policy both start from the initial MusicLM checkpoint.Source: §3.2 · p. 5

3.3. Reward Signals

• Text adherence uses MuLan prompt–audio cosine similarity, while acoustic quality uses a reference-free estimator trained to predict human ratings of clip quality.MuLan scores average three 10-second segments; quality scores average estimates from the first and last 20 seconds of each 30-second clip.Source: §3.3 · pp. 5–6
• User feedback comes from 300,000 pairwise preferences collected through an interface that lets users choose between two 20-second clips without attribute-specific instructions.Only preferences from users who listened to both clips are retained.Source: §3.3 · p. 6, Figure 3 · p. 6
• A Bradley–Terry reward model learns pointwise scores from pairwise preferences and reaches 60% accuracy on a 15,000-pair evaluation split.It is initialized from MusicLM because training from scratch did not predict preferences better than chance.Source: §3.3 · pp. 6–7
• On 156 comparisons, the team’s preferences matched dataset preferences in 60% of cases, highlighting subjectivity in music judgments.The paper contrasts this agreement with 73–77% reported for summarization preferences.Source: §3.3 · p. 7

4.1. Datasets

• Fine-tuning uses captions without ground-truth audio, combining synthetic descriptions and tags with MusicCaps captions and 300,000 user prompts.The caption set is randomly divided into 90% training and 10% evaluation data.Source: §4.1 · p. 7

4.2. Training procedure

• Reward variants share the same RL algorithm and hyperparameters; MusicRL-R uses MuLan and quality rewards, MusicRL-U uses user preferences, and MusicRL-RU adds user-preference tuning to MusicRL-R.The reported training durations are 20,000 steps for metric-reward variants, 5,000 for MusicRL-U, and 1,000 for MusicRL-RU.Source: §4.2 · pp. 7–8
• Sequentially fine-tuning first on MuLan and quality rewards, then on user preferences, outperforms learning from all three rewards simultaneously.The authors hypothesize that the user-preference reward begins to over-optimize in under 2,000 steps, while the other rewards take about 10,000 steps.Source: §4.2 · p. 8

4.3. Evaluation

• Human comparisons use 101 diverse prompts, three experienced raters per comparison, and a win rate defined as wins divided by wins plus losses.Raters assess prompt adherence, acoustic quality, and overall appeal; the evaluation totals 303 ratings per model comparison.Source: §4.3 · p. 8

4.4. Checkpoint selection

• The best checkpoint for each RL-finetuned model is selected by inspecting quantitative results and listening to generations.MusicRL-R, MusicRL-U, and MusicRL-RU use checkpoints from 10,000, 2,000, and 1,000 training steps, respectively.Source: §4.4 · p. 8

5.1. Quantitative Results

• Quality-only RL raises quality from 3.5 MOS to 4.6 MOS, while MuLan-only RL raises MuLan from 0.58 to 0.71.The figure plots these scores against KL divergence; for MusicRL-RU, KL is measured against MusicRL-R.Source: §5.1 · p. 8, Figure 4 · p. 9
• Joint quality and MuLan rewards raise quality from 3.5 MOS to 4.4 MOS and MuLan from 0.58 to 0.71, with a marginal KL increase.The authors describe the simultaneous score optimization as promising and stable.Source: §5.1 · p. 9, Figure 4 · p. 9
• After 10,000 quality-reward finetuning steps, the user-preference reward model assigns lower scores, suggesting susceptibility to reward over-optimisation.The authors cite prior work on reward over-optimization (Coste et al., 2023; Jiang et al., 2020).Source: §5.1 · p. 9
• User-preference RL raises its reward-model score from -1.5 to over 1.5, while quality moves from 3.5 MOS to 4 MOS and MuLan from 0.58 to 0.55.Over-optimizing the preference reward can drastically reduce MuLan; preference reward optimization on a quality-and-MuLan-finetuned model also improves its reward score.Source: Figure 5 · p. 9, §5.1 · p. 10

5.2. Qualitative Results

• MusicRL-RU wins 87% against MusicLM, 66% against MusicRL-R, and 62% against MusicRL-U; all RL-finetuned models outperform MusicLM on MOS and wins.MusicRL-R and MusicRL-U perform comparably, and MusicRL-RU is best on both Table 1 metrics.Source: §5.2 · p. 10, Table 1 · p. 10

5.3. Takeaway

• The authors conclude that text-adherence and quality rewards, user preferences, and their combination improve MusicLM, with MusicRL-RU performing best.They describe the reward signals as complementary.Source: §5.3 · p. 10
• Ablations suggest text adherence and audio quality are not primary preference drivers; removing text barely affects reward-model accuracy, while cropping audio substantially degrades it.The authors also report low user-preference prediction accuracy from text-adherence or audio-quality predictors.Source: Figure 6 · p. 11

6. Understanding Human Feedback Through the Lens of the Reward Model

• The analysis treats text adherence, audio quality, and musicality as possible components influencing users’ choices, while noting that musicality is complex to define and model.The authors frame human feedback as a way to move beyond rule-based limitations.Source: §6 · p. 11

6.1. Importance of the text input

• Removing text tokens leaves reward-model accuracy stable, and the highest-MuLan clip matches the preferred clip only 51.6% of the time, near random accuracy.The authors infer that prompt adherence was not a primary preference driver in their experiment.Source: §6.1 · p. 11

6.2. Importance of the audio quality

• Using 3–5 seconds of audio lowers preference-prediction accuracy from 60% to 56%; the quality predictor matches preferences with 53.3% accuracy.The authors conclude that audio quality is a better signal than text adherence, but is not the only preference driver.Source: §6.2 · pp. 11–12

7. Limitations and Future Work

• Feedback comes from general users, while selected raters evaluate results; the authors propose measuring perceived improvements directly from users.The current setup may therefore not reflect how feedback providers themselves perceive the results.Source: §7 · p. 12
• Preferences were collected on a different MusicLM version from the one used for RL finetuning; the authors propose iteratively collecting on-policy data.They envision deploying finetuned models to gather new feedback while improving user experience.Source: §7 · p. 12
• The preference dataset could be refined by retaining clear, confident choices to reduce noise and by training robust reward models on smaller, relevant datasets.The latter could support personalization for specific users.Source: §7 · p. 12

8. Conclusion

• MusicRL combines sequence-level rewards for caption adherence and acoustic quality with RL finetuning of pretrained MusicLM, then adds user-preference feedback through RLHF.The user-preference reward model uses 300,000 user-generated caption/audio pairs with pairwise preferences.Source: §8 · p. 12
• Both reward-based finetuning and RLHF consistently outperform the pretrained baseline, and combining all reward signals produces the highest-performing model.Source: §8 · p. 12
• Analysis indicates that user-preference feedback contains information beyond text adherence and audio quality, reflecting the subjective and complex nature of musical appeal.The authors identify integrating user feedback as valuable when improving music generation models.Source: §8 · p. 12

A. Qualitative Evaluation.

• Prompts include instrumental scenes such as a snare-and-lute duet, a fast piano cadenza, and sitar with tabla accompaniment.Source: A. Qualitative Evaluation · pp. 18–19
• Examples range across styles and ensembles, including trap with strings, accordion folk waltz, reggaeton, orchestral waltz, and industrial techno.Source: A. Qualitative Evaluation · pp. 18–19
• Other prompts feature varied vocals and combinations, from a church choir and female-led pop to rap, gospel, and solo cello.Source: A. Qualitative Evaluation · pp. 18–20

B. Additional Quantitative Evaluation Plots

• Over-optimizing user-preference reward improves quality but can significantly reduce MuLan score; the plots show both measures across training steps.Source: Figure 7 · p. 20, B. Additional Quantitative Evaluation… · p. 20
• RL finetuning improves the user-preference reward score most for preference-trained models; training on MuLan and/or quality yields only slight gains.Source: Figure 8 · p. 20

• Rater-trained models can improve music generation with RL, but rater data still have limitations and biases.Source: C. Advantages of User Data · p. 21
• Ecological validity concerns whether lab-study findings generalize to real-world settings, a crucial consideration for music.Source: C. Advantages of User Data · p. 21
• Raters may differ from deployed users in country and culture, potentially biasing music-preference judgments.Source: C. Advantages of User Data · p. 21
• Raters receive instructions and have different motivations from users, whose interactions are of interest without instructions.Source: C. Advantages of User Data · p. 21
• Rater preference datasets often contain fewer than 100,000 preferences, while deployed models can generate orders of magnitude more user interactions.Source: C. Advantages of User Data · p. 21