Aligning Generative Music AI with Human Preferences: Methods and Challenges

Introduction

• Likelihood-based training captures surface musical patterns but misses aesthetic and emotional qualities that matter to human listeners.These objectives optimize for training distributions, leaving a gap between computational optimization and human musical appreciation.Source: Introduction · p. 1
• Musical quality resists simple evaluation because it spans long-range coherence, harmony, rhythm, timbre, and subjective aesthetic judgment.These judgments can vary across individuals and cultural contexts.Source: Introduction · p. 1
• Musical quality is user- and context-dependent, and preferences shift with personal experience and changing aesthetic landscapes.The same prompt can support interpretations across genres and emotional functions.Source: Introduction · pp. 1–2
• MusicRL (Cideron et al. 2024) trained on about 300,000 pairwise preferences and outperformed baselines in human evaluations, but its preference data are proprietary.The work illustrates the multidimensional nature of musical preference beyond simple quality metrics.Source: Introduction · p. 2
• DiffRhythm+ (Chen et al. 2025) aligns multiple preferences for full-length songs, from local harmonic progressions to global compositional form.Text2midi-InferAlign (Roy, Puri, and Herremans 2025) instead uses tree search at inference time to adapt preferences without retraining.Source: Introduction · p. 2
• Preference alignment spans large-scale human-feedback learning, inference-time multi-objective optimization, and preference-trained generative architectures.Figure 1 distinguishes training-time optimization from inference-time adaptation without retraining.Source: Introduction · p. 2, Figure 1 · p. 2
• The paper surveys alignment methods and research directions toward music AI that serves human creative needs.It traces related work from reinforcement-learning systems to DPO and inference-time methods.Source: Introduction · p. 2

Background

• RLHF trains a reward model from human preference comparisons, then optimizes a policy while regularizing it toward a reference policy.The second phase can use policy-gradient methods such as PPO, with a KL term controlling similarity to the reference policy.Source: Reinforcement Learning from Human… · p. 3
• RLHF can be unstable and computationally demanding, and reward hacking can exploit weaknesses in the reward model rather than improve quality.These limitations are attributed to its multi-phase training and reward-model dependence.Source: Reinforcement Learning from Human… · p. 3
• DPO removes explicit reward-model training and uses preference comparisons to optimize the policy directly.Because the partition function is independent of the output, comparisons eliminate it; the passage describes the resulting optimization as stable and computationally advantageous.Source: Direct Preference Optimization · p. 3
• Inference-time methods adapt generation without retraining through contrastive decoding, preference-conditioned sampling, or control-vector steering.They can balance text adherence, audio quality, stylistic consistency, and temporal coherence during generation.Source: Inference-Time Alignment Paradigms · p. 3

Musical Preference Complexity

• FAD, IS, and CLAP-based similarity do not capture the subjective nature of musical appreciation or its interacting preference dimensions.The section motivates specialized approaches that respect musical preference’s multidimensional and subjective nature.Source: Musical Preference Complexity · pp. 3–4
• Preference alignment must support coherence across local rhythms, phrases, and global musical forms spanning minutes or hours.These temporal scales range from milliseconds-to-seconds patterns to compositions lasting minutes-to-hours.Source: Musical Preference Complexity · p. 3
• Musical judgments depend on whether harmonic progressions fit their context and on cultural frameworks that shape aesthetic expectations.What counts as good music varies across traditions, historical periods, and social communities.Source: Musical Preference Complexity · pp. 3–4
• Optimization criteria also depend on music’s role, from background ambiance and dance to focused listening, games, and emotional expression.These distinct functions require different criteria.Source: Musical Preference Complexity · p. 4

Large-Scale Preference Learning with MusicRL

• MusicRL combines expert-designed sequence-level rewards with large-scale user pairwise preferences to train preference-aligned music generation.The rewards address text adherence, audio quality, and musical-structure coherence; its preference dataset is private.Source: Large-Scale Preference Learning with… · p. 4
• Human evaluators prefer qualities beyond simple audio quality or text matching, which traditional metrics fail to capture.Source: Large-Scale Preference Learning with… · p. 4
• Both MusicRL-R and MusicRL-U significantly outperform baseline MusicLM in human evaluations across diverse musical styles, with MusicRL-RU achieving the strongest performance.Ablations also show that text adherence and audio quality account for only part of human preferences.Source: Large-Scale Preference Learning with… · p. 4

Multi-Preference Diffusion Optimization with DiffRhythm+

• DiffRhythm+ integrates DPO into diffusion training and uses MuLan style conditioning to control musical attributes in full-length song generation.Its modified objective is adapted to continuous latent spaces, and its evaluation incorporates SongEval and Audiobox-aesthetic criteria.Source: Multi-Preference Diffusion… · p. 4
• Diffusion architectures can optimize global structure and long-range dependencies simultaneously, with reported gains in long-form coherence and aesthetic appeal over baseline diffusion models.Source: Multi-Preference Diffusion… · p. 4
• MusicRL and DiffRhythm+ illustrate the need to balance competing musical objectives; DPO is described as balancing generation objectives while maintaining diversity.The complexity of these objectives motivates preference models that explicitly represent and balance competing criteria.Source: Multi-Preference Diffusion… · pp. 4–5

• Text2midi-InferAlign uses tree search at inference to balance weighted text-audio consistency and harmonic consistency rewards.Source: Inference-Time Alignment Techniques · p. 5
• Caption mutation explores semantically related input variations while maintaining the caption’s core meaning.The reward mechanism navigates the search space without exhaustive enumeration.Source: Inference-Time Alignment Techniques · p. 5
• 29.4% improvement in CLAP scores versus baseline text2midi generation, without model modification, while preserving diversity and enhancing quality.Tree search adds computational overhead, creating a quality–inference-latency trade-off for real-time use.Source: Inference-Time Alignment Techniques · p. 5

• Fréchet Audio Distance and Inception Score provide baselines but fail to capture musical-specific qualities or aesthetic appeal.Source: Benchmarking and Evaluation · p. 5
• Evaluation approaches pair human preference data with text-audio and harmonic consistency metrics, while DiffRhythm+ uses SongEval and Audiobox-aesthetic for broader quality assessment.MusicRL’s pairwise preference data shows automated metrics correlate only partially with human judgment.Source: Benchmarking and Evaluation · p. 5
• Evaluation still needs metrics for diverse musical traditions and tastes, standardized long-form coherence benchmarks, and personalized-preference protocols; western-heavy content limits global applicability.Source: Benchmarking and Evaluation · p. 5

• Preference learning adds training overhead, while inference-time optimization adds computational costs after deployment.Training-time methods add development costs but can deploy with minimal inference overhead; inference-time methods require resource allocation to maintain response times.Source: Technical Implementation Considerations · p. 5
• Preference-data systems need collection platforms, evaluator quality control, and aggregation that accounts for individual biases and cultural backgrounds.Implementation considerations also include privacy, consent, demographic representation, explainability, and quality assurance.Source: Technical Implementation Considerations · p. 5
• Musical preference datasets may need to store both symbolic representations such as MIDI and audio, creating substantial storage requirements.Indexing must support retrieval by harmonic similarity, rhythm, and style.Source: Technical Implementation Considerations · p. 5
• DPO training in diffusion models requires gradients through the denoising chain, creating memory-scaling challenges beyond standard diffusion training.The passage identifies gradient checkpointing and mixed-precision computation as techniques for practical hardware configurations.Source: Technical Implementation Considerations · p. 5
• Preference alignment may require monitoring for preference drift and triggering retraining when alignment quality degrades below acceptable thresholds.Source: Technical Implementation Considerations · p. 6

Challenges and Open Research Questions

• Current approaches struggle with long-form compositions because of attention complexity and the challenge of modeling hierarchy across temporal scales.Music spans local rhythmic patterns through phrase-level structure to global form, from milliseconds to hours.Source: Challenges and Open Research Questions · p. 6
• Multimodal alignment must address video-music synchronization and cross-cultural media integration while maintaining musical coherence during real-time adaptation.Source: Challenges and Open Research Questions · p. 6
• Personalization remains largely unexplored, while systems reflecting predominantly Western traditions limit global applicability and risk cultural appropriation.The passage identifies few-shot preference learning and efficient representations of individual aesthetic judgments as needs.Source: Challenges and Open Research Questions · p. 6
• Inference-time optimization faces computational overhead, energy consumption, and latency constraints for interactive applications.Source: Challenges and Open Research Questions · p. 6
• Traditional metrics capture only partial aspects of human musical judgment, motivating richer representations of implicit understanding, emotional response, and contextual appropriateness.Source: Challenges and Open Research Questions · p. 6
• Cross-domain generalization remains open because current evaluation frameworks are domain-specific, and universal alignment across genres, cultures, and applications is unresolved.DiffRhythm+ relies on SongEval and Audiobox-aesthetic, while Text2midi-InferAlign uses CLAP-based optimization.Source: Challenges and Open Research Questions · p. 6
• Dynamic preference adaptation is a gap: systems need efficient incremental preference learning and theoretical accounts of how tastes change over time.The passage names exposure, expertise development, and cultural evolution as factors in preference change.Source: Challenges and Open Research Questions · p. 6
• Assessing preference alignment itself requires human judgment, creating a circular evaluation dependency that may reproduce biases and limitations.Source: Challenges and Open Research Questions · p. 6

Vision and Roadmap

• Preference-aligned systems can support interactive composition, adaptive film scoring, game audio, therapeutic music generation, and personalized music services.Source: Vision and Roadmap · p. 6
• Technical priorities include open, culturally diverse preference datasets, lower-overhead multi-objective inference, and cross-cultural evaluation frameworks.Source: Vision and Roadmap · p. 6
• Serving diverse creative needs requires collaboration across machine learning, music theory, cognitive science, and ethics.Source: Vision and Roadmap · p. 6

Conclusion

• The paper advocates systematic preference alignment to address the gap between computational optimization and human appreciation.Its examination spans large-scale preference learning, inference-time optimization, and multi-preference diffusion.Source: Conclusion · p. 6
• Preference alignment improves quality and relevance while preserving diversity.Source: Conclusion · p. 6
• Interdisciplinary research combining machine learning, music theory, and human-computer interaction is called for to create music AI serving human creative needs.Source: Conclusion · p. 6