PaperlayerView exact revision on arXiv

Source-linked AI summary

CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction

Yinghao Ma, Haiwen Xia, Hewei Gao, Weixiong Chen, Yuxin Ye, Yuchen Yang, Sungkyun Chang, Mingshuo Ding, Yizhi Li, Ruibin Yuan, Simon Dixon, Emmanouil Benetos

arXiv:2603.00610v3cs.SDcs.AIcs.LGcs.MMeess.AS

TL;DR

Music reward modeling lacks unified evaluation and preference data for generation conditioned on text, lyrics, and reference audio. The paper builds datasets and CMI-RewardBench, then develops CMI-RM to score musicality and alignment across these conditions. CMI-RM provides a unified, parameter-efficient solution with reported gains over Gemini 2.5 Pro on compositional alignment and benefits for inference-time filtering.

  • Problem

    Existing music-evaluation data and metrics do not provide fine-grained, sample-level judgments across heterogeneous compositional instructions involving text, lyrics, and audio.

  • Method

    The paper constructs CMI-Pref-Pseudo and CMI-Pref, proposes CMI-RewardBench, and develops a parameter-efficient CMI-RM supporting text, lyrics, and audio in one architecture.

  • Results

    CMI-RM reaches 70.20% on CMI-Pref w/o audio (CMI+MusicEval) and 79.20% on w/ audio (CMI-Pref fine-tuned), surpassing Gemini 2.5 Pro at 67.2% and 72.8%, respectively.

  • Takeaways & Limitations

    CMI-RewardBench and CMI-RM provide a unified evaluation and reward-modeling framework for heterogeneous multimodal music-generation conditions.

  • Takeaways & Limitations

    The released pipeline includes audio generated via commercial APIs, so restricted components may require application-based access rather than unrestricted redistribution.

Abstract

from arXiv · show

While music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, evaluation mechanisms have lagged behind. In this paper, we bridge this critical gap by establishing a comprehensive ecosystem for music reward modeling under Compositional Multimodal Instruction (CMI), where the generated music may be conditioned on text descriptions, lyrics, and audio prompts. We first introduce CMI-Pref-Pseudo, a large-scale preference dataset comprising 110k pseudo-labeled samples, and CMI-Pref, a high-quality, human-annotated corpus tailored for fine-grained alignment tasks. To unify the evaluation landscape, we propose CMI-RewardBench, a unified benchmark that evaluates music reward models on heterogeneous samples across musicality, text-music alignment, and compositional instruction alignment. Leveraging these resources, we develop CMI reward models (CMI-RMs), a parameter-efficient reward model family capable of processing heterogeneous inputs. We evaluate their correlation with human judgment scores on musicality and alignment on CMI-Pref along with previous datasets. Further experiments demonstrate that CMI-RM not only correlates strongly with human judgments, but also enables effective inference-time scaling via top-k filtering. Code is available at GitHub (https://github.com/Haiwen-Xia/CMI-RewardBench). Model weights: CMI-RM (https://huggingface.co/HaiwenXia/CMI-RM). Datasets: CMI-Pref-Pseudo (https://huggingface.co/datasets/HaiwenXia/cmi-pref-pseudo) and CMI-Pref (https://huggingface.co/datasets/HaiwenXia/cmi-pref)

1. Introduction

Music generation evaluation must assess musicality and instruction following across text, lyrics, and audio conditions, but existing data and metrics remain fragmented. The paper addresses this gap with preference resources, a unified benchmark, and a parameter-efficient reward-model family for compositional multimodal instruction.

  • Flexible multimodal music evaluation must jointly assess musicality and instruction following for text-only, lyric-guided, and audio-referenced inputs.
  • Recommendation datasets capture user-item affinity rather than the fine-grained comparative judgments needed for generative alignment under complex multimodal instructions.
  • Existing evaluation approaches provide sample-level scores but remain fragmented across isolated attributes and narrowly specialized input settings.
  • CMI-RewardBench integrates diverse task-specific datasets to test whether one reward model can evaluate generation quality across heterogeneous instruction sets.
  • 110k pseudo-labeled samples and 4,027 expert-annotated pairs provide fine-grained preferences for musicality, alignment, and confidence across diverse multimodal prompts.
  • A unified five-task benchmark evaluates reward models from absolute musicality scoring through compositional alignment, exposing weaknesses even in state-of-the-art multimodal LLMs.Gemini-2.5-Pro struggles to exceed 80% agreement with human preferences.
  • CMI-RM supports text, lyrics, and audio in one approximately 30M-parameter architecture, matching or exceeding specialized baselines and benefiting top-k filtering.

2. Related Work

Prior music evaluation resources span distribution-level quality metrics, sample-level musicality predictors, alignment scores, preference datasets, and arena-style comparisons. However, these approaches generally remain restricted to particular attributes or text-to-music settings, motivating compositional multimodal evaluation.

  • Music evaluation has traditionally separated distribution-level quality assessment from sample-level alignment metrics, with existing approaches struggling under compositional instructions.
  • Distributional metrics such as FAD assess global corpus quality, while PAM, Audiobox, and SongEval provide sample-level musicality prediction.
  • CLAP, CLaMP3, and MuQ-Mulan measure alignment, but these metrics largely focus on text-to-audio pairs and omit lyrics or audio prompts.
  • Figure 1 frames reward models as proxies for human judgments of both musicality and instruction alignment under text, lyrics, and reference-audio prompts.
  • Expert-annotated corpora and community benchmarks support reward-model development, while newer pairwise resources primarily benchmark text-to-music preferences.
  • Music Arena extends comparative evaluation through a live platform, but existing resources primarily focus on text-to-music alignment rather than lyrics and audio-to-audio conditioning.

3. Method

The method combines heterogeneous music datasets, reward-model baselines, and a compositional architecture to evaluate musicality and instruction alignment across varied input conditions. It also uses held-out benchmark data and best-of-N filtering experiments to assess reward-model utility.

  • Preference datasets: CMI-Pref-Pseudo supplies 110k consistency-checked preference pairs, while CMI-Pref contains 4,027 human-annotated samples with musicality, alignment, confidence, and rationale information.CMI-Pref-Pseudo is generated with Qwen3-Omni; CMI-Pref was constructed by 31 annotators.
  • Preference datasets: CMI-Pref covers text, text+lyrics, text+audio, and text+audio+lyrics conditions, with a balanced 500-pair test set and re-annotation agreement of 75.2% for musicality and 75.0% for alignment.The test set is split equally across the four conditioning patterns.
  • Benchmark construction: CMI-RewardBench uses held-out PAM, MusicEval, Music Arena, and CMI-Pref data to evaluate scalar ratings and pairwise preferences across musicality, text-music alignment, and compositional conditions.Its test resources include 500 PAM ratings, 413 MusicEval ratings, 1,340 Music Arena pairs, and 500 CMI-Pref pairs.
  • Baseline models: The benchmark compares specialized musicality predictors, similarity-based alignment metrics, general-purpose reward models, and zero-shot AudioLLMs across its evaluation tasks.The evaluated baselines include PAM, audiobox, SongEval, CLAP-Score, CLAMP3, MuQ-MuLan, Omni-Reward, Qwen models, and Gemini models.
  • Model architecture: CMI-RM is designed to handle multimodal conditioning and predict fine-grained human scores for generated music.The supplied method description identifies multimodal conditioning and fine-grained human-score prediction as the architecture’s purpose.
  • Inference-time scaling: For each MusicCaps prompt, MusicGen-small and Stable-Audio-Open-small generate 10 samples, and CMI-RM selects top-performing candidates with N ∈ {1, 3, 10} for subjective A/B testing.The experiment uses 2,183 text prompts and 10-second audio samples.

4. Discussion

CMI-RM performs strongly across musicality and compositional alignment evaluations, including settings with text, lyrics, and reference audio. Ablations show that human preference fine-tuning and pseudo-label distillation improve generalization and test-time reranking yields gains with diminishing returns.

  • 78.20% accuracy on CMI-Pref musicality exceeds Gemini 3 Pro at 65.80% and Qwen3-omni at 60.40%.
  • Compositional alignment: 79.20% accuracy with audio and 70.20% without audio show CMI-RM outperforming Gemini 2.5 Pro at 72.8% and 67.2%, respectively.
  • Compositional alignment: 82.40% on the Text+Lyrics+Audio subset exceeds Gemini 3 Pro at 66.8%, highlighting strong performance on the hardest modality combination.
  • Training-set ablations: Fine-tuning on CMI-Pref improves PAM, MusicEval, Music Arena, and CMI-Pref over Distill, reaching 0.6116, 0.7315, 71.41%, and 75.90%.
  • Training-set ablations: Joint CMI-Pref and MusicEval training reaches 0.8266 on MusicEval, 73.43% on Music Arena, and 76.05% on CMI-Pref, with a mild PAM trade-off.
  • Training-set ablations: Distillation initialization improves Music Arena from 71.34% to 73.43%, PAM from 0.2630 to 0.5464, and CMI-Pref from 72.15% to 76.05%.
  • Test-time scaling: Best-of-N reranking improves scores across backbones, but gains are smaller and begin saturating for Stable-Audio-Open-small.

5. Conclusion

The paper introduces a unified ecosystem for evaluating music reward models under compositional multimodal instruction, combining datasets, benchmark tasks, and a parameter-efficient reward model. CMI-RM supports heterogeneous inputs and enables measurable gains through best-of-N reranking.

  • CMI-Pref-Pseudo provides 110k pseudo-labeled pairs, while CMI-Pref provides 4k expert annotations with confidence information.
  • CMI-RewardBench integrates CMI-Pref with existing resources to evaluate reward models across five tasks spanning absolute scoring and pairwise preference.
  • CMI-RM is a parameter-efficient model supporting text, lyrics, and audio conditioning within one architecture.
  • CMI-RM achieves performance competitive with or exceeding specialized open-source baselines and improves best-of-N reranking as an inference-time scaling strategy.

Conflict of Interest Disclosure

The disclosure reports Google support connected to the evaluation of Gemini models and identifies this support as a potential financial conflict of interest.

  • Yinghao Ma acknowledges support from the Google PhD Fellowship.
  • Because the work evaluates Gemini models developed by Google, the support is disclosed as a potential financial conflict of interest.

Impact Statement

The impact statement combines a multimodal music-evaluation resource with documentation, release, consent, and licensing practices. The accompanying analyses characterize heterogeneous data and annotator behavior relevant to interpreting the benchmark.

  • The release follows a TOS-aware policy, uses informed-consent and data-minimization procedures, and addresses copyright concerns through takedown and correction mechanisms.
  • CMI-Pref contains diverse prompts and lyrics spanning musical attributes, languages, styles, and compositional instructions.
  • The dataset combines short and long instructions, multiple semantic controls, diverse musical styles, and lyric-conditioned inputs for compositional multimodal evaluation.
  • The audio-duration distribution is multimodal, with mean 69.4 seconds and median 30.0 seconds, indicating a long right tail.
  • Annotators’ listening time is long-tailed, with mean 22.8 seconds and median 13.5 seconds; most samples fall within 50 seconds.
  • Pearson r = 0.279 indicates only a mild association between audio duration and total listening time across 8,046 filtered samples.
  • Most listening-time points fall below y = x, indicating that annotators often decide before fully listening to a clip and vary effort by sample.
  • Re-annotation agreement reaches 75.2% for musicality and 75.0% for alignment on the 500-pair test set.

A.4. Confidence Scores of Human Annotation

The annotation analysis examines confidence, agreement, positional bias, and pseudo-label filtering. It finds that confidence tracks agreement and that human preferences can diverge between musicality and instruction alignment.

  • Most votes have relatively high confidence, with music-quality judgments slightly more confident than instruction-following judgments.
  • Votes have higher average confidence when instruction-following and music-quality preferences agree, while disagreements often indicate ambiguous trade-offs.
  • Agreement between musicality and instruction-alignment preferences is 91% for pseudo-labels and 81% for human data.
  • Human conflicts can pair high-confidence instruction alignment with lower musicality, supporting separate evaluation of the two dimensions.
  • Position-Consistency filtering retains pseudo-labels only when forward and reverse presentations select the same underlying audio.

B.2. Bias Analysis and Dataset Statistics

The dataset construction reveals substantial positional bias in pseudo-labeling, which is reduced by retaining only order-consistent comparisons. Human annotation then separates preference, confidence, and qualitative rationale across musicality and alignment.

  • Bias Analysis: 51.96% to 59.27%: Candidate A’s Musicality win rate changes when pair presentation order is reversed, demonstrating positional bias.The original and reversed configurations produce different label distributions.
  • Human Annotation: CMI-Pref annotation combines forced-choice preference labels, 1–5 confidence scores, and free-text feedback.These components capture both the decision and its certainty and justification.
  • Human Annotation: Annotators evaluate instructions before listening, review both holistic and granular qualities, and keep alignment distinct from musicality.Alignment concerns prompt adherence, whereas musicality concerns finished, natural, professional sound quality.
  • Human Annotation: Confidence ranges from 5 for clear distinctions to 1 for ambiguous or indistinguishable outcomes.Feedback is expected to cite audible evidence, including specific timestamps when useful.

D. Limitations

The paper positions CMI-RM as a practical evaluator rather than a complete downstream reward-optimization solution. It also identifies calibration, disentanglement, data-shift, and demographic coverage boundaries.

  • Scope: CMI-RM is a practical baseline evaluator, not a complete solution for downstream reinforcement-learning reward optimization.The authors explicitly do not claim that one reward model alone prevents reward hacking.
  • Disentanglement: 82% human agreement between musicality and alignment votes contrasts with 0.853 model SRCC, indicating incomplete disentanglement when dimensions conflict.The correlation can aid transfer but may blur distinct preferences.
  • Coverage Bias: Musicality scores vary by genre, and alignment performance is stronger for English than for low-resource languages.Pairwise preference accuracy shows a less severe genre effect than musicality scores.
  • Calibration: Pseudo-pretraining exhibits distribution shift: real-test cross-entropy can exceed 1.2 versus 0.693 for random binary guessing, reflecting confident errors.Accuracy changes less than cross-entropy because confidently wrong predictions are penalized heavily.
  • Calibration: ε = 0.2 label smoothing improves fine-tuned accuracy to 77.8% Musicality and 74.0% Alignment, but stopping-point robustness varies across datasets.Music Arena declines after roughly 6k steps, while PAM is more sensitive to over-training.

E.2. Ablation on Pseudo-Label Data Size

Pseudo-label scale improves transfer and downstream performance until gains saturate, making 64k examples a compute-efficient operating point. Label smoothing is more reliable than mixing teacher sources, while larger heads help some benchmarks.

  • Pseudo-Label Data Size: 64k pseudo-labeled examples perform comparably to 110k under fixed-epoch training, suggesting diminishing returns beyond 64k.The authors adopt 64k as a compute-efficient operating point for later experiments.
  • Pseudo-Label Data Size: 4k–16k to 32k–64k: increasing pseudo-data size generally strengthens transfer and downstream performance before saturation at larger scales.The comparison uses Pref-Test accuracy and cross-entropy alongside downstream results.
  • Pseudo-Label Sources: Source mixing yields modest and inconsistent gains: it improves PAM and CMI-PREF alone but slightly harms Music Arena.Combining mixing with smoothing gives the best CMI-PREF and Music Arena results but degrades PAM.
  • Pseudo-Label Sources: Label smoothing remains the more reliable regularizer, whereas multi-source distillation mainly changes the teacher boundary.This conclusion follows the cross-dataset comparison of Qwen-only and mixed pseudo-label supervision.
  • Model Capacity: 8.3M reward heads remain competitive on PAM, MusicEval, and CMI-PREF, while the 102M head achieves the strongest overall results.The larger head is especially stronger on MusicEval and CMI-PREF.
  • Text Encoder: Flan-T5-Large gains on one text-only subset, is neutral for text-plus-lyrics, and consistently degrades with audio conditioning.The pattern suggests cross-modal audio-text fusion, rather than text-encoder capacity, is the current bottleneck.
  • Mapping Functions: Mapping-function choice produces no significant performance differences across the evaluated None, Tanh, Linear, and Ordinal settings.All variants begin from the same pseudo-pretraining checkpoint and are fine-tuned on CMI-Pref and MusicEval.

F.4. The Effect of Duration on Model Inference

Longer audio context generally improves agreement with full-context scores, though short excerpts remain adequate for some preferences and are cheaper to use. Prompt conditions also materially improve musicality prediction, especially with reference audio.

  • Inference Cost: Mean 10 is the most compute-intensive inference strategy because it requires more forward passes.With a 120-second maximum and batching, its memory overhead is only slightly larger than first 120.
  • Duration Effects: First 120 achieves the best Music Arena accuracy in most duration bins, while mean 10 is comparable in several cases.On CMI-Pref, first 10 remains competitive in shorter-duration bins.
  • Duration Effects: First 10 versus first 120 yields RMSE-to-STD ratios of 0.5575 for Musicality and 0.6133 for Alignment, showing substantial duration mismatch.Short-context predictions become less correlated with long-context scores as track length grows.
  • Duration Effects: First 10 can provide a coarse preference signal for some downstream uses, but longer temporal context remains beneficial.The authors leave more reliable long-range modeling for future work.
  • Prompt Contribution: 75.60% versus 70.20%: adding prompt conditions improves overall musicality prediction accuracy by 5.40%.The text-only setting gains 1.60%.
  • Prompt Contribution: +13.60% and +10.40%: reference-audio modalities produce the largest musicality gains, whereas text plus lyrics without audio declines by 4.00%.The decline is attributed to difficulty processing raw lyrics without acoustic grounding.

H.1. Significance of Benchmark Improvements

The benchmark analyses show that key preference-based gains are statistically supported, while reward-model performance depends on reranking conditions, temporal shift, annotation confidence, and evaluation proxy choice. CMI-RM captures broader human preferences than lyric-transcription accuracy alone and supports unified multimodal model ranking.

  • Significance checks: 78.60% vs 70.00% musicality accuracy distinguishes CMI-Pref from Gemini2.5-Pro, with p = 9.30 × 10−4.The comparison is reported as a significance check on CMI-Pref musicality.
  • Significance checks: 73.21% vs 69.75% musicality accuracy distinguishes CMI+MusicEval from Gemini2.5-Pro on Music Arena, with p = 2.37 × 10−2.A high-confidence split also reports 85.71% vs 75.96% against Gemini2.5-Pro, with p = 1.09 × 10−3.
  • Test-time scaling: Top-k reranking was evaluated with 2,800 prompts objectively and 50 prompts subjectively across GT, Top-10, Top-3, and Top-1 ranks.Subjective listening produced 200 generations and 300 pairwise comparisons; ties were split equally for exact one-sided binomial testing.
  • Lyrics Transcription as a Proxy Metric: WER-based preference prediction reaches only 60.0% accuracy versus 76.0% for CMI-RM on the same lyric-conditioned setting.Human judgments also consider vocal naturalness, prosody, musicality, structural coherence, and lyric integration with musical style.
  • Confidence analysis: Higher-confidence comparisons are consistently easier to predict, and CMI-Pref fine-tuning produces its largest gains in the high-confidence regime.At low confidence, method differences narrow because near-tie comparisons have smaller preference margins.
  • Unified model ranking: The unified leaderboard uses prompt-wise pairwise CMI-RM comparisons aggregated with a Bradley–Terry-style ranking across modalities and alignment or musicality dimensions.The results report a large open-weight versus closed-source gap, while LEVO, MAGENTA REALTIME, and ACESTEP show competitive open-source performance.
Loading 2603.00610v3…