Source-linked AI summary
Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals
Xiaosha Li, Chun Liu, Ziyu Wang
TL;DR
Whether a music generator can evaluate its own outputs from internal signals matters because waveform-only verifiers approximate ratings without addressing why listeners prefer one clip. The paper aligns MusicGen loss, entropy, and SAE latents with human ratings across five benchmarks; the signals track ratings, with SAE latents carrying most predictive signal.
Problem
Waveform-based verifiers approximate human ratings, but leave open why listeners rate one generated clip more highly than another.
Source: §1 · p. 1Method
The paper extracts per-token loss, predictive entropy, and SAE latents from a pretrained music generator and aligns them with ratings across five benchmarks.
Source: §1 · pp. 1–2Results
Across five benchmarks, a frozen MusicGen’s intrinsic signals align with human ratings, with SAE latents carrying most predictive signal and loss–entropy mismatch marking low-rating moments.
Source: §1 · p. 2, §4.5 · p. 5, §6 · p. 6Takeaways & Limitations
Intrinsic signals are a viable, underused axis for automatic music evaluation within the tested setting.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Current music generative models can produce high-quality music, but does this ability imply that they ``understand'' the musical qualities of their outputs, and is that understanding aligned with human evaluation? Previous attempts to use the likelihood of a generative model to evaluate music, an approach commonly used in text, have proven unsuccessful, leading researchers to rely on standalone supervised music evaluation models. In this paper, we answer this question affirmatively: we show that a model's intrinsic signals---derived from its hidden representations and predictions---are strongly correlated with human ratings. In particular, we study MusicGen and consider three types of features: (1) prediction loss, (2) prediction entropy, and (3) concepts extracted from the model using a sparse autoencoder (SAE). Using these features, we train a lightweight prediction model to estimate subjective ratings. We evaluate these features both individually and in combination. We hypothesize that these signals parallel the listening process: the temporal and frequency-domain structure of loss and entropy reflects listeners' expectation and surprise, while gradient directions in SAE latent space predict perceived quality. Experiments on five human-evaluation benchmarks spanning continuous ratings and pairwise preferences confirm this hypothesis, with SAE latents carrying most of the predictive signal.
1. INTRODUCTION
The paper asks whether a music generator can rate its own outputs from internal signals, addressing a gap left by waveform-only verifiers. It tests this with MusicGen signals aligned to human ratings across five benchmarks, finding predictive value in both loss–entropy patterns and SAE latents.
1. INTRODUCTION
- Existing audio-based verifiers approximate human ratings from generated audio but leave open why listeners rate one clip more highly than another.The cited examples include Meta Audiobox Aesthetics and music-instruction benchmarks.Source: §1 · p. 1
- The paper frames listening as an alignment between exposure-shaped representations and expectations and the audio encountered, drawing on music-cognition accounts of familiarity and surprise.The cited accounts include the Mere Exposure Effect and affective responses to expectation violations.Source: §1 · p. 1
- Model-internal signals from one pretrained music generator align with human ratings across five benchmarks under a unified protocol.The signals are per-token loss, next-token predictive entropy, and sparse-autoencoder latents.Source: §1 · pp. 1–2
- The overconfident-wrong loss–entropy mismatch is the dominant local correlate of low ratings, with audio captioning corroborating the pattern.The mismatch refers to high loss paired with low entropy.Source: §1 · pp. 1–2
- SAE latents trained without ratings separate highly rated clips from poorly rated clips, corroborated by audio captioning and a professional producer.The paper presents this as representational evidence that the latents organize clips along a quality axis.Source: §1 · p. 2
2. RELATED WORK
Prior music evaluation largely scores waveforms or human-rated outcomes, while music cognition and model interpretability suggest complementary internal signals. This paper situates loss, entropy, and SAE representations within those strands.
2. RELATED WORK
- Music evaluation connects waveform-based verifiers and human-rated benchmarks, while model-internal signals remain a largely missing strand for music.The paper positions loss, entropy, and sparse representations as signals commonly used in language modeling and interpretability.Source: §2 · p. 2
- Audiobox Aesthetics scores production and content qualities, while CMI-Bench and CMI-RewardBench evaluate music instruction following and reward-modeling dimensions.The listed dimensions include production quality, content enjoyment, musicality, and text–music alignment.Source: §2 · p. 2
- Human-evaluation benchmarks span expert ratings, full-song ratings, pairwise battles, and survey-style comparisons, but judge from waveforms and leave the why of preference unaddressed.The paper classifies its method as reference-free quality prediction trained on human ratings, using generator internals rather than audio embeddings.Source: §2 · p. 2
- Music-cognition accounts link familiarity and expectation violations to preference, and autoregressive models encode training-shaped representations and next-token expectations.The cited accounts include the Mere Exposure Effect and affective weight from expectation violations.Source: §2 · p. 2
- Prior work finds mean loss alone insufficient for musicality, while curve shape and entropy can carry complementary quality or error information at different temporal scales.Perplexity likewise cannot distinguish confidently correct from confidently incorrect predictions [21].Source: §2 · p. 2
- SAEs map dense activations to sparse, more interpretable units; this paper repurposes music concepts previously used for steering to test alignment with human ratings.The prior music-model SAE work is Singh et al..Source: §2 · p. 2
3. METHODS
The method extracts loss, predictive entropy, and SAE latents from a pretrained autoregressive music model, then trains a neural network to predict human ratings. It applies a unified training objective across continuous and pairwise-preference benchmarks.
3.1 Model-Intrinsic Signals
- Per-step loss, predictive entropy, and SAE latents provide complementary signals from the model’s output distribution and hidden state.Loss is the ground-truth token’s negative log-likelihood; predictive entropy measures uncertainty over next-token predictions, while SAE latents represent hidden activations as sparse, concept-aligned features.Source: §3.1 · p. 3
3.2 Prediction Model
- The three-branch hybrid uses a dedicated 1D convolutional encoder for each signal, then combines their representations by late concatenation.Single- and two-signal ablations drop branches; the shared MLP predicts a clip-level rating.Source: §3.2 · p. 3
3.3 Training Objective Across Datasets
- All variants are trained with one mean-squared-error objective against a per-clip target, regardless of supervision format.MusicEval and SongEval provide average expert ratings directly; MusicPref, AIME, and Music Arena pairwise outcomes are converted to continuous ratings using Elo updates.Source: §3.3 · p. 3
- Table 1 reports Pearson r and Spearman ρ, with trained models evaluated on held-out test splits and Audiobox evaluated zero-shot on full datasets.The table also gives results for the union of the five held-out benchmark splits.Source: Table 1 · p. 4
4. EXPERIMENTS
The experiments compare intrinsic music-model signals with human ratings across five benchmarks and against audio-only and mean-loss baselines. Intrinsic models outperform those baselines, with SAE-based models leading on most benchmarks, although no combination dominates.
4. EXPERIMENTS
- The evaluation compares the hybrid and its ablations with audio-only and intrinsic baselines on held-out test splits across five human-evaluation datasets.Source: §4 · p. 4
4.1 Datasets
- Five public benchmarks span about 241 hours, combining direct expert ratings and pairwise preferences while focusing evaluation on musicality.The benchmarks are MusicEval, SongEval, AIME, MusicPref, and Music Arena.Source: §4.1 · p. 4
4.2 Data Processing
- Direct ratings are averaged and pairwise outcomes converted to continuous Elo targets; clips are matched to raters’ audio, then chunked into 30-second feature streams.Elo rescoring tracks empirical win rate at r≥0.86 across the three pairwise benchmarks.Source: §4.2 · p. 4
4.3 Model Training
- A CNN encodes loss and entropy curves, a parallel path compresses SAE codes, and an MLP predicts ratings; all seven signal combinations use matched splits, preprocessing, and seeds.The full hybrid has about 1.24 million parameters.Source: §4.3 · p. 4
4.4 Baselines
- Baselines are validation-calibrated Audiobox scores on four aesthetics axes and a regressor using mean token loss.Audiobox axes are Content Enjoyment, Content Usefulness, Production Complexity, and Production Quality.Source: §4.4 · p. 5
4.5 Results
- Every intrinsic model beats Audiobox and mean loss; the single best beats its runner-up in only 4 of 24 columns, and SAE models lead on most benchmarks.The strongest single signal is SINGLE-SAE except on MusicPref, where loss leads.Source: §4.5 · p. 5
5. ANALYSIS
The analysis relates token-level loss and entropy configurations, their spectral dynamics, and SAE concepts to human ratings, using the MusicEval clean test split for the loss–entropy analyses.
5.1 Loss–Entropy Configurations
- On MusicEval’s clean test split (n=270), overconfident-wrong tokens correlate with lower ratings (r=−0.18), while uncertain-wrong tokens correlate with higher ratings (r=+0.29).Uncertain-correct tokens correlate with slightly lower ratings (r=−0.12), while confident-correct tokens show no correlation (r=+0.03, n.s.); an audit found 6/10 overconfident-wrong windows audibly disruptive versus 3/10 uncertain-wrong windows. The uncertain-wrong minus overconfident-wrong proportion correlates at r=+0.25, and mean loss minus entropy at r=+0.30.Source: §5.1 · pp. 5–6
5.2 Loss and Entropy Signals in the Spectral Domain
- Phrase-band change aligns with lower ratings (r≈−0.39), beat-rate entropy with higher ratings (r≈+0.41), and loss–entropy coherence at 40–190 ms with lower ratings (r≈−0.54).Sub-beat loss variation also aligns with higher ratings (r≈+0.48); the authors interpret the coherence pattern as a glitch signature.Source: Figure 4 · p. 6, §5.2 · p. 6
5.3 SAE Latents as Interpretable Concepts
- Producer judgments of good- and bad-aligned SAE latents match their rating-aligned attribution ranking, while Gemini captions on the bad side retain clean genre labels.The producer instead describes distortion, including noise, harsh high frequencies, and EOS/delay tokens.Source: Table 3 · p. 6, §5.3 · p. 6
6. CONCLUSION
The conclusion argues that intrinsic signals from a music generator track listener ratings and offer a viable axis for automatic music evaluation.
6. CONCLUSION
- Loss, entropy, and SAE latents from one frozen MusicGen align with human ratings across five public benchmarks and outperform audio-only aesthetics baselines.The conclusion identifies loss–entropy mismatch and SAE latents as complementary quality-related signals.Source: §6 · p. 6
- Loss–entropy mismatch flags overconfident-wrong moments associated with disrupted perceived quality.The conclusion describes SAE latents as organizing clips along a quality axis without ratings.Source: §6 · p. 6
- The authors describe model-intrinsic signals as a viable, underused axis for automatic music evaluation.Source: §6 · p. 6