Source-linked AI summary
MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model
Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen, Yi-Hsuan Yang
TL;DR
Global embedding similarity gives text-to-music alignment a single score, obscuring which attributes are missing. MQAScore verifies atomic concepts across five dimensions with LALMs, and the experiments find concept-level scoring improves both evaluator paradigms while LALMs outperform similarity-based evaluators and approach human agreement on several dimensions.
Problem
Global text-audio similarity collapses prompt attributes into one score, obscuring fine-grained semantic alignment as text-to-music prompts become more detailed.
Source: §1 · p. 1Method
MQAScore asks an LALM a binary question about each atomic concept across five clip-level dimensions and aggregates the concept scores by dimension.
Source: §1 · p. 1, §3.2 · p. 3Results
Takeaways & Limitations
Per-dimension scoring provides a fine-grained complement to global similarity, including vocal distinctions that global similarity barely reflects.
Source: §1 · p. 2, §6 · p. 4The evaluation is limited by LLM-based concept extraction and negative selection, GTZAN’s single-label assumption, its clip-level descriptive concept focus, and a small expert set.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Text-to-music alignment is commonly evaluated by the global similarity between text and audio embeddings, which collapses all described attributes into a single score and obscures which ones are missing. We instead introduce MQAScore, which verifies each atomic musical concept along five clip-level dimensions (genre, instrument, mood/theme, vocal, and usage) by the probability that a large audio language model (LALM) answers affirmatively to a binary question. To assess evaluator reliability, we construct verification labels from presence/absence annotations and LLM-guided concept substitutions, and propose the Sensitivity-Invariance Ratio, which measures how selectively an evaluator responds to meaning-changing substitutions. Experiments show that concept-level scoring improves both question-answering and similarity-based evaluators, and MQAScore can be a more reliable and interpretable fine-grained complement to global similarity for evaluating text-to-music alignment.
1. INTRODUCTION
As text-to-music prompts become more detailed, global text-audio similarity still compresses all attributes into one score, obscuring fine-grained alignment. MQAScore instead evaluates atomic concepts across five clip-level dimensions, with verification labels and a selectivity measure used to assess evaluator reliability.
1. INTRODUCTION
- Global similarity collapses prompt and audio attributes into one score, making detailed text-to-music alignment difficult to interpret.The paper argues evaluation should focus on individual semantic concepts as prompts become more multi-attribute.Source: §1 · p. 1, Figure 1 · p. 1
- The taxonomy retains genre, instrument, mood/theme, vocal, and usage while excluding audio-fidelity and musicological concepts.Listener-facing affective tags are grouped as mood/theme; harmony and rhythm are excluded because LALMs remain poor at detecting them.Source: §1 · p. 1
- MQAScore asks an LALM whether each individual concept is supported by the clip and scores the probability of an affirmative answer.This adapts AQAScore’s affirmative-answer scoring mechanism from full prompts to atomic concepts.Source: §1 · p. 1
- Verification labels combine presence/absence annotations with LLM-selected distant substitutes and close alternatives to test sensitivity to contradictions versus wording changes.The Sensitivity-Invariance Ratio measures whether score changes track semantic contradictions rather than minor edits.Source: §1 · p. 2
- Concept-level decomposition improves both similarity- and question-answering-based evaluators, while LALM scoring further outperforms similarity-based scoring in discrimination and human agreement.Per-dimension results can isolate vocal-generation differences that global similarity barely reflects.Source: §1 · p. 2
2. RELATED WORK
Prior text-audio alignment evaluation often uses global embedding similarity, while other work evaluates finer-grained attributes and human judgment. Music question-answering research and stronger LALMs have also enabled their use as semantic evaluators.
2. RELATED WORK
- Text-audio and text-music alignment studies largely use global similarity, while prior work notes limits in capturing detailed musical attributes.Other evaluation approaches assess individual acoustic events, musical attributes, or compositional constraints and seek agreement with human judgments.Source: §2 · p. 2
- Music QA benchmarks probe LALM understanding and reasoning, and emerging evaluators repurpose LALMs for question answering, rubric-based verification, or concept-level judgments.Recent general and music-specialized LALMs demonstrate increasingly strong music capabilities.Source: §2 · p. 2
3. METHOD
The method builds a five-dimension concept taxonomy and evaluates each extracted concept separately with audio-grounded binary question answering. It aggregates concept scores by dimension to produce a vector of alignment scores.
3.1. Taxonomy Construction
- The taxonomy uses five dimensions and extracts clip-level concepts from captions, assigning each concept to a dimension and merging synonyms after manual review.Subjective modifiers are discarded when extracting atomic concepts.Source: §3.1 · p. 2
- For genre and instrument, negatives are concepts annotated absent from the clip; other dimensions use distant substitutions as negatives and close alternatives as controls.For mood/theme, vocal, and usage, controls help distinguish audio-grounded score changes from effects of editing the text.Source: §3.1 · p. 2
3.2. Concept Verification Scoring
- An LALM answers a dimension-specific binary question for each concept, and MQAScore is the affirmative-answer probability normalized against the No probability.The question template is formed from the concept and its dimension.Source: §3.2 · pp. 2–3
- Each dimension’s alignment score is the mean of its extracted concept scores, producing a per-dimension vector; empty dimensions are unavailable rather than zero.The aggregation uses the concepts assigned to that dimension.Source: §3.2 · p. 3
4. EXPERIMENTS
The experiments construct verification data for five dimensions, assess similarity-based and QA-based evaluators at caption and concept granularity, and measure discrimination, substitution selectivity, and expert agreement.
4.1. Verification Data
- Verification data pairs positive and absent genre/instrument tags, while mood/theme, vocal, and usage use LLM-selected distant and close concept substitutions.Expert pairwise labels come from 11 musically trained annotators; only unambiguous pairs are retained, with Krippendorff’s α = 0.71.Source: §4.1 · p. 3
4.2. Evaluators Assessed
- The evaluation compares two contrastive text-audio models with three LALMs, scoring either full captions or isolated concepts where the data permit.Caption-level scoring applies only to mood/theme, vocal, and usage; concept-level scoring evaluates each extracted concept in isolation.Source: §4.2 · p. 3
4.3. Reliability Metrics
- Reliability is measured with pooled ROC-AUC, within-clip win rate for genre and instrument, substitution-based RS/I, and agreement with expert pairwise labels.RS/I divides normalized score change for a contradiction by change for a close alternative; values much greater than one indicate selective response to contradiction.Source: §4.3 · p. 3
5. RESULTS
Concept-level scoring improves both question-answering and similarity-based evaluators over caption-level scoring. LALM-based evaluators show stronger concept verification on several dimensions and reach agreement with expert labels comparable to human-human agreement.
5. RESULTS
- Concept-level scoring improves both evaluation paradigms over caption-level scoring by preventing other caption concepts from diluting a concept’s contribution.Caption-level scores are unavailable for tag-only genre and instrument data.Source: §5 · p. 3, Table 1 · p. 4
- LALM-based evaluators reach expert-label agreement comparable to human-human agreement across dimensions, while similarity-based evaluators fall far behind, especially on mood/theme.The gap narrows on genre and instrument; all evaluators use concept-level scoring in this comparison.Source: Table 2 · p. 4, §5 · p. 4
- At the concept level, LALMs outperform similarity-based evaluators on mood/theme, vocal, and usage in both AUC and RS/I.Similarity-based evaluators remain competitive on genre and instrument, while usage is the most challenging dimension for all evaluators.Source: §5 · p. 4, Table 1 · p. 4
- Evaluators with similar AUC can differ markedly in RS/I, so discrimination accuracy alone does not capture selective response to meaning-changing substitutions.Caption-level AQAScore has high RS/I despite lower AUC, because replacing one concept barely changes the score under a close alternative.Source: §5 · p. 4
6. CONCLUSION
The paper presents MQAScore for verifying atomic musical concepts across five dimensions with LALMs. Its conclusion highlights improvements from concept-level scoring, stronger LALM performance on abstract dimensions, and several scope limitations.
6. CONCLUSION
- MQAScore verifies atomic musical concepts across five clip-level dimensions using LALMs.The dimensions are genre, instrument, mood/theme, vocal, and usage.Source: §6 · p. 4
- Concept-level scoring improves both question-answering- and similarity-based evaluators over caption-level scoring.Source: §6 · p. 4
- LALMs outperform similarity-based evaluators and approach human agreement on mood/theme, vocal, and usage; usage remains the most challenging dimension.The Sensitivity-Invariance Ratio also reveals differences in selectivity to meaning-changing substitutions.Source: §6 · p. 4
- The study is limited by LLM-based concept extraction and negative selection, GTZAN’s single-label assumption, clip-level descriptive concepts, and a small expert set.Future work will extend the taxonomy to relational concept graphs and musicological attributes and apply concept verification to generated music and reward modeling.Source: §6 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The research uses open-access MusicCaps, GTZAN, and OpenMIC-2018 under their licenses, and expert pairwise labels were collected as dataset annotations.
7. COMPLIANCE WITH ETHICAL STANDARDS
- The study uses open-access MusicCaps, GTZAN, and OpenMIC-2018 in accordance with their licenses.Source: §7 · p. 4
- Annotators participated voluntarily; no personally identifiable or sensitive information about them was collected or analyzed.Expert pairwise labels were collected as dataset annotations describing audio clips, not as a study of annotators.Source: §7 · pp. 4–5