PaperlayerView exact revision on arXiv

Source-linked AI summary

MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

Kun Liu, Lyra Tao

arXiv:2610.22989v1cs.SD

TL;DR

Evaluating musical perception in multimodal language models requires separating audio evidence from answers available through priors, memorized recording associations, or external retrieval. MUB assembles identity-masked recordings with musical annotations and saved baseline predictions; those baselines show the influence of answer distributions, while model comparisons and validation remain incomplete.

  • Problem

    Musical-perception evaluation must distinguish direct audio use from answer-distribution priors, memorized recording associations, and external retrieval.

    Source: §1 · p. 1

  • Method

    MUB uses an existing annotation archive to assemble identity-masked recordings with tempo, tonal, metre, and structural-interval references.

    Source: §3.1 · p. 3, §3.4 · p. 4

  • Results

    The DSP tempo estimator exceeds the constant prior, while the constant metre predictor’s 98.4% accuracy is dominated by the class distribution.

    Source: §6.3 · p. 7

  • Takeaways & Limitations

    MUB’s present value is its specific resource, input-masking protocol, and transparent implementation audit, not evidence of contemporary model capabilities.

    Source: §8 · p. 8

    No MLLM runs or implemented segmentation baselines are available, and the key metric and coverage handling need versioned fixes before new model comparisons.

    Source: §7 · p. 8

Abstract

from arXiv · show

Evaluating musical perception in multimodal language models requires separating audio evidence from answers recoverable through metadata or external tools. We present the Music Understanding Benchmark (MUB), a locally assembled collection of 500 identity-masked recordings with existing annotations for tempo, key, time signature, and structural intervals. The collection covers all 24 major/minor key combinations, tempos of 47--180 beats per minute, and 4,676 annotated intervals. Original titles, artists, and platform identifiers are excluded from evaluation inputs. Relative to recent music question-answering and temporal-grounding work, MUB emphasizes scalar tempo prediction and full-recording interval outputs, together with a documented metadata-masking protocol. Saved baseline predictions yield 53.6\% tempo accuracy for a simple DSP estimator and 98.4\% time-signature accuracy for a constant 4/4 predictor. We identify important limitations, including an asymmetric legacy key metric, unverified independent annotation agreement, extreme metre imbalance, and unresolved recording-level rights clearance. No contemporary multimodal language model comparison has yet been completed. MUB is therefore an evaluation resource and protocol under validation, rather than a completed demonstration of model capabilities.

1 Introduction

MUB addresses the challenge of evaluating musical understanding from audio without relying on recoverable metadata or external tools. It packages four musical outputs for 500 identity-masked recordings with an inspectable evaluation pipeline and documented limitations.

1 Introduction

  • MUB pairs four outputs—tempo, tonic and mode, time signature, and structural intervals—with 500 recordings spanning all 24 major/minor keys and 4,676 intervals.The intervals and labels come from an existing annotation corpus.Source: §1 · p. 1
  • Its masking protocol removes original titles, artists, platform song identifiers, task identifiers, and annotator identities from model-facing inputs.Source: §1 · p. 2
  • The inspectable pipeline supplies waveform-based and answer-distribution reference predictions, but the study neither ranks multimodal models nor establishes a general measure of musical understanding.Identity masking reduces an explicit title-based shortcut but cannot establish that recordings were absent from training; baseline agreement does not prove label validity.Source: §1 · p. 2

2 Related Work

Prior work already covers human-curated music question answering, audio-language models, and global and temporal music tasks. MUB’s distinction is its specific combination of identity-masked recordings, scalar global predictions, and full-recording interval outputs from a reused annotation corpus.

2.1 Music Question Answering and Human Curation

  • MuChoMusic [3] and HumMusQA [4] provide human-validated music questions, while MUB derives global labels and full-recording intervals from an existing workflow whose annotator agreement remains unmeasured.Source: §2.1 · p. 2

2.2 Recent Audio-Language Models and Representation Access

  • Audio Flamingo 3 [1] and Music Flamingo [2] motivate task-level evaluation, while Fang et al. [7] use symbolic representations for annotation judging rather than direct waveform inference.Neither Audio Flamingo 3 nor Music Flamingo has been evaluated on MUB; native-audio evaluation must disclose and constrain intermediate representations.Source: §2.2 · p. 2

2.3 Global and Temporal Music Understanding

  • GaMMA [5] overlaps with MUB’s task areas, while MusTBench [6] tests temporal-grounding questions that differ from exhaustive formal-section segmentation.MUB’s specific focus is identity-masked recordings, scalar global predictions, and full-recording interval outputs; the cited comparisons establish context, not novelty or label validity.Source: §2.3 · pp. 2–3

3 Dataset Construction

MUB reuses an existing annotation archive to assemble a 500-recording collection, then documents how identities are masked and how recordings are selected and matched. The resulting collection spans all 24 major/minor keys, while its source labels and rights status remain subject to important limits.

3.1 Source and Label Provenance

  • The archive supplies timestamped structural intervals and per-interval attributes, but independent annotator agreement and the selector’s quality-review status are unverified.Global labels derive from the last available submission; submission and operator counts do not establish independent verification.Source: §3.1 · p. 3

3.2 Eligibility and Deduplication

  • Title–artist deduplication removes repeated metadata pairs but does not guarantee unique compositions or audio, and no recording-rights field establishes a low-risk subset.The selected 500 recordings therefore cannot be described as the lowest-copyright-risk subset.Source: §3.2 · p. 3

3.3 Sampling

  • Sampling combines tempo bins with key/mode and uses a seeded, smaller-strata-first allocation, but is not proportional and does not stratify by metre, duration, genre, or structural complexity.The selector uses random seed 42 and a final random fill.Source: §3.3 · p. 4

3.4 Identity Masking and Audio Matching

  • The audio_hash is a partial-file fingerprint: SHA-256 of the first 1,048,576 compressed-source bytes, truncated to 16 hexadecimal characters (64 bits); catalogue matching could identify the recording, so it must remain evaluator-side.Audio is not distributed; an accessible recording with a different encoding is insufficient for byte-prefix matching, and reproducible external evaluation needs a legitimate audio-access route.Source: §3.4 · p. 4

3.5 Collection Statistics

  • The collection includes all 24 major/minor keys, but metre is highly imbalanced: 492 recordings are 4/4, four are 3/4, and four are 6/8.The most frequent lowercased structural labels are chorus (1,231), verse (1,114), intro (460), and outro (427).Source: §3.5 · p. 4

4 Tasks and Inference Protocol

MUB defines four prediction tasks using global labels or full-recording intervals from the provided audio. Its inference protocol prohibits external assistance, but compliance and fair comparison require deployment controls and consistent audio coverage.

4 Tasks and Inference Protocol

  • Tempo is a scalar BPM prediction against the upstream modal segment tempo, while key prediction returns tonic pitch class and major/minor mode parsed from modal segment tonalities.A single tempo can omit half/double ambiguity or tempo changes; the key representation excludes modulation, ambiguous centres, and other modes.Source: §4 · pp. 4–5
  • Time signature is one modal segment label, so changing metre is reduced to one answer and aggregate accuracy is limited by the dominance of 4/4.References include fractions such as 4/4, 3/4, or 6/8.Source: §4 · p. 5
  • Structural segmentation returns interval start and end times with short labels, but the current scorer measures overlap only, not label correctness or semantic formal analysis.Full-track output requires localization across the recording, yet this metric alone does not demonstrate semantic formal analysis.Source: §4 · p. 5

4.1 External Assistance and Model Inputs

  • The intended native-audio track prohibits web access, downloads, code execution, external DSP, and callable tools, but no enforcing harness is supplied and prompts cannot prevent memorized recording information.A later tool-assisted track, if added, must be reported separately.Source: §4.1 · p. 5
  • Fair comparison requires recording model and inference settings, tool configuration, verified input fields, and equal audio coverage; full-recording tasks are not comparable if one model receives an undisclosed excerpt.The listed controls include decoding and resampling, context limits, truncation or chunking, parsing, retries, and refusal rates.Source: §4.1 · p. 5

5 Metrics and Implementation Audit

The section specifies how tempo, key, metre, and interval predictions are scored, while documenting important asymmetries and coverage defects in the current implementation.

5.1 Tempo and Time Signature

  • Tempo hits use a 4% relative-error convention, while time signature is matched after numerator and denominator extraction and normalization.The 4% threshold is a benchmark convention, not a claim of perceptual indistinguishability.Source: §5.1 · p. 6

5.2 Legacy Key Metric

  • Legacy key scoring is directional and erroneous; symmetric relative-key rescoring gives 29.4% DSP and 11.0% constant-prior scores.These are diagnostic recalculations, not outputs of a repaired released scorer.Source: §5.2 · p. 6

5.3 Interval Segmentation

  • Interval matching greedily pairs highest-IoU unmatched predictions at IoU at least 0.5, but ignores labels and lacks comprehensive interval validation.The scorer reports per-recording precision, recall, and F1; matching may depend on interval order.Source: §5.3 · p. 6

5.4 Prediction Coverage

  • Prediction coverage allows omitted recordings to leave denominators and repeated identifiers to count repeatedly; Table 4 includes no MLLM results.The two stored baseline files have complete 500-recording coverage, so this defect does not alter their reported aggregates.Source: §5.4 · p. 6, Table 4 · p. 7

6 Reference Baselines and Local Results

The section describes a waveform-based DSP reference and a test-distribution prior, then audits saved predictions under the current scoring rules. The results illustrate how prior dominance and scoring limitations qualify aggregate performance.

6.1 Waveform DSP Reference

  • The DSP reference estimates tempo by spectral-flux autocorrelation, key by pitch-class profiles following Krumhansl, and metre through beat-multiple periodicities.Its metre heuristic always uses denominator four and cannot output 6/8.Source: §6.1 · p. 7
  • The DSP baseline returns null for every structural output, so its zero F1 measures abstention rather than DSP segmentation limits.These heuristics are reference systems, not state-of-the-art baselines, theoretical lower bounds, or performance ceilings.Source: §6.1 · p. 7

6.2 Answer-Distribution Reference

  • The constant predictor uses 101 BPM, C major, and 4/4 from test references, making it a descriptive test-distribution prior rather than an out-of-sample estimate.It predicts no intervals.Source: §6.2 · p. 7

6.3 Results and Evidence

  • An arithmetic audit reproduced 268 DSP tempo hits, 127 legacy key hits, and 183 metre hits from saved files, but did not authenticate execution history.Audio DSP was not rerun for this manuscript revision.Source: §6.3 · p. 7
  • 98.4% for the constant metre predictor reflects class-distribution dominance; DSP tempo exceeds the constant prior without establishing label correctness.A model using informative audio can still underperform a strong prior on this imbalanced task.Source: §6.3 · p. 7
  • The result artifacts are local and untracked, so a public reproducible release must package prediction files, software versions, commands, and data and metric versions.The current local repository is not an already published benchmark release.Source: §6.3 · p. 7

7 Limitations, Rights, and Release Conditions

The paper identifies unresolved rights, annotation reliability, coverage, leakage, and empirical-validation issues that constrain MUB's release and interpretation.

Rights

  • Recording-level rights were not verified, and withholding audio does not establish rights to use recordings or redistribute annotations.A public release needs documented rights for its actual contents and an access policy for recordings.Source: §7 · p. 8

Annotation Reliability

  • Annotation reliability remains unmeasured; workflow provenance and submission counts do not establish independent agreement or gold-quality labels.The paper calls for stratified double annotation with adjudication before stronger gold-quality claims.Source: §7 · p. 8

Coverage and Imbalance

  • Only eight recordings have a non-4/4 reference, while tonality is restricted to major and minor and potentially changing properties are globalized.Balanced accuracy or macro-averaged scores should supplement overall metre accuracy; genre diversity is unquantified.Source: §7 · p. 8

Leakage and Access

  • Removing explicit metadata does not prevent familiar-audio recognition or training contamination, and a byte-prefix fingerprint may permit linkage.Model-visible hashes, audio tags, and revealing filenames must be excluded.Source: §7 · p. 8

Incomplete Empirical Validation

  • This version has no MLLM runs, confidence intervals, audio-ablation controls, or implemented segmentation baselines, and key scoring and coverage need versioned fixes.The proposed controlled comparisons and validation steps are requirements for future work, not results obtained here.Source: §7 · p. 8

8 Conclusion

The conclusion positions MUB as a resource and protocol whose contribution is its specific packaging and implementation audit, not a demonstration of model capabilities.

Conclusion

  • MUB packages 500 identity-masked recordings with tempo, major/minor key, time-signature, and structural-interval references from an existing annotation workflow.Its specific focus is scalar global predictions and full-recording interval outputs alongside input masking and implementation audit.Source: §8 · p. 8
  • Saved baseline predictions expose the importance of answer priors and metric definitions but do not establish contemporary model capabilities.Source: §8 · p. 8
  • Rights documentation, annotation validation, corrected versioned scoring, and controlled MLLM experiments remain necessary before MUB is presented as a validated public benchmark.Source: §8 · p. 8
Loading 2610.22989v1…