Source-linked AI summary
Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Models
Aanya Maheshwari, Vatsal Raina
TL;DR
Multiple-choice accuracy does not show whether music audio-language models know when they are wrong, while independently trained ensembles are too expensive. This paper builds pseudo-ensembles from answer-preserving input changes to estimate uncertainty. On TinyMU and MuChoMusic, four option orderings raise accuracy and expected entropy improves error ranking over single-pass entropy.
Problem
Multiple-choice accuracy does not reveal whether a music audio-language model knows when it is wrong, and independently trained ensembles are too expensive for this setting.
Source: §1 · p. 1Method
The paper builds pseudo-ensembles by applying answer-preserving input changes to one pretrained model and averaging its option distributions, chiefly across shuffled answer orders.
Source: §4 · p. 4Results
Takeaways & Limitations
The procedure uses a few extra forward passes and no retraining, making abstention cheap enough to consider for compact music audio-language models.
Source: §1 · p. 2, §7 · p. 8The study uses one model and one benchmark; it has not shown that the same ranking holds for larger systems or other music-QA sets.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions. This protocol forces the model to commit to an option, so a lucky guess looks the same as real musical understanding. What is missing is a way to tell when the model does not know the answer, so that it can abstain instead of guessing. The usual solution, an ensemble of independently trained models, is far too expensive here, which leaves the entropy of a single predictive distribution as the only available confidence signal. We instead build pseudo-ensembles from one pretrained model by perturbing its input in ways that cannot change the correct answer, then averaging the resulting distributions over the options. Our main construction simply shuffles the order in which the candidate answers are presented; we also study ensembles built from corrupted audio and from swapped option labels. A pseudo-ensemble gives several predictive distributions per question, so it supports the full family of ensemble-based uncertainty measures (entropy of the expected distribution, expected entropy, and their difference, the mutual information) rather than entropy alone. Evaluating TinyMU on MuChoMusic, we find that averaging over four option orderings raises accuracy from 55.7% to 59.2%, and that the resulting uncertainty measures rank the model's errors better than the single-pass entropy baseline, reducing the area under the error retention curve from 0.293 to 0.261. All of this costs a few extra forward passes and no retraining, which makes abstention practical for compact music audio-language models.
1 Introduction
Music audio-language models are usually evaluated by multiple-choice accuracy, which cannot distinguish understanding from a lucky guess. The paper studies abstention using a single-model pseudo-ensemble built from answer-preserving input changes.
1 Introduction
- Multiple-choice accuracy reveals how often models are right, but not whether they know when they are wrong.A forced choice makes a lucky guess look like real musical understanding.Source: §1 · p. 1
- Abstention needs an uncertainty score, but training ensembles of independently trained music audio-language models is too expensive.Selective prediction ranks predictions by uncertainty so a system can abstain or defer when unsure.Source: §1 · p. 1
- The paper builds pseudo-ensembles from one pretrained model using answer-preserving input changes, chiefly shuffling candidate order, to support ensemble uncertainty measures.It also studies corrupted audio and swapped option labels, and evaluates TinyMU on MuChoMusic.Source: §1 · p. 2
- Shuffling candidate answers raises accuracy and ranks errors better than single-pass entropy, according to the paper's stated contributions.The approach requires a few extra forward passes and no retraining.Source: §1 · p. 2
2 Uncertainty quantification for classification
The section treats multiple-choice answering as classification and compares single-distribution scores with uncertainty measures derived from ensembles. The ensemble measures distinguish overall uncertainty from uncertainty associated with member disagreement.
2 Uncertainty quantification for classification
- A multiple-choice model assigns probabilities to candidate answers and predicts the most probable option; uncertainty scores quantify how spread or concentrated its belief is.The section uses five measures: negative confidence, entropy, entropy of expected, expected entropy, and mutual information.Source: §2 · pp. 2–3
- Negative confidence uses only the chosen option's probability, whereas entropy uses the full distribution; neither separates genuine ambiguity from model ignorance.With four options, entropy ranges from zero to about 1.39 nats.Source: §2 · pp. 2–3
- Entropy of expected is the entropy of the averaged member distributions, while expected entropy averages the members' entropies.Because entropy is concave, entropy of expected is never smaller than expected entropy.Source: §2 · p. 3
- Their difference, mutual information, captures member disagreement; expected entropy reflects uncertainty that remains within members, such as ambiguity in the question or audio.A confident but disagreeing ensemble can have low expected entropy and high mutual information, unlike a single distribution.Source: §2 · p. 3
3 Multiple-choice musical question answering
The benchmark presents musical question answering as a four-option classification task. The method reads the answer from the first generated token and uses the resulting option distribution.
3 Multiple-choice musical question answering
- Each question pairs a short recording and a question with four candidate answers labeled A–D, exactly one of which is correct.The benchmark prompt ends with a cue for the model to emit an option label.Source: §3 · p. 3
- The method reads the answer from the first generated token, restricting its logits to the four option-label tokens and predicting their argmax.This takes one forward pass rather than scoring each full option text in a separate pass.Source: §3 · p. 3
- Softmax over the four labels supplies a categorical option distribution, and renormalizing discards probability mass outside the closed-set answer choices.The passage reports that these predictions agree with those obtained by parsing generated text.Source: §3 · p. 3
4 Abstention in multiple-choice question answering
The paper defines abstention through an uncertainty threshold and constructs pseudo-ensembles by applying answer-preserving changes to one model's input. Option shuffling is the main construction; corrupted audio and label swapping are alternatives.
4 Abstention in multiple-choice question answering
- A system answers when its uncertainty is below threshold τ and abstains otherwise; raising τ increases coverage at the cost of reliability.Evaluation measures how well uncertainty ranks questions by their chance of being wrong.Source: §4 · p. 4
- A pseudo-ensemble applies answer-preserving input transformations to one pretrained model, treating the resulting distributions as ensemble members without training additional models.Its cost is M forward passes and no training.Source: §4 · p. 4
- Option shuffling permutes answer order under labels A–D; distributions must be mapped back to the original candidates before averaging.Alignment lets averaging cancel position bias, which the passage gives as the reason the ensemble is more accurate than the default ordering.Source: §4 · p. 4
- Audio corruption changes the recording with mild noise, compression, or time shifts while holding the default option order fixed.A silent-audio diagnostic is not answer-preserving and measures how much of the benchmark can be answered from text alone.Source: §4 · p. 4
- Label swapping changes option symbols while preserving audio and answer order, but TinyMU remains biased toward A/B/C/D and the resulting ensemble is poorly calibrated.The method restricts first-token logits to whichever four label tokens the prompt uses.Source: §4 · p. 4
- The paper focuses on option shuffling because it needs no severity or scheme choice and gives the largest accuracy gain and strongest abstention signal among the three constructions.The passage points to Section 6 for those comparisons.Source: §4 · p. 4
5 Experiments
The experiments evaluate TinyMU on MuChoMusic and compare accuracy with uncertainty-based abstention rankings. They use four-member pseudo-ensembles without further training.
5.1 Data
- MuChoMusic contains 1,187 four-option questions across 644 tracks, covering 23 musical dimensions and divided into knowledge and reasoning subsets.Each question has one correct answer; some may be answerable from text alone, which the silent-audio diagnostic measures.Source: §5.1 · p. 5
5.2 Model
- TinyMU’s released 229M-parameter checkpoint is evaluated without further training, using four members per pseudo-ensemble and deterministic first-token predictions.Inputs are mono 32 kHz audio truncated to 10 seconds; ensemble variants use option orderings, audio perturbations, or label schemes.Source: §5.2 · p. 5
5.3 Evaluation metrics
- Accuracy measures constrained-argmax correctness, while error retention curves assess how well uncertainty ranks errors for deferral; lower AUC-ERC is better.Questions are removed from most uncertain to most confident, and the curve tracks error among retained predictions.Source: §5.3 · p. 5
6 Results
Option shuffling improves TinyMU’s MuChoMusic accuracy and supports better error ranking than the single-pass baseline. Expected entropy ranks errors best among the tested measures, while the benefit of pseudo-ensembles depends on the perturbation and ensemble size.
6 Results
- Single-pass accuracy is 55.69%, below the published 58.6%; silent audio scores 50.7%, indicating some questions can be answered from text alone.The comparison also differs in readout and audio duration: the evaluation uses first-token scores and truncates audio to 10 seconds.Source: §6 · p. 5
- 59.22% accuracy with M = 4 option shuffling is a 3.5-point gain over single-pass TinyMU and slightly exceeds its published figure.The comparison uses the constrained first-token readout; the published figure uses free-form generations.Source: §6 · p. 5, Table 1 · p. 6
- AUC-ERC falls from 0.293 to 0.261 with expected entropy, the strongest measure; mutual information scores 0.306, worse than single-pass entropy.The M = 4 ensemble uses option shuffling; lower AUC-ERC indicates better error ranking.Source: §6 · pp. 5–6, Table 2 · p. 6
- Across M = 4 constructions, option shuffling performs best on accuracy and abstention, audio corruption still beats baseline, and label swapping performs worse.Mild audio corruption produces less diversity than option shuffling, while TinyMU’s continued preference for A/B/C/D under swapped labels leaves those members poorly calibrated.Source: §6 · pp. 6–7, Table 3 · p. 7
- No uncertainty measure approaches the oracle, so even the best ranking leaves confidently wrong answers and is a filter, not a calibrated correctness probability.Figure 3 presents error retention curves for the M = 4 pseudo-ensembles against oracle and random references.Source: §6 · pp. 6–7, Figure 3 · p. 6
- Accuracy rises with ensemble size, with most gains in the first few members; the peak mean exceeds 60.3%, while compute grows linearly.M = 4 is a low-cost operating point, not saturation; Figure 4 uses five draws for M < 24 and one complete ensemble at M = 24.Source: §6 · p. 7, Figure 4 · p. 7
7 Conclusion
Answer-preserving input changes let a single checkpoint support abstention, while option shuffling improves both accuracy and error ranking without retraining.
7 Conclusion
- Four option orderings raise accuracy from 55.7% to 59.2%, and twenty-one orderings raise it to 60.3%.The conclusion identifies shuffling as the most effective of the three constructions tried.Source: §7 · p. 8
- Option shuffling unlocks ensemble uncertainty measures that rank errors better than single-pass entropy, with expected entropy performing best.Source: §7 · p. 8
- The procedure requires a few extra forward passes and no retraining, making abstention practical for a compact on-device model.Those extra passes also raise accuracy, rather than serving abstention alone.Source: §7 · p. 8
8 Limitations
The evidence is limited to one model and one benchmark, and the method depends on multiple-choice prompts and assumptions about labels and option order.
8 Limitations
- The study tests TinyMU on one benchmark, so its uncertainty ranking is not established for larger models or other music QA sets.Source: §8 · p. 8
- First-token scoring assumes the model binds answers to option symbols, while option shuffling may lose value if models become order-invariant.Option shuffling applies only to multiple-choice prompts.Source: §8 · p. 8
- Audio-corruption severity was not extensively tuned, and label swapping failed on TinyMU, a result that may not transfer.Source: §8 · p. 8
- The authors have not audited TinyMU for confident failures across genres, cultures, or recording conditions.Such failures could be hidden by a reliable abstention rule.Source: §8 · p. 8
A Implementation details
Implementation uses the released TinyMU checkpoint and inference-only evaluation, with defined preprocessing and perturbation schemes for the pseudo-ensembles.
A Implementation details
- TinyMU couples the frozen MATPAC++ audio encoder to SmolLM2-135M through a two-layer projector; the released checkpoint is used without further training.The projector and language model were fine-tuned on MusicSkills-3.5M.Source: A Implementation details · p. 11
- Audio is mono at 32 kHz and truncated to 10 seconds; shuffled-member distributions are aligned to the original answers before averaging.Corruption members comprise clean audio, Gaussian noise with σ = 0.01, lossy compression, and a small time shift.Source: A Implementation details · p. 11
- Evaluation is inference-only on public research releases, and the main M = 4 setting requires four forward passes per question.The M = 21 and ensemble-size plots require passes in proportion to M.Source: A Implementation details · p. 11