Audio LLMs Know When They Can't Hear You

1 INTRODUCTION

• Degraded audio can lead an Audio LLM to infer a plausible but incorrect query and confidently answer it.The risk arises because speech interaction requires the model to infer linguistic content from acoustics before responding.Source: §1 · p. 1
• Audio LLM self-assessments often predict reliable transcription even when actual WER is substantial, and two-shot examples only partly reduce this overconfidence.The self-assessment experiment progressively corrupts speech and compares realized transcription error with the model's zero-shot or two-shot reliability judgment.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Reliability labels can transfer across Audio LLM families, with transfer degradation closely related to alignment between model-specific reliability boundaries.When large transfer gaps are associated with systematic boundary differences, correcting boundary-specific shifts can substantially reduce the gap.Source: §1 · p. 2
• 81.10% in-domain and 78.09% cross-domain macro-F1, versus strongest baselines' 70.77% and 66.16%, respectively, on Qwen2-Audio-7B-Instruct.The evaluations hold out samples from the training distribution in-domain and use different train and test distributions cross-domain.Source: §1 · p. 2
• Frozen audio-encoder representations support more accurate reliability prediction than speech-quality predictors, generation uncertainty, and transcript-conditioned WER estimation.The predictor operates on a frozen encoder and assigns recordings to reliable, minor, moderate, or severe degradation classes; unreliable predictions can prompt clarification or repetition without changing the Audio LLM.Source: §1 · p. 2
• Critical SNR varies widely among speech–corruption pairs within each corruption category, so a fixed SNR threshold cannot reliably characterize transcription reliability.The distributions cover background noise, noise with reverberation, and music for Qwen2-Audio-7B-Instruct at a WER threshold of 10%.Source: Figure 2 · p. 3

2 RELATED WORK

• No-reference speech-quality and intelligibility measures provide proxies for transcription reliability, but recordings with similar estimated quality can yield different transcription outcomes depending on the utterance and Audio LLM.The discussed methods include DNSMOS, NISQA, TorchAudio-SQUIM, and Audiobox-Aesthetics.Source: §2 · p. 3
• Generation uncertainty and transcript-conditioned WER estimation provide model-dependent signals but require language generation; this work instead predicts reliability from audio-encoder representations without a transcript.Fe-WER combines acoustic input representations with textual representations of the generated transcript to estimate utterance-level WER.Source: §2 · p. 3
• Prior work studies verbalized uncertainty and self-evaluation, while this paper tests whether an Audio LLM recognizes when degraded speech will produce an unreliable transcription.Related multimodal work has also found calibration and self-assessment challenging.Source: §2 · p. 3

3.1 AUDIO LLM RELIABILITY DATASET CONSTRUCTION

• Critical SNR varies widely even within corruption categories, so a single SNR cutoff cannot reliably determine transcription reliability.The authors estimate boundaries separately for each speech–corruption pair.Source: §3.1 · p. 4
• Reliability labels use four WER-based classes and pair-specific critical SNRs, the lowest SNR at which the model’s WER meets each threshold.The class boundaries are WER = 0, 0 < WER ≤ 10%, 10% < WER ≤ 30%, and WER > 30%.Source: §3.1 · pp. 4–5
• The dataset filters pairs with unsuitable transcription behavior or unreliable boundaries, then samples training SNRs within retained class intervals while keeping validation and test waveforms fixed.The speech, corruption source, and room impulse response, when applicable, remain fixed while SNR varies.Source: §3.1 · p. 5

3.2 AUDIO-ENCODER RELIABILITY PREDICTION

• The predictor freezes the target Audio LLM’s encoder, learns to pool frame representations into an utterance embedding, and classifies it into four reliability classes.The four classes are reliable, minor degradation, moderate degradation, and severe degradation.Source: §3.2 · pp. 5–6, Figure 4 · p. 5
• For Qwen2-Audio-7B-Instruct, the predictor has 0.33M trainable parameters (0.052% of 636.97M frozen audio-encoder parameters) and adds approximately 2.12M FLOPs (0.00011% of 1.89T encoder FLOPs).The additional cost is limited to the pooling module and classification head because the audio encoder already runs during inference.Source: §3.2 · p. 6

4.1 TRANSCRIPTION RELIABILITY PREDICTION

• 81.10% macro-F1, 82.60% accuracy, and 0.18 MAE in-domain; Fe-WER reaches 70.77% macro-F1, 75.05% accuracy, and 0.27 MAE.The predictor improves macro-F1 by 10.33 points and accuracy by 7.55 points while reducing MAE by 0.09.Source: Table 1 · p. 7, §4.1 · p. 7
• 78.09% macro-F1 and 79.55% accuracy under cross-domain evaluation; gains over the strongest baseline for each metric are 11.93 and 9.86 points, respectively.The predictor's cross-domain MAE is 0.22; Qwen mean token probability has the highest baseline macro-F1 at 66.16%, while Fe-WER has the highest baseline accuracy at 69.69% and lowest MAE at 0.33.Source: §4.1 · p. 7
• Cross-domain evaluation reduces the predictor's accuracy by 3.05 points and macro-F1 by 3.01 points, versus Fe-WER drops of 5.36 and 6.10 points.The evaluation uses unseen speech and acoustic sources for the cross-domain test set.Source: §4 · p. 6, §4.1 · p. 7
• Nominal SNR reaches only 65.14% in-domain and 60.85% cross-domain accuracy; generation uncertainty is stronger than quality predictors but remains below the encoder predictor.For Qwen2-Audio-7B-Instruct, the prediction head requires 2.12M FLOPs versus 4.21T FLOPs for the decoder.Source: §4.1 · p. 7

4.2 CROSS-MODEL TRANSFER OF RELIABILITY LABEL BANKS

• Critical-SNR mismatch correlates positively with transfer degradation (r = 0.78), and estimates from a small fraction of shared pairs approach the full-label-bank value.The mismatch estimate's variance decreases as more shared speech–corruption pairs are used.Source: Figure 5 · p. 8, §4.2 · pp. 8–9
• Using 5% of shared pairs to estimate boundary-specific shifts reduces the MOSS←Phi-4 accuracy drop from 12.9 to 2.1 points and MOSS←Qwen2 from 9.7 to 1.7.Transfer between Qwen2 and Phi-4 largely preserves accuracy, while transfers between MOSS and either model show larger degradation.Source: Figure 5 · p. 8, §4.2 · pp. 8–9

5 CONCLUSION

• Audio LLM self-assessment is poorly calibrated, while transcription reliability is strongly encoded in audio-encoder representations that support a predictor outperforming the evaluated baselines.The predictor uses model-grounded reliability labels and maintains strong performance under cross-domain evaluation.Source: §5 · p. 9
• Reliability label banks can be reused across Audio LLM families, but transfer performance depends on the source–target pair and boundary alignment.Boundary-specific shifts can be estimated to reduce large transfer gaps.Source: §5 · p. 9

A APPENDIX OVERVIEW

• The appendix covers label construction, additional model results, confusion matrices, computational overhead, and analyses of temporal aggregation and encoder probe depth.It also provides further detail on zero-shot and two-shot self-assessment settings.Source: A APPENDIX OVERVIEW · p. 12

B AUDIO LLM SELF-ASSESSMENT DETAILS

• Self-assessment compares a model's reliability judgment with its measured WER on noisy held-out recordings, without giving the reference transcript to the model.The experiments use zero-shot prompting and two-shot in-context learning; the latter combines reliable and unreliable demonstrations and aggregates five responses by majority vote.Source: B AUDIO LLM SELF-ASSESSMENT DETAILS · p. 12

C DATASET CONSTRUCTION DETAILS

• Reliability labels follow each target Audio LLM’s transcription performance, so the same SNR can yield different errors across utterances, corruption sources, and models.Critical SNRs are estimated independently for each speech–corruption pair.Source: C DATASET CONSTRUCTION DETAILS · p. 13
• In-domain data use LibriSpeech splits, while cross-domain testing uses LJSpeech, SONYC-UST, and unseen real room impulse responses.Clean utterances with WER ≥5% are removed before constructing corrupted examples.Source: C DATASET CONSTRUCTION DETAILS · p. 13
• Examples use additive interference, optionally after reverberation, with SNR varied for each fixed speech–corruption pair.DNS and MUSAN corruption recordings and OpenSLR26 room responses are partitioned across splits; cross-domain sources are unseen during training.Source: C DATASET CONSTRUCTION DETAILS · p. 13
• The four classes are reliable (WER = 0), minor degradation (0 < WER ≤10%), moderate degradation (10% < WER ≤30%), and severe degradation (WER > 30%).Source: C DATASET CONSTRUCTION DETAILS · p. 14
• Quality control removes pairs with poor clean or reverberant transcription, unstable or misordered SNR boundaries, and excludes a 5% margin beside shared class boundaries.The label bank stores speech, corruption, applicable room responses, and model-specific class intervals rather than pre-generated noisy waveforms.Source: C DATASET CONSTRUCTION DETAILS · p. 14
• Training samples a class and resamples its SNR across epochs, while validation and test waveforms are fixed across methods.For reliable examples, 40% are clean speech and 60% are corrupted speech with zero WER.Source: C DATASET CONSTRUCTION DETAILS · p. 14

D ADDITIONAL EVALUATION RESULTS

• 87.01% macro-F1, 88.02% accuracy, and 0.12 MAE for Phi-4 in-domain beat mean token entropy’s 81.93% macro-F1, 83.44% accuracy, and 0.17 MAE.The reported gains are 5.08 macro-F1 points and 4.58 accuracy points, with MAE reduced by 0.05.Source: D ADDITIONAL EVALUATION RESULTS · p. 14, Table 3 · p. 15
• 83.54% macro-F1, 84.35% accuracy, and 0.16 MAE for Phi-4 cross-domain exceed the strongest baseline for each metric by 3.79 points, 3.60 points, and 0.04 MAE, respectively.Phi-4 mean token probability has the highest baseline macro-F1 at 79.75%, while mean token entropy has the highest accuracy at 80.75%; both have 0.20 MAE.Source: D ADDITIONAL EVALUATION RESULTS · p. 14, Table 3 · p. 15
• 82.97% macro-F1, 84.44% accuracy, and 0.16 MAE for MOSS in-domain beat mean token entropy’s 74.83% macro-F1, 77.95% accuracy, and 0.24 MAE.The corresponding gains are 8.14 macro-F1 points and 6.49 accuracy points, with MAE reduced by 0.08.Source: Table 4 · p. 15, D ADDITIONAL EVALUATION RESULTS · p. 15
• 81.60% macro-F1, 82.86% accuracy, and 0.17 MAE for MOSS cross-domain exceed mean token entropy’s 76.77% macro-F1, 78.78% accuracy, and 0.22 MAE.The predictor’s macro-F1 and accuracy drops from in-domain are 1.37 and 1.58 points; its gains over the baseline are 4.83 and 4.08 points, with MAE reduced by 0.05.Source: Table 4 · p. 15, D ADDITIONAL EVALUATION RESULTS · p. 15

E CONFUSION MATRIX ANALYSIS

• 99.5%, 98.9%, and 98.6% reliable-class recall for Qwen2-Audio, Phi-4-Multimodal, and MOSS-Audio, respectively, limits unnecessary clarification requests for error-free audio.Source: E CONFUSION MATRIX ANALYSIS · p. 16
• 82.6%, 88.0%, and 84.4% accuracy for Qwen2-Audio, Phi-4-Multimodal, and MOSS-Audio, respectively.Source: E CONFUSION MATRIX ANALYSIS · p. 16
• Most incorrect predictions fall between neighboring reliability classes, particularly minor and moderate degradation.Source: E CONFUSION MATRIX ANALYSIS · p. 16

F COMPUTATIONAL OVERHEAD ANALYSIS

• The predictor adds a lightweight head to the already-computed frozen encoder outputs and makes its reliability decision without executing the decoder.Source: F COMPUTATIONAL OVERHEAD ANALYSIS · p. 16
• Generation uncertainty and transcript-conditioned WER estimation require decoder execution before reliability can be estimated.That decoding cost is substantial relative to the prediction head, especially when reliability determines whether generation should proceed.Source: F COMPUTATIONAL OVERHEAD ANALYSIS · p. 16
• For Qwen2-Audio-7B-Instruct, the predictor has 0.33M trainable parameters and requires 2.12M FLOPs, versus 636.97M parameters and 1.89T FLOPs for the encoder and 7.76B parameters and 4.21T FLOPs for the decoder.Source: F COMPUTATIONAL OVERHEAD ANALYSIS · p. 17

• The ablation studies compare methods for aggregating frozen audio-encoder representations into utterance embeddings.Source: G ABLATION STUDY · p. 17
• A second study probes encoder representations at different depths across three Audio LLMs to locate where transcription-reliability information emerges.Source: G ABLATION STUDY · p. 17
• A third study ablates strategies for estimating shifts between source and target critical-SNR boundaries.Source: G ABLATION STUDY · p. 17

• 81.10% macro-F1 and 82.60% accuracy, with 0.18 MAE, are achieved by learned temporal pooling.Mean and max pooling perform worse in this comparison.Source: §G.1 · p. 17
• The comparison holds the frozen Qwen2-Audio-7B-Instruct representations and training data fixed while optimizing only aggregation and classification modules.Source: §G.1 · p. 17
• Learned temporal pooling assigns weights across frames, whereas mean and max pooling may discard reliability-relevant information through uniform aggregation or strongest-activation selection.Source: §G.1 · p. 17

G.2 ENCODER PROBE DEPTH.

• The encoder-depth study freezes each layer's audio representation and trains the same lightweight reliability predictor.Source: §G.2 · p. 17
• Decoder FLOPs are measured for prefill only, excluding autoregressive generation; including generation would modestly increase decoder cost without changing the orders-of-magnitude cost gap.The comparison uses an approximately 12-second input and a single forward pass.Source: §G.2 · p. 17
• Deeper encoder representations generally improve macro-F1 and accuracy while reducing MAE across the three evaluated Audio LLMs.The same lightweight predictor probes frozen representations from increasing encoder depths.Source: Figure 7 · p. 18, §G.2 · p. 18

G.3 CRITICAL-SNR ALIGNMENT STRATEGY.

• Per-boundary median alignment yields the lowest average accuracy drop across the two transfer settings; all tested alignment strategies substantially outperform direct transfer.Separate boundary shifts can invert adjacent critical-SNR boundary ordering, but such violations were extremely rare and excluded.Source: §G.3 · p. 18, Table 7 · p. 18