Source-linked AI summary
SURE-Challenge: Evaluating Speech Evidence Before Speech-LLM Generation
Mengzhe Geng
TL;DR
SURE-Challenge addresses whether speech evidence should be admitted to a speech LLM before answer scoring, a decision usually omitted by answer-only evaluation. It constructs source-disjoint supported and unsupported audio tests, compares front ends, and replays a frozen rule before six models. On the screened SURE-Extended test, the rule raises Qwen2-Audio unsupported rejection from 15/204 to 196/204 while leaving supported accuracy unchanged, with external checks exposing threshold and source-ambiguity boundaries.
Problem
Answer-only evaluation does not measure whether a waveform should be sent to a speech LLM before generation, despite unsupported clips remaining unsafe when answered fluently.
Method
The benchmark uses source-disjoint supported and unsupported audio, compares acoustic and ASR-confidence front ends, and replays one frozen admission rule before six speech/audio LLMs.
Results
15/204 to 196/204: Qwen2-Audio unsupported rejection increases under the fixed rule while supported accuracy remains unchanged on the screened SURE-Extended test.
Takeaways & Limitations
The results identify pre-generation unsupported-input rejection as an error mode missed by answer-only scoring.
Takeaways & Limitations
Audio-only admission cannot determine whether recognized speech answers a particular prompt, and deployment requires threshold selection matched to error costs.
Abstract
from arXiv · showhide
Speech LLMs are usually graded after they answer, although an operating system first has to decide whether a waveform should be sent to the model. We define the Speech-Unsupported Rejection Evaluation Challenge (SURE-Challenge) for this admission step. The benchmark pairs LibriSpeech-derived transcription and first-word question answering with unsupported silence, colored noise, synthetic tones, and source-ambiguous babble under disjoint source splits. Front-end ablations use Qwen2-Audio; the selected energy-plus-Whisper-score rule is then replayed before six speech/audio LLMs. On the 474-row leakage-screened SURE-Extended test set, raw Qwen2-Audio rejects 15/204 unsupported inputs, whereas the fixed rule rejects 196/204 and leaves supported accuracy unchanged. External checks delimit this number: Common Voice retention drops as the Whisper-score threshold is tightened, and no-speed babble gives 18 to 24 rejected clips out of 54 across regenerated seeds. The result identifies a pre-generation error mode missed by answer-only scoring.
1. INTRODUCTION
SURE-Challenge evaluates whether a waveform contains enough speech evidence to justify calling a speech LLM, before generation and answer scoring. It frames unsupported-input rejection as a distinct admission decision involving acoustic evidence, answerability, and source identity.
- Speech LLM deployment first requires deciding whether the waveform contains enough evidence to justify a model call.Unsupported inputs may contain background sound, music, tones, or speech from an unidentified source.
- Speech-evidence filtering admits audio to a speech LLM or abstains without calling it.The decision is separate from semantic answerability and speaker identity.
- SURE-Challenge pairs LibriSpeech-derived transcription and first-word question answering with unsupported audio under source-disjoint splits.The unsupported conditions include silence, colored noise, synthetic tones, and source-ambiguous babble.
- The experiments compare lightweight acoustic and ASR-confidence front ends before replaying one frozen rule across six speech/audio LLM backbones.External corpora probe supported speech, nonspeech and vocal audio, and overlapped speech.
- A cheap fixed rule removes most synthetic unsupported calls, while threshold choice, multilingual decoding, and source ambiguity remain separate operating constraints.
2. RELATED WORK
The paper positions SURE-Challenge between speech and audio-language-model evaluation, selective prediction, ASR confidence estimation, and hallucination studies. Its distinctive focus is the admission decision before a generator can produce a plausible response.
- Recent audio-language models combine acoustic encoders with language-model decoders and report strong speech and audio understanding results.
- Most evaluations score generated answers or unsupported content after generation, whereas SURE-Challenge evaluates the preceding admission decision.
- The front-end signals draw on ASR confidence estimation, voice activity detection, Whisper posteriors, AudioSet speech tags, and selective classification.
- An utterance may be intelligible yet unsupported when the prompt asks about a speaker or source that the audio does not identify.
3. TASK AND DATA
SURE defines source-disjoint benchmark splits for supported speech, unsupported nonspeech, and source-ambiguous babble, with explicit admission and answer-scoring criteria. The leakage-screened SURE-Extended test uses 474 rows after removing cross-split babble cases.
- Each SURE example contains audio, a prompt, an expected behavior label, and, for answerable cases, a reference transcript or answer.SURE-Core uses 40 LibriSpeech validation-clean utterances, while SURE-Extended uses 120 train-clean-100 utterances.
- Rows are split by source order, and front-end parameters are not fit on training data.Rows generated from the same source utterance remain dependent, so counts are not independent per-row samples.
- Supported families include clean speech, acoustic perturbations, and first-word question answering; unsupported families include silence, colored noise, tones, and off-source babble.Babble mixes two or four off-source utterances while asking for the “main speaker.”
- 474 SURE-Extended test rows remain after six cross-split babble cases are removed.Table 1 reports counts per split, while the main scores use the leakage-screened test set.
- Unsupported rows require abstention or clarification, while abstaining on supported speech is counted as an error.Reported metrics include supported accuracy, unsupported rejection, WER on answered supported rows, and downstream calls.
4. FRONT-END ADMISSION RULES
The admission front end combines duration and energy screening with Whisper confidence, using a frozen threshold selected to maximize unsupported rejection without supported false rejects. Separate overlap classifiers address source-ambiguous speech using acoustic summaries.
- The front end receives audio x and prompt q, decides admission before backbone execution, and passes q downstream only after admission.
- The main rule removes extremely short or near-silent clips, then averages Whisper-small maximum token probabilities into s(x).The score is computed across decoding steps.
- The rule abstains when s(x) < τ and selects τ = 0.70 by maximizing unsupported rejection under zero supported false rejects.The threshold is chosen on clean SURE-Extended development data.
- The frozen decision rule is replayed before Qwen2-Audio, Qwen2.5-Omni, Qwen-Audio-Chat, Audio Flamingo Next, Audio Flamingo 3, and MiniCPM-o.
- Source-ambiguous speech uses overlap classifiers based on RMS, spectral, zero-crossing, and MFCC summaries.Thresholds are selected on development speakers with a supported-reject limit of 0.05 and then frozen for test.
5. RESULTS
Across six backbones, a frozen front-end rule substantially improves unsupported-input rejection while preserving supported performance, but external checks expose threshold and source-ambiguity trade-offs.
- Backbone Abstention Without a Front End: 0.957 raw unsupported rejection is reached by fixed-filter replay on Core, while self-abstention suppresses valid generations as well as invalid ones.Self-abstention rejects all 70 unsupported Core rows but leaves supported accuracy at 0.033 for Qwen2-Audio and 0.156 for Qwen2.5-Omni.
- Backbone Abstention Without a Front End: 196/204 unsupported inputs are rejected by the fixed rule, versus 15/204 by raw Qwen2-Audio, with downstream calls reduced by 41%.The result is reported on the leakage-screened SURE-Extended test set.
- Front-End Controls: 120/150 non-speech and 46/54 babble rows are rejected by Whisper scoring; adding energy or Silero yields the same 196/204 total.AST and Silero alone reject all 150 synthetic non-speech rows but none of the babble rows.
- External Audio Checks: 0.964 all-unsupported rejection at Whisper threshold τ = 0.90 reduces supported English retention to 0.505 on external audio.At this threshold, ESC-50 rejection is 0.974, VocalSound rejection is 0.983, and LibriCSS rejection is 0.842.
- Source-Ambiguous Speech: 18–24/54 no-speed babble clips are rejected across regenerated seeds, compared with 40–43/54 for speed-perturbed babble.Raw Qwen2-Audio refuses 2/54 no-speed, non-circular regenerated clips.
- Source-Ambiguous Speech: 0.972/0.966 dev/test ROC-AUC is achieved by an audio-only overlap rule, which also rejects 10/270 supported examples.The classifier is excluded from the headline metric because it uses a pre-screened set and synthetic-babble training.
6. LIMITATIONS AND IMPLICATIONS
The evaluation shows that admission rules have distinct failure modes and operating constraints, while audio-only filtering cannot establish whether speech answers a prompt.
- Implications: 0.012, 0.080, and 0.949 seconds per clip are the average H100 runtimes for support-gate, Whisper, and Qwen2-Audio paths, respectively.These measurements are taken after model loading.
- Limitations: Audio-only admission cannot determine whether recognized speech answers a particular prompt.Semantic-unanswerable and LibriCSS checks place this boundary in the results.
- Limitations: Whisper threshold tightening creates a retention trade-off on Common Voice, while false accepts concentrate in overlap and vocal music.The external checks separate threshold sensitivity from source ambiguity.
- Implications: External-control tables distinguish front-end Gate R. from Sys. R., which also includes Qwen2 self-abstention.This separates filtering effects from refusal behavior inside the backbone.
7. CONCLUSION
SURE-Challenge evaluates whether waveforms should reach speech LLMs before answer scoring. Across six backbones, the fixed rule sharply increases unsupported rejection while preserving supported accuracy, but external checks show that this result depends on operating conditions and source attribution.
- Conclusion: 0.074 to 0.961 is the increase in Qwen2-Audio unsupported rejection produced by the fixed acoustic rule on the held-out SURE split.Supported accuracy is unchanged, while external rows expose different behavior for equal SURE scores and overlap.