PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluation

Mengzhe Geng

arXiv:2608.27817v1cs.SDcs.CLeess.AS

TL;DR

Audio-versus-transcript evaluations do not isolate whether generative audio calls are necessary for known closed-set decisions. The paper uses matched call-decision policies with transcript, encoder, and generative actions, finding strong no-call controls and only a small full-versus-no-call difference on VocalSound.

  • Problem

    Waveform-versus-transcript comparisons reveal acoustic evidence but leave unresolved whether a generative audio model is needed after transcript and encoder evidence are available.

  • Method

    The paper evaluates locked, matched selectors that choose among transcript labels, CLAP, AST, WavLM, and generative audio-model actions, then removes generative actions while preserving selection procedures.

  • Results

    0.925 accuracy with 12.5% generative calls versus 0.921 for the matched no-call selector yields a paired difference of 0.004 with 95% CI [-0.025,0.033].

  • Takeaways & Limitations

    For known-task endpoint claims, assess the marginal value of the generative call after transcript and encoder evidence have already been used.

  • Takeaways & Limitations

    The evaluation inherits biases from ASR systems, audio encoders, and LLMs and is limited to known closed-set label decisions.

Abstract

from arXiv · show

Speech and audio LLMs are often evaluated by asking whether a waveform prompt beats an automatic speech recognition (ASR) transcript. For known closed-set tasks, that comparison conflates two factors: access to acoustic evidence and the need to call a generative audio model. We evaluate this distinction as a controlled call-decision problem. For each example, a policy chooses among keeping a transcript label, using encoder evidence from Contrastive Language-Audio Pretraining (CLAP), Audio Spectrogram Transformer (AST), or WavLM, and calling Qwen2-Audio, Qwen2.5-Omni, or MOSS-Audio; the decisive ablation removes all generative actions while keeping the selector and development protocol fixed. On VocalSound, transcripts reach 0.296 accuracy, so waveform information is needed. Yet supervised CLAP and WavLM controls reach 0.850 and 0.854 with no generative audio calls. A selector with generative actions reaches 0.925 accuracy using 12.5% calls, compared with 0.921 for the matched no-call selector (paired difference 0.004; 95% CI [-0.025,0.033]). Agreement and stacking features improve weaker selectors but do not beat the strongest no-call control. For known-task endpoint claims, the relevant quantity is the marginal value of the generative call after transcript and encoder evidence have already been used.

1. INTRODUCTION

The paper separates the value of waveform evidence from the value of invoking a generative audio model in known closed-set tasks. It evaluates matched policies that choose among transcript labels, local encoders, and generative audio models.

  • 1. INTRODUCTION: Audio-LLM versus transcript comparisons show whether waveforms contain useful evidence, but not whether a generative audio model was necessary.ASR can discard vocalization, emotion, speaker-state, and overlapping-speech information.
  • 1. INTRODUCTION: A deployed closed-set policy can keep a transcript label, score the waveform with a local encoder, or send it to a generative audio model.Generative calls may improve decisions while adding cost and speech-data exposure.
  • 1. INTRODUCTION: The study asks whether generative calls still change decisions enough to justify their use after transcript and encoder evidence are available.The comparison uses the same development-selected policy across conditions.
  • 1. INTRODUCTION: The evaluation uses locked row splits and matched controls across Qwen2-Audio, Qwen2.5-Omni, MOSS-Audio, CLAP, AST, and WavLM.Reported analyses include paired intervals, Holm-adjusted tests, and measured sequential costs.

2. RELATED WORK

Prior benchmarks primarily measure reusable representations, model ability, or collaborative speech-task performance. This work instead fixes a known task and tests the marginal effect of routed generative audio calls after encoder evidence is available.

  • 2. RELATED WORK: SUPERB evaluates reusable speech representations, while Dynamic-SUPERB extends evaluation to collaborative instruction-following speech tasks.Recent audio-language benchmarks also cover generative comprehension, assistant behavior, hallucination, multi-talker speech, and speaker understanding.
  • 2. RELATED WORK: This evaluation fixes a known task and asks whether generative audio calls still change decisions after transcript and encoder controls are included.Its focus differs from benchmarks that primarily measure model ability.
  • 2. RELATED WORK: The study combines ideas from calibration, misclassification detection, selective prediction, ASR confidence, and LLM cascades.Unlike transcript cascades, its first-stage evidence comes from audio encoders including CLAP, AST, and WavLM.

3. GENERATIVE-CALL DECISION TEST

The decision test routes examples among transcript, encoder, and generative actions under development-selected call budgets. Matched full and no-call selectors isolate the marginal value of generative actions while holding selection procedures fixed.

  • 3. GENERATIVE-CALL DECISION TEST: For utterance x_i, ASR produces transcript y_i and a text LLM maps it to transcript label ĉ_text.Transcript confidence and generative-call indicators can enter routing decisions.
  • 3. GENERATIVE-CALL DECISION TEST: A generative audio model predicts ĉ_gen directly from the waveform, and a routing policy selects examples R_b under target call budget b.The policy returns transcript-side or generative decisions according to routing.
  • 3. GENERATIVE-CALL DECISION TEST: R_b contains examples whose routing scores s_i exceed the development-selected threshold τ_b.Thresholds are chosen on development rows rather than holdout rows.
  • 3. GENERATIVE-CALL DECISION TEST: Routing scores combine transcript risk, Whisper uncertainty, text-LLM label-likelihood uncertainty, and an indicator related to transcript-label membership.The weights and threshold are selected using development rows only.
  • 3. GENERATIVE-CALL DECISION TEST: Encoder-first policies score every waveform with CLAP and route low-margin examples to a generative audio model.Supervised CLAP, WavLM, and AST probes serve as deployment controls using labeled development rows.
  • 3. GENERATIVE-CALL DECISION TEST: The full selector fits an L2-regularized logistic correctness model over row–action pairs, while the matched no-call selector removes Qwen2 and Omni actions.Both use the same training and selection protocol; ties favor fewer generative calls.
  • 3. GENERATIVE-CALL DECISION TEST: Parser settings, features, weights, thresholds, and λ are fixed on development rows before unchanged application to holdout rows.Paired intervals use 10k row bootstraps, exact McNemar tests, Wilson intervals, Holm correction, and a 10-seed diagnostic.
  • 3. GENERATIVE-CALL DECISION TEST: Accuracy and measured sequential cost are reported separately for transcript-first routes.This keeps performance comparisons distinct from execution-cost accounting.

4. EVALUATION

VocalSound shows that transcripts alone are insufficient, while supervised encoders are strong deployment controls. Generative routing improves some policies, but the strongest matched no-call selector nearly ties the full selector and remains competitive across comparisons.

  • 4. EVALUATION: VocalSound is a six-way human-vocalization task with 0.167 chance accuracy and a locked 480–719 row holdout.The study also evaluates ESC-50 Animals.
  • 4. EVALUATION: 0.296 transcript-only accuracy confirms that transcripts are insufficient for VocalSound.Supervised CLAP-embedding and WavLM-base+ controls reach 0.850 and 0.854 accuracy without generative calls.
  • 4. EVALUATION: 0.887 accuracy is reached by the best routed policy, which sends low-margin CLAP examples to Omni at a 50.8% generative-call rate.Its paired advantage over WavLM-emb is ∆=0.033 with 95% CI [−0.017, 0.083], so the comparison is inconclusive.
  • 4. EVALUATION: The figure plots VocalSound row-holdout accuracy against generative-call rate, counting only generative audio-model calls.Cheap transcript and local-feature controls still process inputs, and the full selector’s paired advantage over the no-call selector includes zero.
  • 4. EVALUATION: 0.925 versus 0.921 is the direct full-versus-no-call selector comparison, with ∆=0.004 and CI [−0.025, 0.033].Agreement-aware and stacked selectors remain below the strongest no-call policy in cross-family comparisons.
  • 4. EVALUATION: Table 1 reports VocalSound holdout policies with generative-call rate, cached sequential Total and Gen. seconds, and paired contrasts against each row’s no-call baseline.The holdout summary contains n=240 examples.

5. DISCUSSION AND LIMITATIONS

The evaluation is constrained by inherited model and dataset biases, residual audio exposure, and a narrow known-task scope. The authors identify a larger speaker-disjoint rerun as the most important next experiment.

  • The evaluation inherits ASR, encoder, and LLM biases across accents, dialects, disordered speech, code-switching, and low-resource languages.
  • Reducing generative calls does not eliminate audio exposure because selectors still process transcripts, encoder features, routed waveforms, or intermediate summaries.
  • The VocalSound split is locked, while fresh ESD is row-disjoint but small.
  • A larger speaker-disjoint rerun is identified as the most important next experiment.
  • The result is task-scoped, with richer reasoning tasks, unknown task mixtures, few-shot adaptation, calibrated generative log-probabilities, and throughput-aware serving left as separate tradeoffs.

6. CONCLUSION

The conclusion argues that an audio-versus-transcript gain does not establish that generative audio was necessary. It calls for explicit endpoint-call boundaries because strong no-call selectors nearly match full selectors.

  • Audio-versus-transcript gains alone do not show that a generative audio call was necessary.
  • Supervised encoders can be strong on non-speech vocalization tasks when development labels exist.
  • Future reports should define the call boundary and evaluate the marginal value of generative calls after transcript and encoder evidence.
Loading 2608.27817v1…