Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

I. INTRODUCTION

• Standard error rates measure reference agreement but do not reveal whether a correctly recognized word depended on its local audio.Confidence estimation predicts correctness but does not identify which acoustic evidence supported an output.Source: §I · p. 1
• Unlike work targeting specially constructed underdetermined positions, the score measures local acoustic contribution for arbitrary reference-aligned words, including normally recognized clean targets.It measures counterfactual sensitivity to the aligned local span.Source: §I · p. 1
• AEC is the mean decrease in teacher-forced reference-word token log probability when the aligned acoustic span is replaced.Paired clean and edited passes hold the model, target, tokenization, and prefix fixed.Source: §I · p. 1

II. METHOD

• The analysis tracks clean decoding correctness, paired reference-likelihood change, and edited-audio decoding behavior for each target span.The paired score measures local acoustic contribution; the other two quantities describe decoding behavior.Source: §II · p. 1

B. Acoustic Evidence Contribution

• Holding the reference token, prefix, and tokenization fixed makes the paired log-likelihood difference isolate the acoustic intervention from decoded-history changes.The target-span AEC averages word responses; token averaging normalizes for subword-token count, and constituent words receive equal weight.Source: B. Acoustic Evidence Contribution · p. 2
• Positive AEC means editing the target span lowers reference-word likelihood, near-zero means little sensitivity, and negative means edited audio raises it.Off-target word responses are used to evaluate locality and construct the untouched control distribution.Source: B. Acoustic Evidence Contribution · p. 2
• AEF exponentiates AEC; a value of 2 means clean input assigns twice the target likelihood of edited input on the normalized geometric-mean scale.For phrase targets, AEF is the geometric mean of constituent word-level evidence factors.Source: B. Acoustic Evidence Contribution · p. 2

• The high-contribution indicator uses τ = ln 2, marking spans whose target-level normalized likelihood factor exceeds two.Source: C. Outcome–contribution matrix · p. 2
• The matrix counts clean-decoding outcomes—correct, substituted, or deleted—against low or high local acoustic contribution.Source: C. Outcome–contribution matrix · p. 2
• LCCS is the fraction of correct targets with low local acoustic contribution; higher values indicate weaker reliance on the aligned span.Source: C. Outcome–contribution matrix · p. 2
• For phrase targets, AEC is averaged across constituent words, and a phrase is correct only when every word is recognized correctly.Source: C. Outcome–contribution matrix · p. 2

• Edited hypotheses are aligned to the original reference by minimum word-edit alignment, and retention records whether the target remains after intervention.Surplus hypothesis words do not change the retention indicator.Source: D. Behavior after evidence removal · p. 2
• Clean-correct targets are ranked by AEC within each model–corpus cell and divided into five equal-count quintiles.Source: D. Behavior after evidence removal · p. 2
• Retention rates are macro-averaged over systems within each corpus; a multiword target is retained only if every word remains correctly recognized.Source: D. Behavior after evidence removal · p. 2

A. Datasets

• The evaluation samples 1,053 TED-LIUM, 5,181 AMI, and 2,693 LibriSpeech target spans, using silence as the primary intervention and noise or music as robustness checks.One target span is sampled per eligible utterance from specified span categories; aligned spans receive 20-ms cosine crossfades.Source: A. Datasets · pp. 2–3

B. Models

• Six models span 1.5B–24.3B parameters, comprising four specialized ASR models and two general-purpose speech LLMs.All models use greedy decoding; speech LLMs receive a transcription-only prompt.Source: B. Models · p. 3

C. Evaluation

• LCCS and retention are reported for all six systems across three corpora, with 95% confidence intervals from 1,000 utterance-level bootstrap resamples.Source: C. Evaluation · p. 3, B. Models · p. 3
• The evaluation compares edited-span sensitivity with single-pass entropy, paired clean–edited entropy change, and silence-conditioned prior surprisal.Span sensitivity contrasts scores for target and untouched words from the same utterance.Source: C. Evaluation · p. 3

• RQ1 tests whether AEC responds specifically to the edited span rather than to the utterance globally.Source: §IV · p. 3
• RQ2 estimates how often a correct target has low AEC at the fixed threshold.Source: §IV · p. 3
• RQ3 tests whether increasing AEC corresponds to progressively lower retention under edited-audio free decoding.Source: §IV · p. 3

RQ1. Does AEC measure a localized response?

• AEC is concentrated at the edited span and approaches zero for neighboring words, supporting a strongly localized output-level response.The response also decays with word distance from the intervention.Source: RQ1. Does AEC measure a localized… · p. 3, Figure 2 · p. 3
• AEC has the highest AUROCspan on TED-LIUM and AMI and for each intervention family, while ΔH is higher on LibriSpeech.AEC and ΔH provide comparable localization but measure different quantities: reference-target probability change versus distributional uncertainty change.Source: Table I · p. 3, RQ1. Does AEC measure a localized… · p. 3

RQ2. How often do correctly recognized words exhibit weak dependence on their aligned local acoustic span?

• 33.1% for Audio Flamingo Next and 33.8% for Qwen3 versus 48.7% for MERaLiON-3: averaged single-word LCCS spans 33.1–48.7% across systems.Correct recognition therefore does not necessarily imply strong dependence on the aligned local audio.Source: RQ2. How often do correctly recognized… · p. 3, Table II · p. 4
• Across 18 model–corpus cells, LCCS and WER have no evident rank association (Spearman ρ = 0.03), so LCCS captures an acoustic contribution axis WER does not.The comparison is descriptive, not a system quality ranking.Source: RQ2. How often do correctly recognized… · pp. 3–4, Table III · p. 4

RQ3. Does retention decrease as AEC increases?

• 76–83% retention in the lowest AEC quintile versus approximately 1% in the highest: retention decreases monotonically across quintiles in all 18 model–corpus cells.For high-AEC spans, retention ranges from 2.5–17.3%, compared with 69.3–96.1% for low-AEC spans.Source: RQ3. Does retention decrease as AEC… · p. 4

Threshold calibration

• Fig. 4 plots false positive rate on untouched words against true positive rate on removed-span words as the threshold varies.Thin lines represent individual systems; thick lines represent corpus means.Source: Threshold calibration · p. 4, Figure 4 · p. 4
• 69.8% of target spans exceed AEF > 2, while 1.33% of untouched controls do; no system–corpus cell exceeds 3.2% false positives.The paper reports this as a useful low-FPR operating point that was not fitted to optimize the evaluation.Source: Threshold calibration · p. 4, Figure 4 · p. 4
• The prespecified AEF > 2 threshold provides a useful low-FPR operating point without being fitted to optimize this evaluation.Source: Threshold calibration · p. 4

V. CONCLUSION

• AEC is a counterfactual word-level measure of local acoustic contribution, paired with an outcome-contribution framework that separates correctness from local acoustic dependence.Source: §V · p. 4
• Correctly recognized targets become progressively less recoverable after intervention as their local acoustic contribution increases.Source: §V · p. 4
• The prevalence of low-contribution correct outputs varies across systems and is not explained by clean WER.Source: §V · p. 4