PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems

Randy Frans Fela, Pejman Mowlaee

arXiv:2608.30348v1cs.SDeess.AS

TL;DR

The paper investigates whether speech-enhancement distortions propagate from ASR transcripts to downstream LLM intent decisions. It introduces ODR and benchmarks enhancement conditions in cascaded systems, finding that perceptual improvements do not reliably preserve LLM task performance and that standard audio metrics are insufficient for pipeline quality.

  • Problem

    Whether speech-enhancement-induced transcription distortions change downstream LLM task performance remains unresolved despite widespread use of enhancement in spoken-AI pipelines.

  • Method

    The paper introduces Output Divergence Rate (ODR) and benchmarks enhancement conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades with a fixed intent classifier.

  • Results

    MetricGAN+ more than doubles ODR versus unenhanced noisy speech, reaching 0.318 versus 0.135 despite improving PESQ, while echo reaches ODR 0.836 through speaker substitution beyond WER’s scope.

  • Takeaways & Limitations

    ODR should accompany WER when evaluating speech-enhancement front-ends before LLMs because standard audio-quality metrics provide no reliable per-clip divergence signal.

  • Takeaways & Limitations

    ODR is intended for offline development-time comparison because it requires a clean reference recording unavailable during live deployment.

Abstract

from arXiv · show

Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ρ=-0.068$, PESQ $ρ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.

1 Introduction

The paper asks whether speech-enhancement distortions affect downstream LLM intent decisions, then introduces ODR to measure this instability in cascaded voice systems.

  • Motivation: Speech enhancement is assumed acoustically neutral, but prior evidence shows MetricGAN+ and diffusion-based enhancement can worsen transcription-level semantic error.The unresolved issue is whether transcription degradation propagates into LLM reasoning or produces failures that WER misses.
  • Contribution: Output Divergence Rate (ODR) measures how often enhancement changes an LLM’s intent prediction relative to clean speech.The benchmark evaluates enhancement-induced instability at the downstream task level.
  • Benchmark: The benchmark covers 2,974 SLURP test recordings and enhancement conditions simulated with DNS Challenge corpora.Table 1 summarizes the applied conditions; the provided caption does not enumerate them.
  • Findings: Enhancement can cause systematic intent collapse toward catch-all categories, a failure mode invisible to WER and standard audio-quality metrics.The paper also reports that findings replicate across ASR architectures and remain robust to LLM capacity.

2 Related Work

Prior work documents enhancement-related ASR and downstream NLP problems, but does not isolate whether deployed speech-enhancement stages alter LLM semantic outputs.

  • Enhancement and ASR: Prior studies show speech enhancement can increase semantic WER across ASR configurations, but do not test downstream LLM behavior.The paper positions its question beyond transcription-level degradation.
  • Downstream NLP: Denoising has been reported to amplify demographic bias in ASR, while joint SE–intent training addresses downstream NLP concerns through retraining.This paper instead diagnoses already-deployed pipelines without retraining enhancement models.
  • Evaluation gap: VoiceBench evaluates voice assistants under acoustic variation as a black-box pipeline, whereas URGENT benchmarks enhancement mainly with perceptual and ASR-based metrics.The paper identifies missing LLM semantic-output evaluation that isolates speech enhancement as the variable.

3 Benchmark Design

The benchmark evaluates simulated enhancement conditions on real SLURP recordings, passes each condition through two ASR architectures and a fixed closed-set LLM classifier, and computes ODR against clean speech.

  • Dataset: 2,974 real human SLURP recordings span 18 domains and 77 intent classes; the TTS subset is excluded.The benchmark uses the real test split because uniform TTS acoustics would suppress enhancement artefacts.
  • Enhancement conditions: Five enhancement conditions cover noise suppression, echo cancellation, and dereverberation, with DNS Challenge corpora supplying simulated degradations.The conditions include Noisy, MetricGAN+, simulated Echo, Dereverberation, and Echo + DTLN-AEC; the table caption describes six conditions overall.
  • ASR cascades: Whisper large-v3 is the primary ASR model, while wav2vec2-large-960h provides an encoder-decoder versus CTC architectural comparison.All ODR values are measured relative to the same clean baseline; Whisper clean-speech WER is 0.507.
  • LLM classification: A fixed Gemini 2.5 Flash Lite prompt classifies transcripts over the 77 SLURP intents with temperature 0 and a maximum of 15 tokens.Invalid predictions occur at 0.07–0.20% across conditions without systematic condition dependence.
  • Evaluation metrics: ODR compares each condition’s predicted intent with the clean prediction, excluding invalid predictions from the denominator.The WER–ODR gap is also reported; negative values indicate partial LLM robustness to condition-induced transcription errors.
  • Audio-quality analysis: Six audio metrics span perceptual, signal-level, and non-intrusive MOS-prediction families, with Spearman ρ and Pearson r evaluated against binary ODR.Bonferroni correction is applied across the six metric comparisons.

4 Results

Speech enhancement affects downstream intent predictions in condition-specific ways that standard signal and transcription metrics do not reliably capture. MetricGAN+ increases divergence despite improving PESQ, echo causes speaker-substitution failures, and patterns replicate across ASR architectures.

  • ODR Across Conditions: 0.318 ODR under MetricGAN+ more than doubles the 0.135 noisy baseline, despite MetricGAN+ improving PESQ.Dereverberation has the lowest Whisper ODR at 0.108, comparable to noisy speech; wav2vec2 preserves the same ordering at higher absolute values.
  • ODR Across Conditions: 0.836 ODR under unmitigated echo reflects speaker substitution: both ASR models transcribe the far-end signal rather than the user.The resulting transcript can be valid but entirely wrong, making this failure qualitatively different from spectral distortion.
  • ODR Across Conditions: 51.7% relative ODR reduction after DTLN-AEC closely mirrors the 51.6% WER reduction, as ODR falls from 0.836 to 0.404.This proportional recovery supports speaker substitution as the primary echo-damage mechanism.
  • ODR Across Conditions: MetricGAN+ causes systematic intent collapse toward catch-all categories after removing domain-specific lexical cues.Among MetricGAN+ regressions, 48.4% land in general quirky; general quirky and general greet occur at 13.4% and 6.8%, versus 5.2% and 2.0% under clean speech.
  • Quality Metrics as ODR Predictors: PESQ reaches only moderate pooled correlation with ODR (ρ = −0.467), while within-condition correlations reach at most |ρ| = 0.34.Pooled correlations mainly reflect separation between enhancement conditions, not clip-level predictive power within a pipeline.
  • Quality Metrics as ODR Predictors: No PESQ threshold separates safe from unsafe clips across conditions: 82.3% of above-median Echo clips still diverge, while PESQ over-warns under Dereverb and Noisy.Regressions outnumber corrections in every condition, so ODR predominantly reflects harmful rather than benign divergence.
  • ODR Across Conditions: The condition ranking is identical across Whisper and wav2vec2, with wav2vec2 showing higher spectral-distortion ODR by mean Δ = +0.157.Echo remains nearly convergent across architectures (Δ = +0.017), while CTC-based decoding may be more sensitive to frame-level spectral artefacts.

5 Deployment Implications

Deployment evaluation should prioritize downstream LLM task behavior alongside audio and ASR metrics, with echo cancellation treated as a prerequisite for conferencing systems.

  • Evaluation protocol: ODR should accompany WER as an offline criterion for comparing speech-enhancement configurations before deployment.Because ODR requires clean reference recordings, it is a development-time measure rather than an online per-utterance monitor.
  • Echo cancellation: 0.836 ODR under echo means WER can saturate while the LLM processes a semantically unrelated far-end utterance.Echo reaches ODR = 0.836 with WER = 1.448, exposing speaker substitution that WER cannot characterize.
  • Echo cancellation: Functional acoustic echo cancellation should be treated as a prerequisite for LLM integration in conferencing deployments.The echo condition reaches ODR values of 0.836 and 0.853 across the two ASR models, meaning fewer than one in six interactions are handled correctly.
  • Evaluation protocol: PESQ shows only moderate association with ODR (ρ = −0.467), while SQUIM-MOS shows near-zero association (ρ = −0.068).Perceptual quality optimization therefore provides no reliable guarantee of downstream LLM performance.

6 Conclusion

The paper introduces ODR to measure whether speech enhancement changes downstream LLM intent predictions relative to clean speech. Results show that perceptual improvements do not ensure task performance, while echo causes speaker-substitution failures beyond WER’s scope.

  • Conclusion: ODR measures how often speech enhancement changes an LLM’s intent prediction relative to clean speech.It provides a task-level measure of enhancement-induced instability in cascaded ASR–LLM systems.
  • Conclusion: MetricGAN+ more than doubles ODR despite improving PESQ, showing that perceptual quality and LLM task performance can diverge.Under Whisper, ODR rises from 0.135 for noisy speech to 0.318 with MetricGAN+.
  • Conclusion: 0.836 ODR under unmitigated echo reflects speaker substitution, a failure mode beyond WER’s scope.Audio quality metrics range from near-zero to moderate correlation with ODR, and the observed patterns replicate across two ASR architectures.
  • Conclusion: ODR should be used alongside WER as an offline evaluation criterion for speech-enhancement front ends before LLM deployment.The released pipeline supports evaluation across additional LLMs, tasks, and speech-enhancement systems.

Limitations

The study’s conclusions are bounded by simulated English data, selected ASR/LLM/SE systems, closed-set intent evaluation, and offline clean-reference requirements.

  • Real-device recordings may differ in acoustic profile, compound distortion, and room geometry, motivating replication on field-recorded data.
  • Only streaming CTC and attention-CTC hybrid ASR architectures remain untested; other CTC variants may amplify spectral-condition effects.
  • The benchmark uses English-only SLURP data, so enhancement artefacts may interact differently with ASR and LLM processing in other language types.
  • ODR covers fixed 77-class intent labels, while open-ended generation tasks may show different sensitivity to transcript distortions.
  • Primary results use Gemini 2.5 Flash Lite; full multi-family LLM evaluation and all-condition Gemini 2.5 Pro replication remain future work.
  • ODR requires a clean reference per clip and is intended for offline development-time comparison, not online per-utterance monitoring.

Ethical Considerations

The work uses public datasets and models without collecting or generating personally identifiable information, and holds the classifier prompt constant across evaluations.

  • The study uses publicly available SLURP and DNS Challenge datasets and publicly available Whisper, MetricGAN+, WPE, and wav2vec2 models.
  • No personally identifiable information was collected or generated, and SLURP was collected with informed consent from participants.
  • The same classifier prompt was held constant across all conditions, clips, and ASR models.

B ASR Architecture & WER Comparison

The appendix provides the full per-condition ODR and WER breakdown, while domain-level results identify variation in vulnerability across SLURP semantic domains.

  • ASR Architecture & WER Comparison: Table 6 provides the full per-condition ODR and WER breakdown underlying the architecture comparison in Section 4.
  • ASR Architecture & WER Comparison: Mean ODR exceeds 0.38 across conditions in Lists, IoT, and Alarm, whereas QA, News, and General are more robust.
  • ASR Architecture & WER Comparison: Short-command domains are described as especially vulnerable because enhancement-induced lexical loss can affect intents carried by a single keyword.

E Within-Condition Metric Correlations

Within individual enhancement conditions, audio-quality metrics have weak and inconsistent relationships with clip-level output divergence, limiting their predictive value.

  • Within any single condition, no metric exceeds |ρ| = 0.34, with most correlations below 0.15 and several changing sign.
  • Pooled correlations overstate predictive value because they conflate within-condition variation with between-condition separation.
  • The pooled-to-within-condition reduction indicates that pooled correlations do not establish clip-level predictive power.

F Log-Mel Spectrogram Comparison Across Conditions

Figure 4 compares five SLURP utterances across six conditions using shared log-mel spectrogram axes and colour scaling. The visualizations show broadband-noise suppression by MetricGAN+ alongside vertical streak artefacts and altered harmonics.

  • Figure layout: Figure 4 arranges five representative SLURP utterances as rows and six conditions as columns, with time on the horizontal axis and frequency on the vertical axis.Each panel uses shared amplitude colour scaling, and each row displays the utterance’s ground-truth transcript above it.
  • Spectrogram comparison: MetricGAN+ suppresses the high-frequency broadband noise visible in the noisy condition.
  • Spectrogram comparison: MetricGAN+ introduces vertical streak artefacts identified as musical noise, visible above 4 kHz.
  • Domain vulnerability: Short-command domains including Lists, IoT, and Alarm are consistently more vulnerable than QA, News, and General across all enhancement conditions.Table 7 separates domains with mean ODR ≥0.37 from those below that threshold.
  • Metric relationships: MetricGAN+ is the only condition in which any audio metric exceeds an absolute within-condition correlation of 0.2 with binary divergence.STOI reaches −0.340 and PESQ reaches −0.259; Echo + DTLN-AEC shows sign inversion for positive SNR and SI-SDR values.
Loading 2608.30348v1…