PaperlayerView exact revision on arXiv

Source-linked AI summary

Objective Intelligibility Prediction Using Distance Metrics on Speech Foundation Model Representations

Lyonel Behringer, Andreas Brendel

arXiv:2609.13046v1eess.AS

TL;DR

Subjective intelligibility testing is costly, and it remains unclear whether trained predictors based on speech foundation model representations generalize to out-of-distribution signals. This paper evaluates layer-wise embedding distances without further training and finds Whisper representations, particularly with Fréchet Audio Distance, most suitable for intelligibility prediction.

  • Problem

    Subjective intelligibility assessment is costly, while the robustness of trained predictors using speech foundation model representations to out-of-distribution signals remains unclear.

    Source: §1 · p. 1

  • Method

    The study compares layer-wise distances between reference and test embeddings from pretrained speech foundation models with subjective intelligibility scores, without further training.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Whisper representations perform best overall, with the last encoder and decoder layers yielding the best results and correlations improving with model size.

    Source: §1 · pp. 1–2, §4 · p. 4

  • Takeaways & Limitations

    Embedding distances outperform classical signal-processing metrics, are more robust across datasets than WER and CER, and are differentiable for possible use as loss functions.

    Source: §1 · p. 2, §4 · p. 4

Abstract

from arXiv · show

High-dimensional representations of pretrained speech foundation models have proven beneficial for objective speech quality and intelligibility prediction. While existing work on neural intelligibility prediction usually leverages such representations for task-specific fine-tuning, in this work we evaluate the usefulness of such representations for intelligibility prediction without any further training. We conduct a layer-wise analysis of multiple speech foundation models, correlating various embedding distances with subjective intelligibility scores. The results show that embeddings extracted from Whisper speech recognition models are best suited, with the last encoder and decoder layers yielding the best correlations when using the Fr\'echet Audio Distance. Notably, the evaluated distances outperform classical intelligibility metrics and are more robust than Word and Character Error Rates. Further, correlations improve with increasing size of the Whisper model from which embeddings are extracted.

1. INTRODUCTION

Subjective intelligibility tests are costly, while objective assessment methods based on speech foundation model representations have unresolved questions about robustness and generalization. This work evaluates pretrained-model embedding distances without further training and finds Whisper representations especially suitable.

  • Subjective intelligibility tests are costly and time intensive, motivating objective assessment of speech processed by communication technologies.Source: §1 · p. 1
  • Prior neural prediction work uses pretrained representations for task-specific training, but their robustness to out-of-distribution signals is unclear; neural ASR error rates are usually English-limited and nondifferentiable.Source: §1 · p. 1
  • The study compares embedding distances without further training, extending prior layer-wise work to decoder layers, signal-level correlations, and enhanced and coded speech.Source: §1 · p. 1
  • Whisper representations are best suited among the evaluated speech foundation models, and correlations improve with increasing Whisper model size.Source: §1 · pp. 1–2
  • The last encoder and decoder layers of Whisper yield the best results in the layer-wise analysis.Source: §1 · pp. 1–2
  • Fréchet Audio Distance performs best overall, and embedding distances are more robust across evaluated datasets than Character Error Rate.Unlike nondifferentiable baselines, the distances are differentiable and may be used as a loss function.Source: §1 · p. 2

2. METHODOLOGY

The method extracts layer-wise embeddings from pretrained speech foundation models and compares clean reference speech with corresponding test signals. It measures differences between their embeddings using centroid- and distribution-based distances, computed at signal level.

2. METHODOLOGY

  • Distances compare layer-wise embeddings of clean reference speech, assumed perfectly intelligible, with embeddings of corresponding test signals.A pretrained speech foundation model maps speech frames to embedding vectors at intermediate layers or its final output.Source: §2 · p. 2
  • L2 and cosD measure differences between the sample-mean vectors of reference and test embeddings.cosD is defined on sample-mean embedding vectors.Source: §2 · p. 2
  • Squared FAD compares Gaussian approximations of the reference and test embedding distributions, capturing their spread as well as their centroids.The distributions are estimated from sample means and covariance matrices.Source: §2 · p. 2
  • MMD provides a distribution-free, unbiased alternative based on a positive-definite kernel; the method uses an RBF kernel with bandwidth selected by the median-distance heuristic.Source: §2 · p. 2
  • All metrics are computed at signal level, where each signal’s embeddings form a set; FAD and MMD therefore use a different aggregation level from their traditional system-level computation.For L2 and cosD, each signal’s embeddings are averaged over time.Source: §2 · p. 2

3. EXPERIMENTAL SETUP AND RESULTS

The experiments compare pretrained speech-model embeddings and baseline metrics against subjective intelligibility scores on NCLEIR and TMHINTQI. Whisper embeddings and Fréchet Audio Distance (FAD) perform especially well, with results varying by dataset and layer.

3.1. Datasets

  • The evaluation uses NCLEIR coded English speech and TMHINTQI noisy Mandarin speech, comparing each dataset’s test utterances with the specified clean or noisy references.Utterance intelligibility is the correctly transcribed word ratio averaged across listeners.Source: §3.1 · p. 2

3.2. Evaluated Features and Baseline Measures

  • Features include three speech foundation models, MFCC baselines, and intelligibility metrics; embeddings are extracted from every Transformer layer, including Whisper decoder layers.Baselines include STOI, ESTOI, Whisper-based WER and CER, and MOSA-Net+; Whisper Tiny, Base, and Large-v3 are evaluated for model-size comparisons.Source: §3.2 · pp. 2–3

3.3. Model Comparison

  • Whisper Base gives the strongest overall cosine-distance correlations, with its last decoder layer best on NCLEIR and last encoder layer best on TMHINTQI.W2V2-PT shows no correlation; W2V2-ASR has moderate PCC and SRCC on NCLEIR but weak correlations on TMHINTQI, which the authors attribute to English-only ASR fine-tuning data.Source: Table 1 · p. 3, §3.3 · p. 3

3.4. Layer-Wise Evaluation

  • On TMHINTQI, embedding distances exceed WER and CER in absolute PCC, while for SRCC CER is comparable but remains below last-encoder-layer FAD.On NCLEIR, CER and WER PCC exceed all embedding distances; intermediate decoder layers show substantial PCC drops, but only slight SRCC decreases for FAD and L2 on TMHINTQI.Source: §3.4 · p. 3
  • FAD is the most robust distance overall, with the final encoder and decoder layers yielding the best results; the best layer varies by test set.Across datasets, the last decoder layer with FAD yields the overall best correlations.Source: §3.4 · pp. 3–4, Figure 2 · p. 4

3.5. Impact of Whisper Model Size

  • Whisper Large-v3 shows the strongest embedding-distance correlations across metrics, while correlations generally improve with model size at the cost of higher model complexity.Base is mostly more correlated than Tiny across both datasets.Source: Table 2 · p. 4, §3.5 · p. 4
  • MOSA-Net+ has the strongest TMHINTQI correlation, but on NCLEIR its correlation is similar to FAD and is surpassed by WER and CER.MOSA-Net+ was trained on TMHINTQI, while WER and CER may be preferable when test data is expected to resemble Whisper training data, predominantly English.Source: §3.5 · p. 4

4. CONCLUSION

The study evaluates pretrained speech-foundation-model embedding distances for intelligibility prediction without further training. Whisper and FAD perform best overall, with larger Whisper models and final encoder and decoder layers producing stronger correlations.

4. CONCLUSION

  • Pretrained Whisper representations, FAD, and the final encoder and decoder layers yield the strongest intelligibility correlations; larger Whisper models improve correlations, and embedding distances are more robust across datasets than WER and CER.The distances are differentiable and may be used as intelligibility-based training losses; FAD showed no numerical issues in practice despite expected lower sample efficiency.Source: §4 · p. 4
Loading 2609.13046v1…