Source-linked AI summary
Perceptual Differences and Preferences in Speech Enhancement
Simon Dahl Jepsen, Jesper Rindom Jensen
TL;DR
Objective and subjective speech-enhancement evaluations do not establish whether listeners can distinguish processed signals or systematically prefer one. The paper uses crowdsourced triangle and pairwise preference tests across four systems, finding that discrimination and preference vary across systems and acoustic conditions and that most objective metrics correlate poorly with both.
Problem
Conventional speech-enhancement scores do not establish whether listeners can distinguish processed signals or systematically prefer one, a question relevant as systems achieve similar benchmark scores.
Source: §1 · p. 1Method
The study evaluates four speech-enhancement systems with crowdsourced triangle discrimination and pairwise preference tests, and examines whether objective-metric differences predict these dimensions.
Source: §1 · p. 1Results
Systems with similar objective scores can remain perceptually distinguishable, discrimination is not necessarily systematic preference, and most objective metrics correlate poorly with both.
Source: §5 · p. 4Takeaways & Limitations
Discriminability and preference provide complementary evaluation dimensions, and utterance-level results show that both vary with acoustic conditions.
Source: §1 · p. 1, §4.1 · p. 3, §4.2 · p. 4Crowdsourced judgments may vary with unmeasured listener and playback factors, and the test utterances come from a corpus different from the models’ training corpus.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Speech enhancement (SE) systems are commonly evaluated with objective metrics, with and without references, and with categorical subjective listening tests, all measuring speech quality and intelligibility. So far, it is unclear from such evaluations whether the processed speech is perceptually different or is preferred by everyday listeners. We investigate discrimination and preference as complementary dimensions of SE evaluation through crowdsourced triangle and pairwise preference tests across four SE systems, with comparisons to clean references. Results show that both listeners' ability to distinguish and whether they exhibit systematic preference vary across systems and substantially across acoustic conditions. Pairwise evaluations of objective metrics showed variable correlation with both discriminability and preference, motivating further studies into these novel aspects in speech enhancement evaluation.
1. INTRODUCTION
Speech enhancement seeks to infer clean speech from noisy recordings, but conventional evaluation does not establish whether listeners can distinguish processed signals or systematically prefer one. This work adds discrimination and preference tests alongside objective metrics to evaluate four contemporary systems.
1. INTRODUCTION
- Speech enhancement aims to infer clean speech from noisy observations, with learning-based approaches including discriminative and generative methods.Evaluation commonly combines subjective listening tests with objective measures of fidelity, intelligibility, and perceptual quality.Source: §1 · p. 1
- Many objective measures require a clean reference, which is generally unavailable for real-world recordings; non-intrusive measures instead estimate quality from the observed signal.Reference-free models can also estimate established measures without the corresponding clean speech.Source: §1 · p. 1
- Metric differences do not necessarily mean listeners can distinguish signals, and distinguishability does not establish systematic preference.These questions matter as high-performing systems achieve increasingly similar benchmark scores.Source: §1 · p. 1
- The study evaluates discrimination and preference as complementary dimensions using crowdsourced tests of four high-performing systems and comparisons across speech material.It also investigates whether objective-metric difference scores predict these dimensions.Source: §1 · p. 1
2. METHODS
The methods compare systems using utterance-level objective-metric differences and subjective listening tests. Triangle tests measure discrimination, while paired preference tests distinguish decisive choices from no-preference responses.
2. METHODS
- Objective systems are compared using utterance-wise metric-score differences, with the magnitude described as the absolute metric difference.The paired differences support comparison with paired subjective evaluations.Source: §2 · pp. 1–2
2.1. Discrimination Testing
- The triangle test presents three versions of the same utterance, two from one system and one from another, and asks listeners to identify the odd one out.Repeated judgments are modeled as binomial, with chance probability 1/3; significance is assessed using an exact one-sided binomial test.Source: §2.1 · p. 2
- For discrimination, the study correlates paired objective-metric differences with estimated discrimination rates and models their relationship with a psychometric function.The function uses the triangle test's chance rate and logistic parameters estimated by maximum likelihood; confidence intervals are bootstrapped over utterances.Source: §2.1 · p. 2
2.2. Preference Testing
- Listeners hear two versions of an utterance and select which they prefer for everyday listening, with a no-preference option for cases where neither is preferred.Decisiveness captures whether a listener chooses either system rather than no preference; signed preference direction distinguishes which system is chosen.Source: §2.2 · p. 2
- Preference analyses relate decisiveness to discrimination and preference direction to metric differences, modeling direction with a bidirectional logistic function.The signed preference score is the difference between choices for systems A and B, with no-preference responses contributing zero.Source: §2.2 · p. 2
3. EXPERIMENTAL SETUP
The study evaluates four speech-enhancement systems on 45 DNS2020 test utterances, comparing system pairs and clean references. Participants complete discrimination and preference trials, with the figure summarizing triangle-test discrimination across pairs.
3. EXPERIMENTAL SETUP
- Four systems are evaluated on 45 randomly sampled DNS2020 test utterances, using VoiceBank+DEMAND-trained models and publicly available checkpoints.The systems are MP-SEnet 2, MambAttention 3, MeanFlowSE 4, and StoRM 5.Source: §3 · p. 2
- The incomplete pairwise design compares every pair among the four systems and DNS clean references, yielding 10 system-pairs and 135 utterance/system-pair conditions.Each utterance is truncated to four seconds and model outputs are loudness-normalized.Source: §3 · pp. 2–3
- Triangle-test discrimination is plotted by system pair, with open dots for utterance/pair conditions, filled dots for across-utterance averages, and bootstrap 95% confidence intervals.The figure caption identifies the plotted estimates and intervals.Source: Figure 1 · p. 3
- Ninety-two volunteers completed 30 trials in each test block, with headphone screening and randomized, balanced presentation schedules.Each participant evaluated one utterance/pair condition per trial; schedules aimed to reduce familiarization and balance order effects.Source: §3 · p. 3
4. RESULTS
Triangle-test discrimination was above chance but varied across utterances and system pairs; preference decisiveness and direction also varied. Objective metric correlations with perceptual responses were inconsistent, and the incomplete pairwise design limits inter-pair comparisons.
4.1. Triangle Test
- Triangle-test discrimination averaged pD=1775/2541≈0.7, significantly above chance; StoRM–clean was highest (0.83), while MambAttention–MeanFlow was lowest (0.56).Source: §4.1 · p. 3
- Mean utterance-level discrimination across system pairs ranged from 0.34 to 1.0, with substantial variation across utterances.The figure shows utterance-level estimates and their averages across system pairs.Source: §4.1 · p. 3, Figure 2 · p. 3
- TorchAudio-Squim STOI correlated most with discrimination (ρS=0.62), while Score-Q (0.59), DistillMOS (0.51), and PESQ (0.05) showed varying correlations.Source: Table 1 · p. 3, §4.1 · p. 3
4.2. Preference and Decisiveness
- Listeners preferred either system in 68% of trials, while decisiveness varied by pair: MF–clean was highest (0.84), followed by StoRM–clean (0.79), and MF–StoRM was lowest (0.51).Source: §4.2 · pp. 3–4, Figure 4 · p. 4
- MF, Sto, and MP were directionally preferred over MA and clean references, but no clear preference separated MF, Sto, and MP; MambAttention was equally preferred to clean references.High decisiveness did not necessarily correspond to a consistent preference direction; Score-Q and TorchAudio-Squim correlated most with decisiveness and preference, while PESQ, SI-SDR, and DNSMOS correlated more weakly.Source: §4.2 · p. 4, Figure 5 · p. 4
4.3. Limitations
- Unmeasured crowdsourcing factors and mismatched acoustic conditions may contribute to variation, and the incomplete pairwise design limits inter-pair comparability.The authors advise comparing utterance-wise averages descriptively.Source: §4.3 · p. 4
5. CONCLUSION
The results show that similar objective scores can still accompany perceptually distinguishable systems, and that discriminability does not necessarily imply systematic listener preference. Most objective metrics correlated poorly with both dimensions; the authors call for controlled listening studies and further modeling.
5. CONCLUSION
- Systems with similar objective scores can remain perceptually distinguishable, and discrimination does not necessarily indicate systematic listener preference.The authors frame these as distinct dimensions of speech-enhancement evaluation.Source: §5 · p. 4
- Most objective metrics showed poor correlation with both discriminability and preference.Source: §5 · p. 4
- Future work should substantiate the findings with controlled laboratory listening, more repeated judgments, and direct comparison with ITU listening-test standards.Source: §5 · p. 4
- Probabilistic choice models could account for listener and stimulus variation and support model comparison and transitivity.Source: §5 · p. 4