Source-linked AI summary
Perceptual Quality Loss or Loss of Perceptual Quality?
Danilo de Oliveira, Tal Peer, Maurício do V. M. da Costa, Timo Gerkmann
TL;DR
Speech-enhancement models may score better on a metric without improving perceived quality, complicating evaluation and comparison. The paper tests two PESQ-loss approaches with instrumental and listening evaluations, finding listener preference for models without PESQ loss and PESQ dominance in composite metrics.
Problem
Because perceptual metrics are imperfect proxies, optimizing a metric may raise its score without establishing comparable gains or preserving performance on other metrics.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The findings support evaluating speech enhancement with a comprehensive panel of metrics and datasets, ideally including ratings from a significantly large human-subject pool.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Contemporary deep speech enhancement (SE) models are often trained with specific auxiliary terms in the loss function as a way to improve their performance in terms of perceptual metrics. Nevertheless, a higher score on a perceptual metric does not necessarily correlate with an improved listening experience. Through objective and subjective experiments, we assess the performance of SE models trained with two different types of auxiliary PESQ loss terms. The numerical evaluation on a suite of standard metrics suggests that, while models optimized for PESQ naturally obtain higher PESQ scores in the test set, for most other metrics the scores do not significantly change. In some cases, the PESQ loss even results in worse PESQ scores on mismatched data. A formal listening experiment reveals that the models without a PESQ loss were generally preferred over models that include it, across all settings. Finally, we analyze the relative importance of PESQ in the composite metrics CSIG, CBAK and COVL, and find that PESQ dominates all of them. Our study highlights the perils of over-reliance on PESQ and stresses the importance of a complete evaluation procedure for SE.
1. INTRODUCTION
Speech-quality metrics are imperfect proxies for subjective perception, and optimizing an evaluation metric can make model comparisons less direct. The paper investigates two auxiliary PESQ-loss approaches using instrumental metrics and a formal listening experiment.
1. INTRODUCTION
- Prior work found that high PESQ scores need not correspond to high perceived quality, including when distortions are purposefully induced.Source: §1 · p. 1
- Metric losses can raise scores on their target metric without establishing comparable gains, and imperfect perceptual modeling may harm other metrics.The introduction frames this risk even when metric losses are combined with regular distance-based losses.Source: §1 · p. 1
- The study evaluates two representative PESQ-loss versions using a suite of instrumental metrics and a formal listening experiment.It also analyzes metrics included in a widely used composite-measure evaluation script.Source: §1 · p. 1
2. PESQ-BASED LOSS FUNCTIONS
PESQ is a reference-based speech-quality metric used in model optimization. The section describes a differentiable PESQ loss and a GAN-style surrogate approach, including how they are incorporated into the evaluated systems.
2. PESQ-BASED LOSS FUNCTIONS
- PESQ is a reference-based objective speech-quality metric that maps perceived distortions to a mean-opinion-score scale from 1 to 5.Higher scores represent better perceptual quality.Source: §2 · p. 1
2.1. Differentiable PESQ Loss
- The differentiable implementation skips time alignment and performs level alignment via IIR filtering; SB-SGMSE+ evaluates PESQ-loss weights of 0, 0.0005, and 0.001.These weights correspond to no PESQ, PESQ-mid, and PESQ-high, respectively.Source: §2.1 · p. 2
2.2. GAN Metric Loss
- The GAN-style loss trains a discriminator to predict rescaled PESQ from clean and enhanced magnitude spectrograms, while its generator loss encourages higher discriminator scores.SEMamba is evaluated with and without the GAN metric loss alongside its other loss terms.Source: §2.2 · p. 2
3. EXPERIMENTS
The experiments evaluate speech-enhancement models across instrumental metrics, matched and mismatched datasets, and a blind listening test. They also examine how PESQ contributes to three composite metrics.
3.1. Instrumental Metrics
- The instrumental suite spans composite quality scores, intelligibility, signal-to-noise and distortion measures, objective quality estimates, and ASR word accuracy.CSIG, CBAK, and COVL combine instrumental measures; WAcc is 1 − WER, while ESTOI measures speech intelligibility and POLQA succeeds PESQ.Source: §3.1 · p. 2
- WAcc measures ASR performance on enhanced audio, while SCOREQ and DistillMOS provide learned speech-quality estimates.SCOREQ has reference-based and reference-free modes; DistillMOS is trained using human-labeled and teacher-pseudo-labeled data.Source: §3.1 · pp. 2–3
- The evaluation compares SB-SGMSE+ and SEMamba on matched VB-DMD and mismatched EARS-WHAM using instrumental metrics.The table captions identify these model and dataset comparisons but do not provide the table values.Source: Table 1 · p. 3, Table 2 · p. 3
3.2. Data
- VB-DMD supplies the matched test set, while EARS-WHAM provides a more challenging mismatched test set; both are down-sampled to 16 kHz.The sets contain 824 and 886 samples, respectively.Source: §3.2 · p. 3
3.3. Listening Experiment
- The blind preference test compares SEMamba with and without LMetric and SB-SGMSE+ models M5, M6, and M7 in randomized pairwise comparisons.Thirty samples per test set were selected, and stimuli were stratified by input SNR; VB-DMD files under three seconds were filtered, while EARS-WHAM segments were cropped to about six seconds.Source: §3.3 · p. 3
- Twelve audio experts rated speech pairs in a sound-proofed booth, with each stimulus evaluated by three participants.Each expert rated 56 pairs, including eight screening tests.Source: §3.3 · pp. 3–4
3.4. Composite Metric Analysis
- The composite-metric analysis estimates PESQ’s share of variance in CSIG, CBAK, and COVL across noisy and enhanced benchmark data.The component measures are PESQ, log-likelihood ratio, WSS distance, and segSNR.Source: §3.4 · p. 4
4. RESULTS
Listeners generally preferred models trained without PESQ loss, while instrumental results varied by model and test condition. PESQ also dominated the variance in the composite measures.
4. RESULTS
- For SB-SGMSE+, M6 (PESQ-high) led in PESQ, CSIG, and COVL; M7 (PESQ-mid) led in POLQA and CBAK; M5 (no PESQ) led in segSNR, SDR, and ESTOI.SEMamba results were generally similar across matched metrics except PESQ and composite measures, while most mismatched metrics favored the model without metric loss, including PESQ and composites. The authors attribute this to weak generalization by the discriminator trained on VB-DMD, whose size and SNR range were limited.Source: §4 · p. 4
- Approximately 15 percentage points separated no-PESQ models from their alternatives in mismatched listening tests; SB-SGMSE+ rankings were 0.534 for no-PESQ, 0.371 for PESQ-mid, and 0.095 for PESQ-high.Across settings, listeners significantly preferred models optimized without PESQ loss terms.Source: Figure 3 · p. 4, §4 · p. 4
- PESQ dominated variance in CSIG, CBAK, and COVL; WSS accounted for less than 1% in each, and ESTOI had low variance.The table reports effective importance of individual metrics across models and both datasets.Source: §4 · p. 4, Table 3 · p. 4
5. CONCLUSION
The paper compares two representative speech-enhancement models trained with and without PESQ loss. Instrumental and listening results show risks from PESQ-focused optimization and support comprehensive evaluation.
5. CONCLUSION
- The study analyzes two representative SE models with and without PESQ loss, considering both differentiable PESQ and a PESQ-surrogate GAN-style discriminator.Source: §5 · p. 4
- Instrumental metrics showed no consensus beyond PESQ and composite measures; the GAN-style loss was detrimental to PESQ in mismatched testing, and listeners preferred models without PESQ loss.The listening preference was more pronounced in the mismatched setting.Source: §5 · p. 4
- The study highlights risks of perceptual losses in speech enhancement and the importance of a complete evaluation procedure.Source: §5 · p. 4