Source-linked AI summary
DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors
Chandan K A Reddy, Vishak Gopal, Ross Cutler
TL;DR
Speech-quality evaluation needs scalable non-intrusive alternatives because subjective tests are costly and intrusive metrics require clean references. This paper trains DNSMOS P.835 to predict speech, background-noise, and overall quality, reporting strong generalization and accurate noise-suppressor ranking on an unseen test set.
Problem
Subjective evaluation is costly and difficult to scale, while existing objective metrics can correlate poorly with human ratings or require unavailable clean references.
Method
The authors train a CNN-based non-intrusive metric on ITU-T P.835 ratings to predict speech quality (SIG), background-noise quality (BAK), and overall quality (OVRL).
Results
DNSMOS P.835 generalizes well on an unseen test set and shows excellent model-level correlation with human ratings, though SIG remains an area for improvement.
Takeaways & Limitations
DNSMOS P.835 provides an accurate metric for ranking noise suppressors by estimated P.835 mean opinion scores.
Takeaways & Limitations
Per-clip performance is limited by using only 5 ratings per clip because of cost restrictions.
Abstract
from arXiv · showhide
Human subjective evaluation is the gold standard to evaluate speech quality optimized for human perception. Perceptual objective metrics serve as a proxy for subjective scores. We have recently developed a non-intrusive speech quality metric called Deep Noise Suppression Mean Opinion Score (DNSMOS) using the scores from ITU-T Rec. P.808 subjective evaluation. The P.808 scores reflect the overall quality of the audio clip. ITU-T Rec. P.835 subjective evaluation framework gives the standalone quality scores of speech and background noise in addition to the overall quality. In this work, we train an objective metric based on P.835 human ratings that outputs 3 scores: i) speech quality (SIG), ii) background noise quality (BAK), and iii) the overall quality (OVRL) of the audio. The developed metric is highly correlated with human ratings, with a Pearson's Correlation Coefficient (PCC)=0.94 for SIG and PCC=0.98 for BAK and OVRL. This is the first non-intrusive P.835 predictor we are aware of. DNSMOS P.835 is made publicly available as an Azure service.
1. INTRODUCTION
Subjective speech-quality evaluation is the most reliable approach but is difficult to scale because it is laborious, time-consuming, expensive, and listener-intensive. Conventional objective metrics such as PESQ correlate poorly with human ratings for some speech-enhancement tasks, while intrusive metrics require an unavailable clean reference in realistic recordings.
- Subjective evaluation is the most reliable way to assess speech-enhancement methods, but it requires many listeners and is laborious, time-consuming, and expensive.
- Conventional objective metrics such as PESQ correlate poorly with human ratings for speech-enhancement tasks involving perceptually invariant transformations.
- Intrusive metrics cannot evaluate real recordings in realistic scenarios when a clean reference is unavailable.
2. RELATED WORK
Prior work established P.835’s three-dimensional subjective framework and advanced non-intrusive speech-quality assessment with DNNs, while DNSMOS previously predicted only overall quality. This paper extends DNSMOS to predict SIG, BAK, and OVRL using P.835 labels and diverse noise-suppressed audio.
- P.835 Framework: ITU-T P.835 provides separate subjective ratings for speech quality (SIG), background noise quality (BAK), and overall quality (OVRL).Hu and Loizou showed that OVRL can be accurately modeled linearly from SIG and BAK, while Naderi and Cutler used this relationship to analyze DNS Challenge 3.
- Novelty: DNSMOS P.835 is presented as the first P.835-based non-intrusive speech-quality assessment model the authors are aware of.Table 1 compares common DNN-based non-intrusive speech-quality assessment methods; ACR denotes Absolute Category Rating.
- DNSMOS: DNSMOS has been used for noise-suppression training, model selection, and ablation studies after demonstrating greater robustness and reliability than some popular intrusive metrics.The metric previously predicted only the overall audio score and was adopted by over a hundred researchers within several months of release.
- Contribution: The paper extends DNSMOS from overall audio quality to speech quality (SIG), background noise quality (BAK), and overall quality (OVRL).Training uses subjective labels from ITU-T P.835 collected for DNS Challenge 3 and internally processed noisy clips.
3. DATA AND SUBJECTIVE RATINGS
The study trains DNSMOS P.835 on DNS Challenge V3 data containing diverse noisy speech processed by roughly 40 suppression models and about 30,000 clips with P.835 MOS ground truth. The dataset is unusually large and artifact-diverse, though its ratings are skewed toward mid-to-high quality, especially for background noise.
- 3. DATA AND SUBJECTIVE RATINGS: 600 noisy speech clips processed by about 40 suppression models span over 100 noise types and speakers across varied SNRs and target levels.The clips come from the DNS Challenge V3 test set, with real recordings covering diverse acoustic conditions.
- 3. DATA AND SUBJECTIVE RATINGS: Ratings span MOS=1 to MOS=5, with SIG and OVRL concentrated mostly between 3 and 4, while BAK is strongly skewed above 4.The training-score distributions contain fewer ratings in both tails for SIG and OVRL, unlike the high-quality bias for BAK.
- 3. DATA AND SUBJECTIVE RATINGS: About 30,000 audio clips received MOS ground-truth ratings across repeated P.835 runs comparing suppression methods with anchors.Each run included the best-performing suppressor, original noisy speech, and intermediate-quality methods; some clips were rated multiple times.
- 3. DATA AND SUBJECTIVE RATINGS: The dataset is described as the largest known P.835 dataset and the only one used to train a DNN non-intrusive speech-quality model.Its roughly 40 deep noise suppression models provide varied suppression artifacts intended to support generalization across noise suppressors.
4. DNSMOS P.835
DNSMOS P.835 uses log power spectrograms rather than raw waveforms to represent 9-second, 16-kHz speech clips for MOS prediction. The method explores CNN-based architectures, including a best-performing configuration described in Table 2.
- Model architecture: CNN-based configurations are explored for MOS prediction, with the best-performing architecture presented in Table 2.The supplied passage does not include the architecture table's layer details.
- Model input: The CNN input is a 900 x 161 log power spectrogram computed with a 320 FFT, 20 ms frame size, and 10 ms hop length over each 9-second clip.Two models with nearly identical architectures were trained, differing in their last layer; the passage ends before specifying the layer functions.
5. EXPERIMENTAL RESULTS
DNSMOS P.835 was evaluated on an unseen P.835 test set spanning diverse speech and noise categories, showing strong generalization and excellent model-level agreement with human ratings. Clip-level performance was also good, although lower than model-level performance, and SIG remained an area for improvement.
- The unseen test set contains P.835 evaluations of 17 Microsoft internal noise-suppression models across 850 clips covering emotional, English, non-English, tonal-language, stationary-noise, and mouse-click categories.The test set was created for a future DNS challenge and has categories similar to the training data, with improved emotional speech quality.
- For noise-suppressor stack ranking, the evaluation computes SRCC and PCC between ratings averaged across the test set for each model and the corresponding averaged DNSMOS predictions.PCC and MSE are commonly used to assess objective-metric accuracy, while aggregation by model improves the repeatability of P.835 comparisons.
- DNSMOS P.835 shows excellent model-level correlation with human ratings on the unseen test set, with SIG still identified as an area for improvement.The results indicate good generalization across the evaluated noise categories and environments; comparison with other metrics was not possible because this was the first non-intrusive P.835 speech-quality metric reported by the authors.
- DNSMOS P.835 also achieves good clip-level correlation with human ratings, though performance is lower than when ratings are aggregated at the model level.The clip-level evaluation used 30 ratings per clip, compared with 5 ratings per clip for the per-model correlation, to improve the accuracy of human ratings.
6. CONCLUSION AND FUTURE WORK
DNSMOS P.835 accurately stack-ranks noise suppressors and achieves strong performance through data, task, architecture, training, and aggregation choices. Per-clip performance could improve by increasing ratings beyond the current five per clip.
- Conclusion: DNSMOS P.835 accurately stack-ranks noise suppressors with high accuracy.Its performance is attributed to a large high-quality dataset, a limited speech-quality impairment category, model and training optimizations, and aggregation by noise-suppression model.
- Future Work: Increasing the number of ratings per clip beyond the current five could improve per-clip performance.The current per-clip rating count is only 5.