Source-linked AI summary
DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors
Chandan K A Reddy, Vishak Gopal, Ross Cutler
TL;DR
Speech-quality evaluation needs a scalable alternative to subjective testing and a reference-free metric for recordings without clean speech. DNSMOS combines a CNN trained on P.808 human ratings with multi-stage self-teaching to handle noisy labels, and it correlates better with MOS than PESQ, POLQA, and SDR while generalizing to challenging conditions.
Problem
Subjective testing is costly and unscalable, while intrusive metrics require unavailable clean references and existing non-intrusive approaches correlate poorly with human ratings.
Method
DNSMOS is a CNN-based perceptual speech-quality metric trained on P.808 human ratings using multi-stage self-teaching for noisy labels.
Results
DNSMOS correlates significantly better with MOS than SDR, PESQ, and POLQA and generalizes to emotional speech, singing, and non-English languages absent from training.
Takeaways & Limitations
DNSMOS is designed to accurately stack-rank noise suppressors and is reported to generalize across varied speech impairments.
Abstract
from arXiv · showhide
Human subjective evaluation is the gold standard to evaluate speech quality optimized for human perception. Perceptual objective metrics serve as a proxy for subjective scores. The conventional and widely used metrics require a reference clean speech signal, which is unavailable in real recordings. The no-reference approaches correlate poorly with human ratings and are not widely adopted in the research community. One of the biggest use cases of these perceptual objective metrics is to evaluate noise suppression algorithms. This paper introduces a multi-stage self-teaching based perceptual objective metric that is designed to evaluate noise suppressors. The proposed method generalizes well in challenging test conditions with a high correlation to human ratings.
1. INTRODUCTION
Existing speech-quality evaluation is either expensive and unscalable when subjective, dependent on unavailable clean references when intrusive, or poorly aligned with human ratings in no-reference settings. DNSMOS addresses this gap with a CNN trained on P.808 human ratings and multi-stage self-teaching for noisy labels.
- Subjective speech-quality tests are reliable but laborious, time-consuming, expensive, and difficult to scale.
- Intrusive metrics such as PESQ, POLQA, and SDR require clean reference speech, which is unavailable for realistic recordings.
- P.563 operates without a reference signal but was developed for narrow-band applications and limited impairment types.
- DNSMOS estimates MOS for ranking Deep Noise Suppression methods using a CNN trained on P.808 human ratings.
- Multi-stage self-teaching is used to learn from noisy human-rating labels and improve generalization across speech impairments.
2. DATA AND SUBJECTIVE RATINGS
The study builds subjective-rating data from diverse noisy and processed speech clips evaluated through repeated P.808 runs. The resulting ratings span a broad quality range but contain run-to-run variation because human raters are inconsistent.
- The training data used 600 synthetic and real noisy speech clips spanning over 100 noise types and speakers.
- More than 200 noise suppression methods processed the clips, producing MOS ratings from 1, very poor, to 5, excellent.
- The MOS distribution was skewed toward 3<MOS<4, with fewer ratings in both tails.
- Several P.808 runs produced about 120,000 nine-second audio clips with associated ground-truth MOS scores.
- Absolute scores varied between P.808 runs because raters differed and humans were inconsistent in perceptual tasks.
3. DNSMOS
DNSMOS combines a CNN speech-quality predictor with multi-stage self-teaching to address noisy and biased MOS labels. The model uses log-power Mel spectrograms and empirically improves accuracy through weighted teacher predictions.
- 3.1. Learning in the presence of label biases: Rating variation across P.808 runs and clips with few ratings create noisy labels that complicate direct MOS prediction.
- 3.1. Learning in the presence of label biases: Self-teaching improves predictor generalization by training successive models for the regression problem with MSE loss.
- 3.1. Learning in the presence of label biases: The student uses weighted predictions from teacher models together with original human MOS ratings, while M0 is trained on the original ratings.
- 3.3. Prediction model: The models use 120-band log-power Mel spectrograms from nine-second, 16-kHz clips, producing 900 x 120 inputs for CNN prediction.
- 3.3. Prediction model: Batch normalization was avoided because it reduced accuracy on low-volume clips, whose amplitudes humans tend to rate lower.
4. EXPERIMENTAL RESULTS
The evaluation measures how well DNSMOS tracks human ratings and ranks noise suppressors, comparing it with established objective metrics. DNSMOS correlates better with MOS and improves generalization across challenging speech categories through multi-stage self-teaching.
- 4.1. Evaluation metric: SRCC and PCC are computed between per-model averaged human ratings and averaged DNSMOS scores, with SRCC measuring stack-ranking accuracy.Averaging ratings across each model’s entire test set supports ranking noise suppression systems.
- 4.2.1. DNSMOS vs other objective speech quality metrics: DNSMOS (M0) correlates significantly better with MOS than SDR, PESQ, and POLQA on the Interspeech DNS challenge blind test.The scatter plots also show closer alignment between DNSMOS and MOS than the competing metrics.
- 4.2.1. DNSMOS vs other objective speech quality metrics: PESQ and POLQA form a left-top cluster because they penalize some artifacts more than human listeners do.This mismatch explains why their scores can diverge from human perceptual judgments on the blind test.
- 4.2.2. Generalizability of DNSMOS: SRCC between MOS and M1 was higher for α0 = 0.8 than for the other tested α0 values in one P.808 run.The run contained 12 closely stack-ranked models and examined the weighting of teacher model M0 versus human ratings.
- 4.2.2. Generalizability of DNSMOS: M1 increased SRCC from 0.65 to 0.9 for α0 = 0.8 on ICASSP 2021 DNS Challenge 2 results.The models were evaluated on emotional speech, singing, and English and non-English languages absent or underrepresented in training; optimal α0 varied by category.
5. CONCLUSION AND FUTURE WORK
DNSMOS is presented as a robust metric for accurately stack-ranking noise suppressors, with multi-stage self-teaching improving accuracy. Future work will study αi selection to extend training to additional impairment types without large human-labeled datasets.
- Conclusion: DNSMOS is designed to accurately stack-rank noise suppressors, and multi-stage self-teaching significantly improves its accuracy.The conclusion identifies network distortions, codec artifacts, and reverberation as future application areas.
- Future work: Future work will deepen understanding of how αi affects Mi accuracy across datasets and support training on new impairments without large-scale human labels.The proposed target impairments include network distortions, codec artifacts, and reverberation.