Source-linked AI summary
A scalable noisy speech dataset and online subjective test framework
Chandan K. A. Reddy, Ebrahim Beyrami, Jamie Pool, Ross Cutler, Sriram Srinivasan, Johannes Gehrke
TL;DR
Speech-enhancement research lacks broadly covering, scalable noisy-speech datasets and large-scale subjective evaluations. The paper introduces MS-SNSD and a crowdsourced online MOS framework, then applies them to noise suppressors and objective metrics. It reports that larger datasets improve noise-suppression performance and that MOS remains necessary because objective metrics correlate insufficiently with subjective quality.
Problem
Existing speech-enhancement datasets are relatively small, while offline subjective evaluation is expensive and time-consuming, limiting coverage and evaluation scale.
Method
The paper introduces scalable MS-SNSD and an open-source crowdsourced online subjective-evaluation framework, then retrains and evaluates multiple suppressors.
Results
Larger training datasets improve RNNoise performance, while objective metrics correlate insufficiently with MOS; POLQA reaches the highest reported correlation at 0.78.
Takeaways & Limitations
Scalable data generation and online subjective testing support broader speech-enhancement comparisons, with MOS remaining required for reliable quality assessment.
Takeaways & Limitations
Test noise clips came from recordings different from training data but remained within similar noise categories.
Abstract
from arXiv · showhide
Background noise is a major source of quality impairments in Voice over Internet Protocol (VoIP) and Public Switched Telephone Network (PSTN) calls. Recent work shows the efficacy of deep learning for noise suppression, but the datasets have been relatively small compared to those used in other domains (e.g., ImageNet) and the associated evaluations have been more focused. In order to better facilitate deep learning research in Speech Enhancement, we present a noisy speech dataset (MS-SNSD) that can scale to arbitrary sizes depending on the number of speakers, noise types, and Speech to Noise Ratio (SNR) levels desired. We show that increasing dataset sizes increases noise suppression performance as expected. In addition, we provide an open-source evaluation methodology to evaluate the results subjectively at scale using crowdsourcing, with a reference algorithm to normalize the results. To demonstrate the dataset and evaluation framework we apply it to several noise suppressors and compare the subjective Mean Opinion Score (MOS) with objective quality measures such as SNR, PESQ, POLQA, and VISQOL and show why MOS is still required. Our subjective MOS evaluation is the first large scale evaluation of Speech Enhancement algorithms that we are aware of.
1. Introduction
The paper addresses limited coverage and non-scalability in noisy-speech datasets and subjective evaluations by introducing MS-SNSD and an open-source crowdsourced framework. It uses these resources to compare noise suppressors, dataset-size effects, and MOS against objective metrics.
- Motivation: Existing DNN speech-enhancement datasets are relatively small, limiting coverage of diverse noisy speech conditions.The paper contrasts datasets such as 23K clips with ImageNet-scale data and links limited coverage to poorer performance on unseen conditions.
- Motivation: Scalable datasets are needed to vary speakers, noise types, and SNR levels while accommodating different data requirements across DNN techniques.The paper notes that more training data improves objective performance and that fixed-size datasets may not meet every method’s threshold.
- Contributions: The paper introduces MS-SNSD, an open-source noisy-speech dataset whose size can grow through augmentation across noise clips and SNR levels.With 14 noise sources and 5 SNR levels, augmentation increases dataset size by 70 times relative to using neither factor.
- Contributions: It provides an open-source online subjective-evaluation framework using crowdsourcing and references for large-scale comparison of speech-enhancement methods.The framework is designed to address the expense and time required by conventional offline subjective tests.
- Contributions: The study compares four noise suppressors with MOS, analyzes performance by noise type, and examines why MOS remains necessary alongside objective metrics.The contributions include dataset-size analysis for RNNoise and correlations between MOS and SNR, PESQ, POLQA, and VISQOL.
2. Dataset
The dataset is designed to scale across speakers, noise types, and SNR levels while supporting flexible clip configuration and organization for different DNN-based speech-enhancement needs.
- MS-SNSD scales with the number of speakers, noise types, and SNR levels, and can accommodate new noisy conditions.
- Users can set audio-clip lengths and sampling rates, and segregate clips by noise type and SNR level.
1 Source code and data is available at http://aka.ms/ms-snsd
The study combines established clean-speech corpora with selected environmental noises to generate large, configurable training and test sets using randomized mixing and SNR variation.
- The training data use Edinburgh’s 56-speaker clean-speech corpus, containing over 23K clips resampled to 16 kHz.The speakers comprise 28 male and 28 female participants, and the median sentence length is 2.6 seconds.
- Testing uses 4,720 sentences from 20 speakers, concatenated into clips averaging 10 seconds.
- Fourteen hand-selected noise types from DEMAND and Freesound.org provide application-relevant environmental conditions that can be extended with new categories.
- Test noises are different recordings from training noises but come from similar categories.
- 50,000 noisy clips totaling 42 hours train the DNN models, while 5,500 test clips totaling 16 hours support evaluation.Clean speech and noise segments are randomly selected and mixed at controlled SNR levels; all files use 16 kHz sampling and are normalized to -25 dBFS.
3. Online subjective evaluation framework
The evaluation framework uses crowdsourced online listening tests to collect subjective speech-quality ratings, with training, qualification, and reference-based normalization procedures.
- Microsoft UHRS provides the online platform for subjective experiments, with a JavaScript Hit Application controlling the user interface and other parameters.Mechanical Turk can also be used, and source code is provided.
- The Hit Application presents audio clips that judges listen to and rate according to perceived speech quality.
- Judges must read guidelines, complete training, and pass qualification before entering the main experiments.Training includes five clips spanning clean and noisy speech; qualification uses clean and extremely noisy examples associated with five-star and one-star ratings.
- Future speech-enhancement MOS results are linearly scaled to match reference Noisy and Wiener Filter values.The authors provide a normalization utility and the full Wiener Filter source code.
4. Overview of the SE methods compared
The comparison includes a Wiener Filter baseline and three neural speech-enhancement methods: RNNoise, WaveNet, and SEGAN, each representing a different enhancement strategy.
- Wiener Filter: The Wiener Filter minimizes mean squared error between estimated and original clean-speech magnitude spectra.
- Wiener Filter: Its optimal filter uses estimated clean-speech and noise power spectral densities, while enhanced speech is obtained by filtering the noisy spectrum.The reconstructed time series uses the noisy phase; noise power spectra are estimated during noise-only frames using a VAD.
- RNNoise: RNNoise uses recurrent neural networks to estimate suppression gains in critical bands rather than raw PCM samples.The implementation is adapted from 48 kHz to 16 kHz wideband speech and limits attenuation to approximately 15 dB.
- WaveNet: WaveNet performs end-to-end denoising with non-causal convolutions whose dilation factors increase exponentially.The supervised model predicts target fields while minimizing an energy-conserving loss.
- SEGAN: SEGAN maps noisy speech to clean speech with a generator and uses a discriminator to distinguish clean from enhanced inputs.Training is formulated as a minmax game, with the discriminator functioning as a trainable loss.
5. Experimental results
The experiments compare four speech-enhancement methods on large-scale subjective tests and examine how noise type, objective metrics, and training-set size affect evaluation. RNNoise performs best overall, while objective metrics correlate imperfectly with MOS.
- Performance analysis: RNNoise clearly outperforms the other compared noise suppressors in MOS, while SEGAN’s speech distortion lowers its score and WaveNet shows no statistically significant improvement.The evaluation used 5,500 noisy clips, four methods, 27,500 total ratings, and 10 judges per clip.
- Performance analysis: Announcements have the lowest MOS among the examined noise types, indicating that these commonly encountered noises require special attention.
- Comparison with objective measures: POLQA has the highest correlation with MOS at 0.78, but this correlation remains too low to track enhanced-speech quality accurately.
- Impact of dataset size: Training-size experiments compare RNNoise models trained on 16-hour and 42-hour datasets using an independent test set spanning four noise types and two SNR levels.The reported loss is based on mean squared error for the denoising gain function and a separate voice-activity-detection objective.
6. Conclusions
The paper provides a scalable noisy-speech dataset and an open-source framework for subjective evaluation of speech-enhancement methods using crowdsourcing and reference normalization.
- MS-SNSD supports arbitrarily sized noisy-speech datasets through data augmentation, while the accompanying framework enables subjective evaluation of noise-suppression algorithms.