Source-linked AI summary
The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results
Chandan K. A. Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, Puneet Rana, Sriram Srinivasan, Johannes Gehrke
TL;DR
Speech-enhancement research lacks scalable subjective evaluation and representative real-world testing because objective metrics and synthetic test sets have important limitations. The DNS Challenge addresses this gap by releasing large training data, mixed synthetic and real test sets, and an online P.808 framework, then evaluates submissions on a blind test set. The challenge also imposes a real-time computational track with a 40 ms look-ahead limit.
Problem
Objective speech-quality metrics often correlate poorly with subjective tests, while synthetic evaluation may not predict performance on diverse real recordings and lab tests do not scale well.
Method
The challenge releases large clean-speech and noise datasets, synthetic and real test sets, an online ITU-T P.808 framework, and an NSNet baseline for speech-enhancement research.
Results
28 submissions from 19 teams were evaluated subjectively in two phases, with blind-test comparisons using P.808 and ANOVA-based significance testing.
Takeaways & Limitations
The challenge provides common datasets and subjective evaluation infrastructure intended to support more comparable speech-enhancement assessments across publications.
Takeaways & Limitations
The real-time track constrains processing to less than T/2 for frames up to 40 ms, with at most 40 ms of look-ahead.
Abstract
from arXiv · showhide
The INTERSPEECH 2020 Deep Noise Suppression (DNS) Challenge is intended to promote collaborative research in real-time single-channel Speech Enhancement aimed to maximize the subjective (perceptual) quality of the enhanced speech. A typical approach to evaluate the noise suppression methods is to use objective metrics on the test set obtained by splitting the original dataset. While the performance is good on the synthetic test set, often the model performance degrades significantly on real recordings. Also, most of the conventional objective metrics do not correlate well with subjective tests and lab subjective tests are not scalable for a large test set. In this challenge, we open-sourced a large clean speech and noise corpus for training the noise suppression models and a representative test set to real-world scenarios consisting of both synthetic and real recordings. We also open-sourced an online subjective test framework based on ITU-T P.808 for researchers to reliably test their developments. We evaluated the results using P.808 on a blind test set. The results and the key learnings from the challenge are discussed. The datasets and scripts can be found here for quick access https://github.com/microsoft/DNS-Challenge.
1. Introduction
The challenge addresses weaknesses in objective and synthetic-only evaluation by releasing shared datasets, representative test data, and scalable subjective testing tools for speech enhancement.
- Objective metrics such as PESQ, POLQA, ViSQOL, and SDR often correlate poorly with subjective speech-quality tests.
- Synthetic test performance does not ensure robust behavior on real recordings captured under diverse noisy and reverberant conditions.
- The DNS Challenge open-sources large clean-speech and noise datasets, configurable synthesis scripts, and development and blind test sets.
- The challenge also releases an online ITU-T P.808 subjective evaluation framework and an NSNet baseline implementation.
- The paper describes the datasets, evaluation framework, baseline method, participant training data, and challenge results.
2. Datasets
The DNS datasets combine curated clean speech and balanced noise sources with synthetic and realistic test recordings designed to reflect deployment conditions.
- 2.1. Clean Speech: The clean-speech corpus contains approximately 441 hours from 2,150 speakers selected from Librivox clips with MOS values from 4.3 to 5.
- 2.2. Noise Dataset: Noise data are balanced across about 150 AudioSet classes, contain 60,000 clips, and add 10,000 clips from Freesound and DEMAND.
- 2.4.1. Development stage test set: The development test set combines synthetic non-reverberant, synthetic reverberant, and real recordings, with 300 clips in each of four categories.
- 2.4.2. Blind test set: The blind test set contains 600 clips, split evenly between synthetic and real recordings, using unseen clean speech, noise, and room impulse responses.
- 2.4.1. Development stage test set: Real recordings were collected through MTurk across varied noisy acoustic conditions and recording devices to provide realistic noisy speech.
3. Baseline SE method
The baseline NSNet is an efficient recurrent speech-enhancement model released with inference code and an ONNX model for challenge comparison.
- NSNet uses log power spectra to predict frame-level enhancement gains with GRU and fully connected layers.
- NSNet processes a 20 ms frame in 0.16 ms on an Intel quad-core i5 using ONNX Runtime.
- The challenge releases NSNet's inference script and ONNX-format model in the DNS-Challenge repository.
4. Online Subjective Evaluation Framework ITU-T P.808
The evaluation framework applies ITU-T P.808 crowdsourced listening tests to estimate MOS while qualifying and filtering raters for scalable quality assessment.
- The framework uses Absolute Category Ratings under ITU-T P.808 to estimate Mean Opinion Scores for comparing speech-enhancement methods.
- Raters complete hearing and environment qualification before rating clips, and immediate access after qualification increases rating speed by approximately 5X.
- Clip groups include gold clips and trapping questions to filter raters who provide unreliable responses.
5. DNS Challenge Tracks
The DNS Challenge separates participating speech-enhancement methods into real-time and non-real-time tracks, while limiting future context to 40 ms. Winners are selected using subjective speech quality on a blind test set.
- Track 1: Track 1 targets real-time applications with processing below T/2 milliseconds for a frame of size T.The limit is measured on an Intel Core i5 quad-core processor at 2.4 GHz or equivalent.
- Track 2: Track 2 imposes no computational-time constraint, allowing researchers to explore deeper models for speech quality.
- Shared constraint: Both tracks allow a maximum of 40 ms look-ahead when inferring the current frame.Algorithms may access any number of past frames but only 40 ms of future frames.
- Evaluation: Three winners per track are selected from the blind test set using subjective speech quality evaluated with ITU-T P.808.
6. Datasets used and Challenge Results
Participants used the DNS data alongside additional augmentation strategies, and the challenge evaluated a broad range of models through staged P.808 subjective testing. Phase 2 combined ratings and ANOVA to establish statistically significant rankings among leading systems.
- Datasets and augmentation: Many teams augmented DNS data with additional clean speech, noise, and reverberation, while attempting joint dereverberation and noise suppression.Participants were free to use other datasets and modify the configurable synthesis scripts.
- Phase 1: 28 submissions from 19 teams were evaluated in phase 1 using P.808, with 10 raters per clip and a 95% confidence interval of 0.02.The evaluation included submissions and the noisy test set across synthetic, reverberant-synthetic, and real-recording conditions.
- Phase 1: The wide span of dMOS values reflects substantial variation among participant models.dMOS is the difference between enhanced-speech MOS and the noisy blind-set MOS before enhancement.
- Phase 2: Phase 2 used 10 additional raters for leading teams, combined 20 ratings per clip, and achieved a 95% confidence interval of 0.01 per model.The top 3 real-time teams and top 4 non-real-time teams were evaluated.
- Ranking: ANOVA used a 0.05 p-value threshold, and all real-time teams showed statistically significant speech-quality differences.An overlap for second place in the non-real-time track was resolved using computational complexity.
7. Conclusion
The DNS Challenge promotes real-time single-microphone noise suppression evaluated by subjective speech quality. Its open datasets and P.808 tool were useful to participants, while future work targets shared evaluation, separate speech and noise quality, personalization, and no-reference MOS prediction.
- The challenge promotes real-time single-microphone noise suppression for exceptional subjective speech quality.
- Participants found the open DNS datasets and P.808 subjective evaluation tool useful.
- Future work includes evaluating models on a common test set to make comparisons across publications easier.
- Planned directions include P.835-based separate speech and noise assessment, personalized noise suppression, and a no-reference MOS predictor.