Source-linked AI summary
Interspeech 2021 Deep Noise Suppression Challenge
Chandan K A Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, Sriram Srinivasan
TL;DR
Real-time speech enhancement remains difficult because background noise harms perceptual speech quality in communication and assistive applications. DNS Challenge 3 expands diverse training and test data, adds wideband and fullband tracks, and introduces P.835-based evaluation with DNSMOS; the best model still reached only about 3.78 Overall MOS, below perfect quality.
Problem
Background noise degrades speech quality and intelligibility, while prior DNS challenges remained about 1.6 DMOS from ideal MOS 5 on realistic test data.
Method
The challenge provides expanded multilingual, emotional, singing, noise, and reverberation datasets, two real-time denoising tracks, P.835 evaluation, and DNSMOS.
Results
About 3.78 Overall MOS was achieved by the best model, approximately 1.2 DMOS below perfect MOS 5.
Takeaways & Limitations
The datasets and tools support research on real-time speech enhancement, while challenge results show that superior noise suppression without speech distortion remains difficult.
Abstract
from arXiv · showhide
The Deep Noise Suppression (DNS) challenge is designed to foster innovation in the area of noise suppression to achieve superior perceptual speech quality. We recently organized a DNS challenge special session at INTERSPEECH and ICASSP 2020. We open-sourced training and test datasets for the wideband scenario. We also open-sourced a subjective evaluation framework based on ITU-T standard P.808, which was also used to evaluate participants of the challenge. Many researchers from academia and industry made significant contributions to push the field forward, yet even the best noise suppressor was far from achieving superior speech quality in challenging scenarios. In this version of the challenge organized at INTERSPEECH 2021, we are expanding both our training and test datasets to accommodate full band scenarios. The two tracks in this challenge will focus on real-time denoising for (i) wide band, and(ii) full band scenarios. We are also making available a reliable non-intrusive objective speech quality metric called DNSMOS for the participants to use during their development phase.
1. Introduction
The challenge targets real-time speech enhancement under realistic noise, motivated by degraded speech quality and intelligibility in communication and assistive applications. DNS Challenge 3 expands datasets and evaluation tools while addressing the remaining gap from ideal perceptual quality.
- Background noise degrades perceived speech quality and intelligibility, causing fatigue in calls and challenging hearing aids and smart devices.
- Earlier DNS challenges showed progress, but the best suppressor remained about 1.6 DMOS from the ideal MOS of 5 on a realistic test set.
- DNS Challenge 3 introduced wideband and fullband real-time denoising tracks with expanded, publicly available speech, noise, and reverberation resources.
- The challenge adopted ITU-T P.835 evaluation and provided DNSMOS, a non-intrusive metric that does not require clean reference speech.
2. Challenge Tracks
The challenge defines two real-time denoising tracks: wideband and fullband. Both require computational processing within tight latency and runtime constraints.
- Track 1 evaluates real-time denoising for the wideband scenario.
- The suppressor must process each frame faster than the stride time on an Intel Core i5 quad-core machine or equivalent processor.
- Total algorithmic latency, including frame size, stride, and look-ahead, must be no more than 40 ms.
- Track 2 evaluates real-time denoising for the fullband scenario and must satisfy Track 1 requirements.
3. Training Datasets
The training resources combine diverse clean speech, balanced noise, and configurable reverberation data to support speech-enhancement models across wideband and fullband settings.
- 3.1. Clean Speech: The clean-speech collection totals 760.53 hours, including read speech, singing, emotion data, and Chinese Mandarin, versus 562.72 hours in DNS Challenge 1.
- 3.1. Clean Speech: Clean speech is organized into read, singing, emotional, and non-English subsets sourced from public or licensed corpora.
- 3.1. Clean Speech: The multilingual collection includes 170+ hours of clean Mandarin speech along with Spanish, German, Italian, French, and Russian data.
- 3.2. Noise: The noise dataset contains about 150 balanced audio classes, 60,000 clips, 10,000 additional clips, and 181 hours of noise data.
- 3.3. Reverberation: The resources provide 3,076 real and approximately 115,000 synthetic room impulse responses for reverberating clean speech before adding noise.
- 3.3. Reverberation: Reverberation time T60, clarity C50, and an isReal flag support selecting training data and distinguishing real from synthetic room responses.
4. Test set
The test set is designed to measure denoising under diverse languages, noises, recording conditions, and emotionally expressive speech.
- The test set contains English and non-English utterances recorded with varied background noises, SNRs, target levels, and acoustic conditions.
- It includes tonal and non-tonal languages, diverse speakers, reverberant recordings at 1–5 meters, and gender-balanced crowdsourced data.
- Development and blind test sets include laughter, crying, yelling, surprise, and some musical-instrument backgrounds to assess effects on emotional speech and sounds.
5. Challenge Results and Key Takeaways
The challenge evaluated real-time denoising models on blind test data using P.835-based subjective ratings across speech, background noise, and overall quality. Results show strong noise suppression but persistent speech distortion and a substantial gap from ideal overall quality.
- Evaluation setup: 19 teams entered Track 1, while only 3 teams entered Track 2; evaluation used five qualified raters per clip and reported Speech, Background Noise, and Overall MOS.DMOS measures the difference between processed and original noisy speech MOS; Overall MOS determined ranking, subject to other requirements.
- Track results: All but the top two teams performed worse than noisy speech on Speech MOS, likely because aggressive suppression distorted speech components.The paper attributes this likely to inaccurate frame-level noise-power estimation.
- Track results: Nearly all models improved background-noise quality, with the top three reaching almost 2 DMOS, equivalent to an MOS of 4.6.This represented a significant improvement over noisy speech.
- Track results: About 3.78 Overall MOS was achieved by the best model, approximately 1.2 DMOS below perfect MOS 5.The paper characterizes superior suppression without speech distortion as challenging and describes perceptual speech enhancement as still nascent.
- Key takeaways: Improving Speech MOS while maintaining excellent Background Noise MOS offers the largest potential gain in Overall MOS.The takeaway prioritizes speech quality improvements without sacrificing noise suppression.
6. Conclusions
The INTERSPEECH 2021 DNS Challenge brought academia and industry together around speech-enhancement research. It released diverse datasets, scripts, and perceptual metrics, which participants found useful for training noise-suppression models.
- The challenge was organized to bring academic and industry researchers together to address speech enhancement.
- Large, inclusive, diverse training and test datasets were open sourced with supporting scripts and perceptual objective metrics.
- Many industry and academic participants found the released datasets useful.