Source-linked AI summary
ICASSP 2022 Deep Noise Suppression Challenge
Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, Robert Aichner
TL;DR
The challenge targets perceptual speech-quality degradation in realistic noisy audio by benchmarking personalized and non-personalized fullband DNS. It releases datasets, test sets, evaluation frameworks, and APIs, and ranks models using P.835 scores and WAcc. The results show strong personalized-track performance, but no team improved SIG and WAcc was 2% worse than noisy, leaving room for improvement.
Problem
Noisy real-world audio calls degrade perceived speech quality and intelligibility, motivating improved deep noise suppression evaluation.
Method
The challenge provides fullband datasets, real-world blind and development tests, personalized and non-personalized tracks, P.835 evaluation, DNSMOS P.835, and WAcc.
Results
The personalized DNS track greatly exceeded the first personalized challenge, but no team improved SIG and WAcc was 2% worse than noisy.
Takeaways & Limitations
The released resources and evaluation tools are intended to push DNS research forward, while challenging noisy scenarios still require substantial improvement.
Takeaways & Limitations
Personalized DNS assumes 2.5 minutes of clean enrollment speech for each unique talker.
Abstract
from arXiv · showhide
The Deep Noise Suppression (DNS) challenge is designed to foster innovation in the area of noise suppression to achieve superior perceptual speech quality. This is the 4th DNS challenge, with the previous editions held at INTERSPEECH 2020, ICASSP 2021, and INTERSPEECH 2021. We open-source datasets and test sets for researchers to train their deep noise suppression models, as well as a subjective evaluation framework based on ITU-T P.835 to rate and rank-order the challenge entries. We provide access to DNSMOS P.835 and word accuracy (WAcc) APIs to challenge participants to help with iterative model improvements. In this challenge, we introduced the following changes: (i) Included mobile device scenarios in the blind test set; (ii) Included a personalized noise suppression track with baseline; (iii) Added WAcc as an objective metric; (iv) Included DNSMOS P.835; (v) Made the training datasets and test sets fullband (48 kHz). We use an average of WAcc and subjective scores P.835 SIG, BAK, and OVRL to get the final score for ranking the DNS models. We believe that as a research community, we still have a long way to go in achieving excellent speech quality in challenging noisy real-world scenarios.
1. INTRODUCTION
The challenge addresses degraded speech quality and intelligibility in noisy remote audio calls by advancing deep noise suppression. It provides datasets, real-world tests, evaluation frameworks, and APIs to support model development and ranking.
- Motivation: Remote audio calls with background noise can substantially degrade perceived speech quality and intelligibility.The motivating examples include barking dogs, crying babies, kitchen noise, neighboring talkers, and in-car noise.
- Motivation: Deep learning-based noise suppression has shown better speech quality than classical approaches.
- Challenge scope: The challenge covers personalized and non-personalized fullband DNS and supplies training datasets for both tracks.Personalized denoising additionally suppresses neighboring talkers while preserving the primary talker.
- Evaluation support: Real-world test sets, P.835 subjective evaluation, DNSMOS P.835, and WAcc APIs support model optimization and assessment.The APIs were provided during the challenge to help participants iteratively optimize their models.
- Evaluation support: Blind-test ranking combines P.835 SIG, BAK, and OVRL subjective scores with WAcc from speech recognition.DNSMOS P.835 predicts the three P.835 scores for noisy test clips.
2. CHALLENGE TRACKS
The challenge evaluates non-personalized and personalized fullband DNS through separate tracks, shared challenge resources, and objective and subjective measures. WAcc was added to capture speech-recognition effects alongside perceptual quality.
- Challenge tracks: The challenge has non-personalized DNS and personalized DNS tracks, both targeting fullband audio.The personalized track is abbreviated PDNS.
- Challenge resources: Training data synthesizers, clean speech, noise, impulse responses, and track-specific baselines support model development.The same noise and impulse responses are provided for both tracks, while each track has its own synthesizer.
- Evaluation: WAcc was adopted as an objective metric to measure DNS effects on speech recognition.The motivation was prior WAcc degradation from over-suppression of noise or speech.
- Evaluation: Participants submitted enhanced clips for one or both tracks, which were evaluated using P.835 and WAcc computations.
3.1. Training Datasets
The training resources combine multilingual clean speech, balanced speech-free noise data, impulse responses, and synthesizers for fullband DNS model development. Personalized training additionally organizes clean speech by talker for speaker-aware modeling.
- Resources: Training resources provide clean speech, noise, impulse responses, and synthesizers for both DNS tracks.The same noise and impulse responses are shared across tracks, with separate synthesizers for each track.
- Clean speech: Clean-speech data spans English, French, German, Italian, Russian, and Spanish, with English also including singing.The PDNS training set contains 3230 talkers.
- Personalized data: The personalized track concatenates each talker’s clips and provides speaker embeddings for talker-aware training and adaptation.Clean speech is filtered using DNSMOS P.835 thresholds, including PDNS OVRL ≥4.25.
- Noise: Noise data contains about 62,000 clips across 150 classes and 181 hours of audio.Speech activity detection removes noise clips containing voice content, and sampling balances classes to at least 500 clips each.
- Data synthesis: The synthesizer can add noise to reverberant clean speech using 248 real and about 60,000 synthetic room impulse responses.Participants may use clean or reverberant speech as DNS training targets.
3.2. Test set
The challenge provides development and blind fullband test sets built from real-world recordings, with the blind set reserved for unseen ranking. Personalized testing includes enrollment speech for preserving the primary talker.
- Test sets: The development set supports model optimization, while the blind set ranks models as an unseen generalization test.
- Test sets: Both test sets contain fullband audio clips recorded in real-world scenarios.The supplied passage introduces these recordings as crowdsourced test data.
- Non-personalized test set: The non-personalized development set contains 930 real English noisy-speech recordings, including 193 emotional-speech clips.It includes six emotion types and diverse everyday noise conditions.
- Non-personalized test set: Non-personalized development clips are 10–20 seconds long, recorded at 48 kHz, and captured near-field or at 1, 2, or 3 meters.All clips were recorded using laptop or desktop computers.
- Personalized test set: The personalized development and blind sets provide 2.5 minutes of enrollment speech for each primary talker.PDNS uses speaker embeddings to preserve the primary talker while suppressing neighboring talkers and noise.
3.3. Blind test set
The common blind test set supports direct comparison between non-personalized and personalized DNS across desktop and mobile real-world recordings.
- Blind test set: The blind test set contains 859 real clips, each lasting 10 seconds.
- Blind test set: Recordings were collected through crowd-sourcing on desktop and mobile platforms.
4. RESULTS & DISCUSSIONS
The challenge evaluates non-personalized and personalized DNS models using P.835 subjective scores, WAcc, and a combined ranking score. Results include submissions across both tracks, tied top placements, device-specific score differences, and comparisons of model characteristics.
- Evaluation methodology: The evaluation uses ITU-T P.835 because objective metrics such as PESQ, SDR, and POLQA correlate poorly with subjective speech quality.Personalized DNS uses a modified P.835 procedure that provides enrollment speech for recognizing the primary speaker.
- Challenge participation: 24 and 10 submissions were received for Tracks 1 and 2, respectively.Each team submitted a processed blind test set.
- Evaluation and ranking: P.835 scores, WAcc, and final scores rank challenge entries in decreasing performance order.The reported differences dSIG, dBAK, dOVRL, and dWAcc compare enhanced and noisy clips.
- Ranking outcomes: The 2nd, 3rd, and 4th places were tied in Track 1, while the 1st and 2nd places were tied in Track 2.Teams 17, 19, and 42 were disqualified for not submitting papers.
- Device breakdown: Mobile-device clips had higher MOS scores than desktop-device clips.The authors suggest that mobile scenarios had better acoustic devices or environments than desktop scenarios.
- Model comparison: Top-performing Track 1 models used no additional datasets, whereas Track 2 models did.The comparison also reports low correlation between model size or real-time factor and performance.
5. CONCLUSION
The challenge resources and top-performing papers are intended to advance deep noise suppression research. Future editions will broaden realistic evaluation and standardize efficiency and lookahead comparisons.
- The challenge dataset, test set, evaluation framework, DNSMOS P.835, and top-performing papers are intended to help advance the field.
- Future test sets will cover more languages, accents, devices, and realistic noisy scenarios.
- A dedicated evaluation setup will measure submitted models' complexity and inference time, while lookahead validation will support fair comparisons.