Source-linked AI summary
ICASSP 2023 Deep Noise Suppression Challenge
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Hannes Gamper, Mehrsa Golestaneh, Robert Aichner
TL;DR
The challenge addresses joint denoising, dereverberation, and interfering-talker suppression in Headset and Speakerphone conditions. It provides fullband data, enrollment speech, and shared evaluation for personalized and non-personalized models. Most submissions were personalized, and the challenge reports winners in both tracks with improvements in Score over the noisy blind testset.
Problem
Deep speech enhancement must preserve primary speech while suppressing noise, reverberation, and simultaneously active interfering talkers.
Method
The challenge uses separate Headset and Speakerphone tracks with fullband data, enrollment speech, personalized and non-personalized models, and Personalized P.835 plus WAcc evaluation.
Results
Winners were reported in both tracks, with most submitted models personalized and challenge performance evaluated using subjective P.835 scores and WAcc.
Takeaways & Limitations
Separate device tracks and enrollment-based evaluation create a benchmark for joint enhancement and suppression of neighboring talkers under varied real-world conditions.
Takeaways & Limitations
WAcc can be affected by secondary-talker leakage because the speech-recognition engine is not personalized and does not use enrollment speech.
Abstract
from arXiv · showhide
Deep Speech Enhancement Challenge is the 5th edition of deep noise suppression (DNS) challenges organized at ICASSP 2023 Signal Processing Grand Challenges. DNS challenges were organized during 2019-2023 to stimulate research in deep speech enhancement (DSE). Previous DNS challenges were organized at INTERSPEECH 2020, ICASSP 2021, INTERSPEECH 2021, and ICASSP 2022. From prior editions, we learnt that improving signal quality (SIG) is challenging particularly in presence of simultaneously active interfering talkers and noise. This challenge aims to develop models for joint denosing, dereverberation and suppression of interfering talkers. When primary talker wears a headphone, certain acoustic properties of their speech such as direct-to-reverberation (DRR), signal to noise ratio (SNR) etc. make it possible to suppress neighboring talkers even without enrollment data for primary talker. This motivated us to create two tracks for this challenge: (i) Track-1 Headset; (ii) Track-2 Speakerphone. Both tracks has fullband (48kHz) training data and testset, and each testclips has a corresponding enrollment data (10-30s duration) for primary talker. Each track invited submissions of personalized and non-personalized models all of which are evaluated through same subjective evaluation. Most models submitted to challenge were personalized models, same team is winner in both tracks where the best models has improvement of 0.145 and 0.141 in challenge's Score as compared to noisy blind testset.
1. INTRODUCTION
The challenge expands deep speech enhancement evaluation with two tracks, improved personalized assessment, and shared resources for model development. Participants could use external or challenge datasets and submit personalized or non-personalized models.
- Shared resources: Open-sourced datasets, test sets, evaluation frameworks, DNSMOS P.835, and WAcc APIs reduce barriers to iterative enhancement-model development.These resources provide standardized tools for evaluating deep speech enhancement models.
- Participation: Participants could train with external corpora or challenge datasets, provided they documented their training data in detail.The challenge did not prohibit using headset corpora for Speakerphone models or vice versa.
- Challenge changes: The challenge introduced Headset and Speakerphone tracks with desktop and mobile recordings, enrollment speech, improved Personalized P.835 evaluation, and joint ranking of personalized and non-personalized models.The improved framework adds voice recognition, spam filtering, more accurate evaluation with noise and neighboring talkers, and cleaned enrollment speech.
- Submissions: Most submitted models were personalized, and participants could also submit Preference-2 enhanced clips for future semi-supervised DNSMOS P.835 training.Preference-2 clips were not sent for subjective evaluation.
2. CHALLENGE TRACKS
The challenge targets joint denoising, dereverberation, and interfering-talker suppression while preserving the primary talker. Separate Headset and Speakerphone tracks support distinct recording conditions and enrollment robustness.
- Challenge objective: The central task is balancing speech-quality preservation against suppression of noise, reverberation, and neighboring talkers.Models are expected to preserve the primary talker while suppressing everything else.
- Track rationale: Headset acoustics may enable non-personalized suppression of neighboring talkers without enrollment speech, supporting privacy-sensitive applications.The challenge studies whether headset and speakerphone conditions yield different modeling insights.
- Track design: Track-1 Headset and Track-2 Speakerphone use different development and blind test sets collected with the corresponding device types.Both tracks otherwise used similar collection procedures.
- Enrollment conditions: Each test clip includes enrollment speech for the primary talker, which may be noisy or reverberant to test personalized-model robustness.The passage specifies 30 seconds of enrollment speech for each test clip.
3. CHALLENGE DATASETS
The challenge provides broad fullband speech, noise, and real-world test data spanning devices, languages, emotions, paralinguistic events, and difficult acoustic conditions. Training resources also include speaker identities and embeddings for personalization.
- Personalization resources: Training data includes speaker IDs, ECAPA-TDNN speaker-embedding code, emotional speech, and English plus 10 non-English languages.These resources facilitate personalized-model development across varied speech conditions.
- Speech data: Track-2 training speech totals 760.53 hours, while Track-1 speech is derived through near-end speech extraction from the Track-2 dataset.The Track-2 collection includes read, singing, emotional, and Mandarin speech.
- Noise data: The noise set contains 152 balanced AudioSet classes, 60,000 clips, and 181 hours of 48 kHz noise data after speech removal and upsampling.The sampling procedure ensures each class has at least 500 clips.
- Test data: Real-world test clips were crowd-sourced across devices and include new noise types, paralinguistic events, and previously unavailable blind-test recordings.The blind set was designed to cover relevant scenarios and device variety.
- Development sets: The non-personalized development set contains 600 real recordings, while the personalized set contains 1,443 real recordings.Both sets include English noisy speech and emotional clips.
- Stress conditions: Blind tests include emotional speech, high reverberation, noisy reverberation, and interfering talkers across office and home noise scenarios.Emotional examples include happiness, sadness, anger, yelling, crying, and laughter.
4. EVALUATION SETUP
Evaluation combines personalized subjective ratings with speech-recognition accuracy on blind test clips. Cleaned enrollment speech supports human recognition of the primary talker, while the challenge score integrates OVRL and WAcc.
- Baselines: The challenge provided personalized and non-personalized baseline models for the two tracks.The baselines were variants of models presented in prior work.
- Enrollment cleaning: Enrollment clips were cleaned with a non-causal fullband E3Net-based model trained to remove background noise and reverberation.The model uses a learnable encoder-decoder and bidirectional LSTM for offline processing.
- Subjective evaluation: Personalized P.835 uses 5 seconds of cleaned enrollment speech to help raters recognize the primary talker during subjective scoring.Raters focus on primary-talker quality when multiple talkers are present.
- Subjective metrics: The evaluation scores speech quality, background noise quality, and overall audio quality as SIG, BAK, and OVRL, while treating interfering talkers as undesirable signal.All challenge models were evaluated using the Personalized P.835 framework.
- Objective metric: WAcc is computed from a state-of-the-art speech-recognition system and defined as WAcc = 1 − WER.WER is the recognition system’s word error rate.
- Transcript evaluation: WAcc uses entire blind test clips and primary-talker ground-truth transcripts, unlike subjective evaluation, which uses manually selected short segments.Blind clips range from 10 seconds to more than six minutes.
- Evaluation limitation: Secondary-talker leakage can produce WER ≥ 0.5 because the non-personalized recognizer transcribes interfering speech as insertions.The recognition system does not use enrollment speech to focus on the primary talker.
- Ranking: Challenge models are ranked using SIG, BAK, OVRL, and WAcc measured on the blind test set.The final ranking combines subjective P.835 scores with word accuracy.
5. RESULTS & DISCUSSIONS
The challenge evaluated personalized and non-personalized speech-enhancement models across two tracks, with most submissions using personalization. Performance was compared using subjective P.835 measures, word accuracy, and the challenge Score.
- Submissions and baselines: 11 submissions were received in each track, with 10 teams participating in both; almost all models were personalized.Track-1 included personalized and non-personalized baselines, while Track-2 used a personalized baseline.
- Submissions and baselines: Track-1 included two baselines, including one non-personalized model, whereas Track-2 used a personalized baseline.The challenge motivated a non-personalized Track-1 baseline because headset acoustic properties may help suppress neighboring talkers and noise.
- Evaluation metrics: Figure 1 ranks all teams by personalized P.835 scores, WAcc, and the challenge Score.The plotted measures include dMOS for SIG, BAK, and OVRL, plus dWAcc, each defined relative to the corresponding noisy clip.
- Evaluation metrics: dMOS measures enhanced-minus-noisy differences for SIG, BAK, and OVRL, while dWAcc measures the corresponding WAcc difference.ANOVA tests were conducted on top models in each track to assess statistical significance.
6. CONCLUSIONS
The challenge used a diverse crowdsourced blind test set and found signal-quality degradation for most models. The authors attribute this to the test set’s difficulty and possible suppression of the primary talker.
- Test set and evaluation: The blind test set was collected through crowdsourcing from multiple data vendors and included paralinguistic and leakage test clips.Enrollment speech and noisy test segments were selected to support robust subjective evaluation.
- Observed outcomes: Most models showed degradation in signal quality on the NRT test set used to verify whether winning models were causal.The authors identify test-set difficulty and suppression of the primary talker as possible reasons.
- Observed outcomes: Noticeable SIG degradation was associated with suppression of the primary talker’s speech and leakage of interfering talkers or noise.The conclusion connects these outcomes to the challenge’s difficult evaluation conditions.