Source-linked AI summary
WHAMR!: Noisy and Reverberant Single-Channel Speech Separation
Matthew Maciejewski, Gordon Wichern, Emmett McQuinn, Jonathan Le Roux
TL;DR
Speech-separation research has focused on clean mixtures, whereas real recordings often include noise and reverberation. The paper introduces WHAMR!, evaluates separation and enhancement systems under these conditions, and finds value in learned-basis and cascaded approaches while identifying scaling mismatch as a cascade limitation.
Problem
Most prior separation studies use clean, near-anechoic mixtures, while WHAM! adds noise but not reverberation, limiting realism for indoor recordings.
Method
The paper introduces WHAMR!, a synthetically reverberated extension of WHAM!, and evaluates baseline and cascaded enhancement-separation architectures.
Results
WHAMR! experiments show that noise and reverberation degrade performance, while learned-basis representations and cascaded enhancement-separation systems provide strong baseline results.
Takeaways & Limitations
WHAMR! provides a more realistic benchmark with ground truth suitable for training and evaluating speech separation and enhancement systems.
Takeaways & Limitations
Scale-invariant loss leaves cascade outputs unconstrained in dynamic range, causing scaling mismatch and performance degradation when sub-models are trained separately.
Abstract
from arXiv · showhide
While significant advances have been made with respect to the separation of overlapping speech signals, studies have been largely constrained to mixtures of clean, near anechoic speech, not representative of many real-world scenarios. Although the WHAM! dataset introduced noise to the ubiquitous wsj0-2mix dataset, it did not include reverberation, which is generally present in indoor recordings outside of recording studios. The spectral smearing caused by reverberation can result in significant performance degradation for standard deep learning-based speech separation systems, which rely on spectral structure and the sparsity of speech signals to tease apart sources. To address this, we introduce WHAMR!, an augmented version of WHAM! with synthetic reverberated sources, and provide a thorough baseline analysis of current techniques as well as novel cascaded architectures on the newly introduced conditions.
1. INTRODUCTION
Existing speech-separation research largely uses clean, near-anechoic mixtures that do not represent far-field recordings with noise and reverberation. WHAMR! addresses this gap by adding synthetic reverberation to WHAM! and evaluating baselines and cascaded systems across realistic conditions.
- Overlapped speech degrades single-speech processing technologies and can also be difficult for human listeners.
- The wsj0-2mix benchmark uses synthetically mixed studio recordings and is not representative of many real-world overlapped-speech scenarios.Far-field speakers may be captured from a distance in rooms such as meetings or homes.
- Reverberation causes spectral smearing, while WHAM! added ambient noise without modeling reverberation or speech spatialization.These effects challenge systems that rely on time-frequency spectral structure.
- WHAMR! augments WHAM! with synthetic reverberated speech sources while retaining the noise augmentation of wsj0-2mix.Its synthetic construction provides clean and reverberated speech ground truth for training and evaluation.
- The paper evaluates clean, noisy, reverberant, and noisy-plus-reverberant separation and enhancement, including end-to-end trainable cascaded systems.
2. WHAMR! DATASET
WHAMR! extends the noise-augmented wsj0-2mix corpus with synthetically generated room reverberation and spatialized speech components. Its source-complete design supports multiple enhancement and separation tasks with common anechoic targets for comparable evaluation.
- WHAMR! extends WHAM!, which itself is a noise-augmented version of wsj0-2mix.The underlying wsj0-2mix mixtures combine WSJ0 utterances with random gain to create overlapping speech.
- Reverberation is generated by convolving speech with room impulse responses sampled from domestic- and classroom-like room configurations.Reverberation is classified qualitatively as high, medium, or low.
- The corpus includes anechoic and reverberant versions of the speech components, while the binaural-recorded noise is already spatialized.The study uses only the left channel for single-channel separation.
- Anechoic direct-path sources serve as dereverberation targets, avoiding the need to model spatialized-source time delays during training.
- WHAMR! supports enhancement, separation, and joint enhancement-separation experiments because source, noise, reverberated, and combined components are included.
- The four core separation tasks map clean or corrupted mixtures to anechoic sources, enabling more direct raw-SDR comparisons across conditions.The defined conditions are clean, noisy, reverberant, and noisy and reverberant.
3. EXPERIMENTAL CONFIGURATIONS
The experiments compare spectro-temporal masking systems using STFT or learned-basis features with BLSTM or TCN masking networks, and evaluate modular enhancement-separation cascades. Training uses permutation-invariant SI-SDR, while separate cascaded models require scaling correction because SI-SDR leaves output magnitude unconstrained.
- Network configurations: All baseline systems project waveforms into a spectro-temporal representation, estimate masks, suppress interference, and reconstruct source waveforms.Enhancement systems estimate one mask for noise and/or reverberation.
- Network configurations: The feature extractors are STFT and TasNet-style learned bases, paired with BLSTM or TCN masking networks.
- Network configurations: Learned-basis configurations are evaluated for separation, while enhancement receives full feature comparisons because TasNet-like enhancement systems are less extensively evaluated.
- Training objective: Permutation-invariant SI-SDR is both the waveform-level training objective and evaluation metric, supporting end-to-end training of cascaded models.
- Cascaded models: Scale-invariant loss leaves outputs unconstrained in dynamic range, which can cause performance problems when separately trained sub-models are cascaded.
- Cascaded models: Cascades optionally place enhancement before or after separation, assigning noise and reverberation removal to selected stages.Post-separation denoising is excluded because noise does not belong clearly to either separated speech signal.
- Cascaded models: Output rescaling uses the mixture and estimated source to reduce scaling mismatch between separately trained cascade stages.The factor is motivated by approximate orthogonality between interference and target, and improves as the estimate becomes more colinear with the target.
- Cascaded models: Best-performing cascades receive additional end-to-end tuning through all sub-models, using direct waveform-to-waveform joint training.
4. EXPERIMENTAL RESULTS
Experiments compare core separation, enhancement, and cascaded systems across noisy and reverberant conditions using SI-SDR. Results favor learned-basis BLSTM systems and indicate benefits from modular enhancement, especially for reverberation.
- Core systems: Reverberation is more challenging than noise, as reflected by lower SI-SDR in the core systems.
- Core systems: TasNet-BLSTM consistently outperforms TasNet-TCN, despite having many more parameters and contradicting prior results.
- Core systems: TasNet-BLSTM results are considerably better than the corresponding results in the original WHAM! paper.
- Enhancement: TasNet-BLSTM is the best enhancement architecture, while learned-basis models substantially outperform STFT-feature models.
- Evaluation: The experiments report SI-SDR for two-speaker enhancement, cascaded-model comparisons, additional training, and 16 kHz conditions.
- Cascaded systems: Separating enhancement from separation generally helps, with post-separation dereverberation slightly better than pre-separation dereverberation.
5. CONCLUSION
The paper introduces WHAMR! to extend noisy speech separation toward more realistic conditions by including reverberation. Its results support learned-basis representations, cascaded enhancement and separation, and further joint fine-tuning.
- WHAMR! extends WHAM! with reverberation to promote speech separation research under more realistic conditions.
- Noise and reverberation degrade overall performance, while learned-basis representations remain effective for separation and enhancement.
- Cascaded models combining pretrained separation and enhancement modules, followed by joint fine-tuning, provide strong WHAMR! baseline results.