Source-linked AI summary
WHAM!: Extending Speech Separation to Noisy Environments
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, Jonathan Le Roux
TL;DR
Speech-separation benchmarks commonly use nearly fully overlapping, low-sampling-rate, noise-free mixtures, limiting realism. The paper introduces WHAM!, a noisy ambient extension of wsj0-2mix, and benchmarks enhancement and separation methods. Initial experiments show that time-frequency approaches remain effective in noise, with substantial gains over noisy inputs for most approaches.
Problem
Existing speech-separation studies largely evaluate nearly fully overlapping speech at low sampling rates without external background noise, leaving more realistic noisy scenarios underrepresented.
Method
The paper creates WHAM! by pairing two-speaker wsj0-2mix mixtures with unique ambient noise scenes and benchmarks phase-sensitive enhancement and chimera++-based separation objectives.
Results
Time-frequency separation approaches still perform effectively with noise, while finetuned cascaded enhancement and separation outperforms jointly trained chimera++ approaches.
Takeaways & Limitations
WHAM! provides a publicly available benchmark for evaluating speaker-independent separation and enhancement robustness in noisy environments.
Abstract
from arXiv · showhide
Recent progress in separating the speech signals from multiple overlapping speakers using a single audio channel has brought us closer to solving the cocktail party problem. However, most studies in this area use a constrained problem setup, comparing performance when speakers overlap almost completely, at artificially low sampling rates, and with no external background noise. In this paper, we strive to move the field towards more realistic and challenging scenarios. To that end, we created the WSJ0 Hipster Ambient Mixtures (WHAM!) dataset, consisting of two speaker mixtures from the wsj0-2mix dataset combined with real ambient noise samples. The samples were collected in coffee shops, restaurants, and bars in the San Francisco Bay Area, and are made publicly available. We benchmark various speech separation architectures and objective functions to evaluate their robustness to noise. While separation performance decreases as a result of noise, we still observe substantial gains relative to the noisy signals for most approaches.
1. Introduction
Modern deep learning has advanced monaural enhancement and separation, but standard speech-separation benchmarks omit important real-world conditions. WHAM! addresses these limitations with noisy, more realistic evaluation and initial enhancement and separation experiments.
- Motivation: Speech enhancement separates speech from unstructured background noise, whereas speech separation reconstructs multiple structured speech signals and resolves unknown output-source correspondences.Deep clustering and permutation-free mask inference are common approaches to the speaker-separation problem.
- Limitations of existing benchmarks: The wsj0-2mix benchmark uses instantaneous, noise-free mixtures of simultaneous speakers, usually truncates mixtures to near-complete overlap, and commonly reports results at 8 kHz.These design choices limit evaluation of diarization-like scenarios, higher-frequency speech, and external-noise robustness.
- WHAM! contribution: WHAM! pairs each two-speaker wsj0-2mix utterance with a unique binaural recording from non-stationary ambient environments such as coffee shops, restaurants, and bars.The dataset is publicly available and preserves parity with wsj0-2mix for evaluating robustness to noise.
- Experiments: The study evaluates enhancement with BLSTM-based phase-sensitive masking and separation mainly with chimera++ variations for simultaneous speech separation and noise removal.It also compares joint enhancement-separation with enhancement followed by a separate separation network.
2. WHAM! dataset1
WHAM! extends two-speaker mixtures with real urban ambient recordings collected and filtered for realistic noise conditions. Its construction preserves speaker-level controls while sampling challenging speech-to-noise ratios.
- Source mixtures: The source wsj0-2mix dataset contains 20,000 training, 5,000 validation, and 3,000 test two-speaker mixtures totaling 45 hours.Training and validation share speakers, whereas test speakers are different; speaker levels differ by 0–5 dB before mixing.
- Noise collection: Ambient noise was recorded across urban locations in the San Francisco Bay Area using a binaural microphone, then downsampled from 48 kHz to 16 kHz and 8 kHz.Recording settings included tripod placement at 1.0–1.5 m and 15–17 cm inter-microphone spacing.
- Noise filtering: Foreground speech was estimated with iZotope RX 7 Dialogue Isolate, and 10-second chunks were screened using speech-versus-residual SNR before inclusion.The filtering procedure targeted removal of high-SNR, intelligible speech from the ambient corpus.
- Mixture construction: Noise is mixed at a uniformly sampled SNR between -6 and +3 dB while preserving the original relative levels between the two speakers.The range was chosen from recordings of conversations in environments similar to those used for noise collection.
3. Speech separation objective functions
The separation objectives estimate permutation-free real-valued masks and combine phase-sensitive reconstruction with deep clustering. For noisy separation, the objectives offer several ways to represent or exclude background noise.
- Permutation-free mask objectives: The tPSA objective estimates a real-valued mask for each source from the mixture spectrogram while minimizing loss over all source permutations.The formulation uses source and mixture phase information and truncates targets to the mixture magnitude range.
- Task-specific objectives: For enhancement, the reconstruction loss targets only the speech signal, while noisy separation excludes noise from the source set used in the loss.This differs from the multi-source separation setting, where all speech outputs require accurate reconstruction.
- Deep clustering: Chimera++ adds a deep clustering regularization term to mask estimation, with the combined loss weighted between deep clustering and phase-sensitive objectives.The deep clustering term can use classic or whitened k-means variants, with α typically set to 0.975.
- Noise-aware clustering: Deep clustering represents time-frequency bins with embedding matrix V and source-dominance labels with one-hot matrix Y, using magnitude-ratio weights to discount low-energy bins.For noisy separation, alternatives include treating noise as an additional source, weighting nonspeech bins out, or separating noise-dominated embeddings from speech-dominated ones.
- Cascaded processing: A cascaded alternative first removes background noise with an enhancement network and then separates individual speakers with a second network.This approach separates the enhancement and speaker-separation stages rather than jointly optimizing them.
4. Experimental results
Experiments use WHAM! to compare enhancement and separation across controlled tasks, sampling rates, oracle masks, objectives, and architectures. Noise and task conditions affect performance, while deep-clustering and finetuned cascades provide important distinctions.
- Task setup: WHAM! evaluates enhance-single, enhance-both, separate-clean, and separate-noisy tasks with retrained chimera++ networks.The architecture uses four bidirectional LSTM layers and separate mask-inference and deep-clustering heads.
- Oracle results: Oracle SI-SDR performance is similar at 8 kHz and 16 kHz across tasks, although enhance-both improves about 2 dB less than enhance-single.This indicates that removing noise from mixtures containing multiple speakers is harder than removing noise from one speaker.
- Scatter-plot analysis: Most utterances cluster near 10 dB SI-SDR improvement, but gains for very noisy signals are less pronounced than for clean separation.The results suggest that improving relatively quiet speakers is more difficult when background noise is present.
- Objective comparisons: Three deep-clustering sources with the unmodified whitened k-means objective perform best for noisy separation.Treating noise as a source outperforms two-source weighting and the LDC,N objective.
- Cascaded systems: The cascaded enhancement-then-separation system underperforms without finetuning but surpasses jointly trained chimera++ approaches after finetuning.The finetuned cascade is more computationally expensive and may suit systems with an existing enhancement algorithm.
- Other benchmarks: The study also benchmarks TasNet and a fully convolutional spectrogram model on WHAM! separate-clean and separate-noisy tasks.The TasNet implementation uses an SI-SNR objective, while the convolutional model uses dilated and pointwise convolutions.
5. Conclusion
The paper introduces WHAM! for evaluating speaker-independent speech separation in noisy environments and benchmarks enhancement and separation approaches. Initial results indicate that time-frequency-based separation remains effective despite noise.
- WHAM! enables evaluation of speaker-independent separation in noisy environments and benchmarking of speech enhancement and separation approaches.
- Initial results show that time-frequency-based separation approaches remain effective in the presence of noise.