Source-linked AI summary
Single-Channel Multi-Speaker Separation using Deep Clustering
Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, John R. Hershey
TL;DR
Single-channel mixtures remain difficult to separate because source assignments must be inferred from overlapping spectrograms. The paper extends deep clustering with improved training, enhancement, and end-to-end signal approximation, achieving large gains in separation and recognition, including WER reduction from 89.1% to 30.8%.
Problem
Single-channel cocktail-party mixtures remain difficult to separate and recognize, especially under general conditions.
Method
The paper combines deep-clustering embeddings and masking with enhancement layers and end-to-end signal-reconstruction training.
Results
89.1% to 30.8% WER: end-to-end signal approximation produces unprecedented performance in automatic speech recognition evaluation.
Takeaways & Limitations
The extended framework substantially improves both signal-quality metrics and speech-recognition error rates for multi-speaker separation.
Abstract
from arXiv · showhide
Deep clustering is a recently introduced deep learning architecture that uses discriminatively trained embeddings as the basis for clustering. It was recently applied to spectrogram segmentation, resulting in impressive results on speaker-independent multi-speaker separation. In this paper we extend the baseline system with an end-to-end signal approximation objective that greatly improves performance on a challenging speech separation. We first significantly improve upon the baseline system performance by incorporating better regularization, larger temporal context, and a deeper architecture, culminating in an overall improvement in signal to distortion ratio (SDR) of 10.3 dB compared to the baseline of 6.0 dB for two-speaker separation, as well as a 7.1 dB SDR improvement for three-speaker separation. We then extend the model to incorporate an enhancement layer to refine the signal estimates, and perform end-to-end training through both the clustering and enhancement stages to maximize signal fidelity. We evaluate the results using automatic speech recognition. The new signal approximation objective, combined with end-to-end training, produces unprecedented performance, reducing the word error rate (WER) from 89.1% down to 30.8%. This represents a major advancement towards solving the cocktail party problem.
1. Introduction
The paper tackles single-channel cocktail-party separation by extending deep clustering with improved training, enhancement, and end-to-end signal approximation. These extensions substantially improve separation and recognition performance.
- Single-channel speech separation in general conditions has remained an extremely challenging computer-science problem.
- Deep clustering assigns embeddings to STFT time-frequency indices, whose clusters determine source-separating masks.
- The paper improves training, investigates three-speaker separation, and adds enhancement layers with joint end-to-end optimization.
- 89.1% to 30.8% WER: end-to-end signal approximation produces unprecedented automatic speech recognition performance.
2. Deep Clustering Model
Deep clustering learns embeddings whose geometry represents source membership, then converts clustered embeddings into masks for source estimation.
- The model represents each spectrogram time-frequency bin with an embedding and learns embeddings that reflect source-dominance partitions.
- The permutation-independent affinity target marks whether pairs of time-frequency elements belong to the same source cluster.
- At test time, K-means clusters the embeddings, and the resulting assignments become binary masks applied to the complex mixture spectrogram.
3. Improvements to the Training Recipe
The training recipe improves the baseline through regularization and systematic choices of model architecture, optimization, and preprocessing.
- The study evaluates dropout, model size and shape, and training schedule as routes to improve baseline deep clustering.
- Global mean-variance normalization is applied before training, alongside RMSProp, a fixed learning-rate schedule, and cross-validation early stopping.
- Recurrent dropout samples dropout nodes once per sequence and applies them consistently across that sequence.
4. Optimizing Signal Reconstruction
Deep clustering segments spectrograms but leaves strongly interfered regions unresolved, so the paper adds enhancement and trains reconstruction directly.
- Deep clustering identifies regions dominated by each source but does not recover source content in regions dominated by other sources.
- An enhancement network combines the mixture amplitude spectrogram with each deep-clustering estimate to produce refined source masks.
- The enhancement objective minimizes permutation-invariant squared reconstruction error between true and final source estimates.
- Direct signal-reconstruction training can improve deep clustering, particularly where another source dominates the mixture.
5. End-to-End Training
The paper replaces nondifferentiable hard K-means with soft K-means so clustering and enhancement can be trained jointly for signal fidelity. The soft clustering steps are unfolded into network layers, allowing gradients to pass backward through clustering.
- 5. End-to-End Training: Soft K-means enables end-to-end training because hard K-means masks are nondifferentiable and generally cannot represent optimal continuous masks.The method jointly trains the deep clustering and enhancement stages to improve signal fidelity.
- 5. End-to-End Training: Deep unfolding treats the soft K-means steps as layers in a clustering network.This allows the clustering procedure to participate in end-to-end optimization.
- 5. End-to-End Training: Back-propagation passes gradients from each clustering step to preceding layers during end-to-end training.The unfolded clustering computation is differentiated using standard back-propagation.
- 5. End-to-End Training: The clustering hardness is controlled by α, with larger values making soft assignments approach K-means.Embedding weights are set to 1 except for silence time-frequency bins, which receive weight 0.
6. Experiments
Experiments evaluate deep clustering on single-channel, speaker-independent mixtures of two and three speakers, then test architectural improvements and end-to-end enhancement. Results are assessed with SDR improvements and speech recognition error rates.
- Experimental setup: The evaluation uses two- and three-speaker mixtures with all gender combinations in a single-channel, speaker-independent separation task.Two-speaker mixtures are drawn from WSJ0, with training mixtures spanning randomly selected 0–10 dB SNRs.
- Baselines and comparisons: The baseline and comparison methods include deep clustering, CASA, an ideal binary mask, and an oracle Wiener-like filter.The ideal binary mask selects the dominant source at each time-frequency bin, while the Wiener-like filter uses source magnitudes to form a soft mask.
- Model improvements: Regularization and training changes were evaluated using dropout, recurrent dropout, gradient normalization, larger architectures, and longer training segments.The initial system used 100-frame segments and a 300×2 BLSTM architecture; the regularization study used feed-forward dropout p = 0.5, recurrent dropout p = 0.2, and gradient normalization |∇| ≤200.
- Model improvements: 9.4 dB SDR improvement was obtained with a deeper 300 × 4 architecture containing four BLSTM layers and 300 units in each LSTM.Architecture experiments varied the number of hidden units and BLSTM layers on the two-speaker test set.
- End-to-end enhancement: The enhancement network uses two BLSTM layers per source and a soft-max masking layer, with separate training followed by end-to-end fine-tuning.The enhancement stage is trained to improve signal reconstruction, particularly in regions where another source dominates.
- Speech recognition evaluation: 30.8% WER was obtained with the end-to-end network, compared with 89.1% for the noisy mixture baseline and 19.9% for clean speech.The raw deep-clustering output performed poorly for ASR despite good perceptual quality, while enhancement networks mitigated this degradation.