Source-linked AI summary
Deep attractor network for single-microphone speaker separation
Zhuo Chen, Yi Luo, Nima Mesgarani
TL;DR
Single-channel speech separation must handle arbitrary source permutations and unknown source counts. The proposed deep attractor network forms source-specific attractors in an embedding space and converts their similarities with time-frequency bins into masks for end-to-end reconstruction. It performs separation across different numbers of mixing sources, with fixed attractors supporting real-time processing, and was effective on two-speaker separation tasks.
Problem
Speech separation remains challenging because source order is arbitrary and the number of sources in a mixture can vary.
Method
The deep attractor network forms embedding-space attractors and converts similarities between attractors and time-frequency embeddings into soft separation masks for end-to-end training.
Results
The proposed framework performed effectively on two-speaker separation tasks and worked with different numbers of mixing sources.
Takeaways & Limitations
Fixed attractors generalized to unseen mixtures and enabled real-time frame-by-frame processing without K-means post-processing.
Abstract
from arXiv · showhide
Despite the overwhelming success of deep learning in various speech processing tasks, the problem of separating simultaneous speakers in a mixture remains challenging. Two major difficulties in such systems are the arbitrary source permutation and unknown number of sources in the mixture. We propose a novel deep learning framework for single channel speech separation by creating attractor points in high dimensional embedding space of the acoustic signals which pull together the time-frequency bins corresponding to each source. Attractor points in this study are created by finding the centroids of the sources in the embedding space, which are subsequently used to determine the similarity of each bin in the mixture to each source. The network is then trained to minimize the reconstruction error of each source by optimizing the embeddings. The proposed model is different from prior works in that it implements an end-to-end training, and it does not depend on the number of sources in the mixture. Two strategies are explored in the test time, K-means and fixed attractor points, where the latter requires no post-processing and can be implemented in real-time. We evaluated our system on Wall Street Journal dataset and show 5.49\% improvement over the previous state-of-the-art methods.
1. INTRODUCTION
Single-channel speech separation remains difficult because target ordering is arbitrary and mixtures may contain varying numbers of sources. The attractor network addresses both issues through source-specific embedding-space reference points and directly learned separation masks.
- Speech separation remains unresolved despite advances in deep learning for other speech-processing tasks.
- The permutation problem arises because multiple source-output orderings are equally valid for the same mixture.For a mixture of speakers A and B, both (A,B) and (B,A) are acceptable solutions.
- The output dimension mismatch problem prevents fixed-output networks from flexibly separating mixtures with arbitrary source counts.
- Deep clustering addresses permutation and output-dimension problems but does not directly optimize separated signals through an end-to-end objective.Its separation error is minimized using an unfolding clustering system and a second network trained iteratively.
- The attractor network forms one embedding-space reference point per source, estimates masks from embedding–attractor similarity, and supports arbitrary source counts without permutation handling.The mask-learning formulation is designed to enable efficient end-to-end training and reduce computation relative to deep clustering and permutation-invariant training.
2. ATTRACTOR NEURAL NETWORK
The attractor network maps mixture spectrograms into an embedding space, estimates source attractors, and reconstructs sources through similarity-based masks optimized by reconstruction error. It supports end-to-end training and flexible test-time attractor estimation, including K-means and stable fixed attractors.
- 2.1. Model: The network maps mixture sound X to a K-dimensional embedding for each time-frequency bin and forms masks to extract each source.S denotes clean source spectrograms, X the mixture spectrogram, M the source mask, and V the embedding matrix.
- 2.1. Model: Source attractors are centroids in embedding space, computed from source memberships and used to represent the C sources.The source membership function assigns each time-frequency bin to the source with the highest energy.
- 2.1. Model: Each source mask is estimated from the similarity between every embedding and its attractor, using an inner product followed by a sigmoid or softmax.The resulting mask is larger when a time-frequency embedding is closer to that source's attractor, and softmax can enforce masks summing to one.
- 2.1. Model: An L2 reconstruction error compares masked signals with clean references, directly optimizing embeddings for source separation.This objective makes the proposed system an end-to-end deep attractor network, or DANet.
- 2.2. Results: DANet removes DC's stepwise pre-training and remains flexible to mixtures with varying numbers of sources.This flexibility follows from forming attractors from embedding affinities rather than fixing output patterns.
- 2.3. Estimation of attractor points: At test time, attractors can be estimated with post-processing K-means or by using their relatively stable locations as fixed attractor points.Alternative estimation methods include weighted averages that emphasize salient time-frequency bins and learned representative embeddings.
- 2.4. Relation with DC and PIT: The DC objective becomes a special case of the attractor network after label-based averaging, with equivalence at the optimum V V^T = Y Y^T.The transformation uses Y^T and the normalizer U = (Y^T Y)^-1 to form source centers and compare embeddings with them.
3. EVALUATION
The attractor network was evaluated for WSJ0 simultaneous speech separation using SDR, SAR, and SIR, with experiments examining embeddings, attractor formation, real-time inference, and three-speaker separation.
- Experimental setup: The evaluation used WSJ0 mixtures generated from 30 hours of training data, 10 hours of validation data, and a 5-hour evaluation set involving 16 unseen speakers.Mixtures were created at randomly selected SNRs between 0 dB and 10 dB.
- Experimental setup: Performance was reported using signal-to-distortion ratio (SDR), signal-to-artifacts ratio (SAR), and signal-to-interference ratio (SIR).SDR was defined as scale-invariant SNR in this evaluation.
- Separation examples: DANet formed symmetric attractor centers that separated the two speakers into distinct embedding regions with a clear boundary between them.Figure 2 visualized T-F bins by relative speaker power and marked attractor locations with X.
- Attractor structure: The network appeared to learn two attractor pairs across 10,000 mixtures, suggesting unsupervised discovery of different numbers of attractors.The pairs were labeled A1 and A2, with four symmetric centers visible after PCA visualization.
- Comparative results: Thresholding T-F samples used to form attractors improved performance beyond the plain DANet, while curriculum training produced the best overall performance.The curriculum continued training with 400-frame input, and the thresholding gain was attributed to increased influence from salient segments.
- Comparative results: Fixed attractor vectors generalized well to unseen mixtures and enabled frame-by-frame real-time processing, although they were slightly worse than the best model.Compared with K-means, the fixed-attractor approach required no post-processing.
- Comparative results: DANet outperformed the deep clustering baseline in the three-speaker separation experiment.The reported explanation is that DANet focuses on signal reconstruction, whereas deep clustering was trained to estimate a binary mask.
4. CONCLUSION
The deep attractor network forms embedding-space attractors to produce soft separation masks, supporting end-to-end and real-time separation across different numbers of sources.
- The framework forms attractor points in a high-dimensional signal-embedding space and converts attractor–embedding similarities into soft separation masks.
- The proposed system supports end-to-end, real-time separation and can operate with different numbers of mixing sources.
- Table 1 reports evaluation metrics for networks with different configurations, while Table 2 evaluates three-speaker separation.