Source-linked AI summary
Voice Separation with an Unknown Number of Multiple Speakers
Eliya Nachmani, Yossi Adi, Lior Wolf
TL;DR
The paper addresses supervised single-microphone separation of mixtures with multiple unseen speakers, including cases where the speaker count is unknown. It introduces recurrent mask-free processing with multiple losses and speaker-consistency modeling, alongside model selection for speaker count. The reported results outperform existing methods by a sizable gap, while performance degrades more gradually as the number of speakers increases.
Problem
Supervised single-channel separation must separate mixtures containing multiple unseen speakers, while maintaining consistent speaker assignments and supporting an unknown speaker count.
Method
The method uses a mask-free sequence of bidirectional recurrent blocks, compound multi-step losses, a voice-identity loss, and activity-detector-based speaker-count selection.
Results
The method achieves better results than existing methods by a sizable gap, with more gradual degradation as the number of speakers increases.
Takeaways & Limitations
The approach provides a practical solution for voice separation with an unknown number of speakers in a setting where previous methods degrade rapidly as speaker count increases.
Takeaways & Limitations
The identity loss does not completely eliminate output flipping between different sources.
Abstract
from arXiv · showhide
We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.
1. Introduction
The paper presents a supervised single-microphone separation method designed to handle mixtures with an unknown number of speakers. It combines recurrent processing, multi-step training losses, speaker-consistent outputs, and model selection to improve separation as speaker count grows.
- Problem: The method targets supervised single-channel separation of mixtures containing multiple unseen speakers.Training uses paired mixtures and individual voices to separate novel mixtures from a single microphone.
- Model and training: A mask-free sequence of RNNs processes the audio, with reconstruction error evaluated after each RNN through a compound loss.This addresses the increasing difficulty of extracting and suppressing more content from partial representations as speaker count increases.
- Model and training: Each recurrent block uses two bidirectional RNNs in parallel, element-wise multiplication, and a bypass connection carrying the layer input.The layer output concatenates the multiplied recurrent outputs with the skip-connected input.
- Speaker consistency: A voice-representation loss compares output voices with channel identities, while averaging separations initialized at multiple audio locations provides an additional improvement.The identity-based loss addresses speaker switching between output channels, particularly during transient silence episodes.
- Unknown speaker count: The method trains one model per speaker count and selects the number of speakers with an activity detector.The selection procedure enables operation when the mixture's speaker count is unknown.
- Results: Performance improves over published methods by a sizable margin, with the gap increasing as the number of speakers increases and degradation remaining gradual.Baseline methods instead show sharp degradation as speaker count rises.
2. Model
The model encodes and chunks a single-channel mixture, processes the resulting tensor with alternating bidirectional RNN blocks, and decodes intermediate representations into output channels. Training combines permutation-invariant SI-SNR reconstruction at multiple stages with an identity loss that encourages consistent speaker assignments across channels.
- 2.1. Model Description: The encoder converts the mixture waveform x into an N-dimensional latent representation z using a 1-D convolution.The convolution uses kernel size L and stride L/2, followed by a ReLU activation.
- 2.1. Model Description: The latent representation is divided into overlapping chunks and concatenated into a 3-D tensor for separation processing.The chunks have length K and hop size P, producing v ∈ R^N×K×R.
- 2.1. Model Description: Alternating RNN blocks process the tensor along time-dependent and chunking dimensions to capture short-term and long-term representations.The model uses bidirectional RNNs with MULCAT blocks, whose parallel outputs are multiplied element-wise and combined with the input through a skip connection.
- 2.2. Training Objective: After each pair of RNN blocks, a convolution decodes the activations into C output channels, which are reconstructed with overlap-and-add.The multi-scale loss therefore evaluates reconstruction after each pair of blocks.
- 2.2. Training Objective: The training objective applies utterance-level permutation-invariant SI-SNR loss to unordered outputs and an identity loss to matched channels.The identity loss uses a speaker-recognition network’s embeddings to impose long-term consistency on output-stream speaker identities.
- 2.3. Model Selection: A learning-free activity-detector procedure selects the speaker count by moving to models with fewer output channels when a channel is silent.The resulting unknown-speaker-count selection is relatively accurate and only moderately worse than using the known count.
3. Experiments
Experiments evaluate the proposed separator across speaker counts, kernel sizes, ablations, noisy and reverberated conditions, test-time augmentation, unknown-speaker selection, and music separation. The model outperforms prior methods across four speaker-count datasets and remains competitive across additional settings.
- Speaker-count evaluation: The proposed model is superior to previous methods by a sizable margin across all four datasets with different numbers of speakers.Each dataset is evaluated using a model trained for its corresponding speaker count.
- Ablation analysis: Each ablated component contributes to the performance gain, with the multi-layer loss contributing more than the other tested components.The ablation removes gating, multiloss, or identity loss; adding identity loss to DPRNN also improves performance.
- Kernel-size analysis: Kernels with L > 2 do not harm the model’s performance and lead to faster convergence toward results in the range of recently published methods.Training curves compare L = 2, 4, 8, 16 over the first 60 hours of training.
- Robustness evaluation: The proposed approach is superior to baseline methods under noisy and reverberated conditions on WHAM! and WHAMR! benchmarks.Comparisons include Conv-TasNet, Chimera++, learnable filter bank, and DPRNN using SI-SNRi.
- Test-time augmentation: Random time shifts improve performance, while results reported elsewhere use no such test-time augmentation for direct literature comparison.The results from multiple starting points are averaged.
- Unknown speaker counts: Silent output channels identify the number of speakers in many cases while maintaining SI-SNRi close to oracle performance.The method selects the speaker-count model using silence detection and compares it with known-speaker-count oracle results.
- Music source separation: The proposed model achieves the best overall SDR on the MusDB music source-separation benchmark, with the largest improvement in the Vocals category.The reported metric is SDR, where higher values are better.
4. Related Work
Prior work spans spectrum-domain, time-domain, clustering, speaker-embedding, recurrent, convolutional, and multichannel approaches to speech separation. Related methods include permutation-invariant training, beamforming, and recursive separation for increasing speaker counts.
- Earlier speech separation research traditionally applied signal-processing techniques and often operated on input spectra.
- Deep learning approaches introduced clustering, speaker embeddings, permutation-invariant training, and time-domain encoder–separator–decoder architectures.
- Conv-TasNet replaced recurrent separation with fully convolutional blocks, while later work used speaker clustering, ensembles, and dual-path recurrent blocks.
- Recursive separation was proposed for multiple speakers, and a model trained on two and three speakers was shown to separate four speakers.
- The chimera network combined mask-inference and deep-clustering networks, improving WSJ0-2mix performance by 0.7 dB over alternative methods.
- Multichannel methods estimate beamforming filters, with FaSNet improving MVDR baseline results by 14.3% relative word error rate reduction.
5. Conclusions
The paper frames cocktail-party speech separation as instance segmentation with occluding voices and addresses failures from silent speech intervals. It combines voice constancy with a recurrent, multi-loss separator and reports sizable gains over prior methods.
- The cocktail party problem is treated as difficult instance segmentation because overlapping speech instances cannot be separated by continuity alone.
- Silent parts in speech motivate an identification-based constancy loss, which also detects the number of instances in a mixed signal.
- The proposed practical solution combines a recurrent block with two bidirectional RNNs, a skip connection, multiple losses, and a voice constancy term.
- The method achieves better results than existing methods by a sizable gap in a rapidly evolving research domain.