PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks

Morten Kolbæk, Dong Yu, Zheng-Hua Tan, Jesper Jensen

arXiv:1703.06284v2cs.SDcs.LGeess.AS

TL;DR

Speaker-independent multi-talker separation remains difficult because frame-level PIT leaves an inference-time permutation problem. The paper introduces utterance-level PIT with recurrent networks to align each speaker’s frames consistently, and reports favorable results across datasets, speakers, languages, and mixture sizes.

  • Problem

    Speaker-independent multi-talker separation is limited by label permutation and, with frame-level PIT, an unresolved inference-time speaker-tracing problem.

  • Method

    uPIT extends PIT with an utterance-level training criterion in deep LSTM RNNs, forcing frames from each speaker onto the same output stream.

  • Results

    uPIT outperforms NMF- and CASA-based techniques, compares favorably with DPCL and DANet, generalizes to unseen speakers and languages, and handles two- and three-speaker mixtures with one model.

  • Takeaways & Limitations

    uPIT provides a practically applicable end-to-end separation approach without clustering, attractor estimation, or additional speaker tracing during inference.

Abstract

from arXiv · show

In this paper we propose the utterance-level Permutation Invariant Training (uPIT) technique. uPIT is a practically applicable, end-to-end, deep learning based solution for speaker independent multi-talker speech separation. Specifically, uPIT extends the recently proposed Permutation Invariant Training (PIT) technique with an utterance-level cost function, hence eliminating the need for solving an additional permutation problem during inference, which is otherwise required by frame-level PIT. We achieve this using Recurrent Neural Networks (RNNs) that, during training, minimize the utterance-level separation error, hence forcing separated frames belonging to the same speaker to be aligned to the same output stream. In practice, this allows RNNs, trained with uPIT, to separate multi-talker mixed speech without any prior knowledge of signal duration, number of speakers, speaker identity or gender. We evaluated uPIT on the WSJ0 and Danish two- and three-talker mixed-speech separation tasks and found that uPIT outperforms techniques based on Non-negative Matrix Factorization (NMF) and Computational Auditory Scene Analysis (CASA), and compares favorably with Deep Clustering (DPCL) and the Deep Attractor Network (DANet). Furthermore, we found that models trained with uPIT generalize well to unseen speakers and languages. Finally, we found that a single model, trained with uPIT, can handle both two-speaker, and three-speaker speech mixtures.

I. INTRODUCTION

The paper frames speaker-independent multi-talker separation as an unsolved machine version of the cocktail party problem and introduces uPIT to address frame-level permutation instability. The proposed approach avoids separate speaker tracing and clustering during inference while generalizing across speakers, languages, and mixture sizes.

  • Motivation: Speaker-independent multi-talker separation remains a difficult, broadly useful machine solution to the cocktail party problem.Applications include meeting transcription, captioning, multi-party interaction, and hearing aids.
  • Prior approaches: CASA groups time-frequency units using hand-engineered cues such as pitch, onset/offset, and periodicity.The grouped units are used to extract speakers from the mixture.
  • Prior approaches: Deep-learning separation has mainly succeeded under closed-set speaker conditions, while label permutation limits speaker-independent systems.Only a small number of deep-learning works had addressed the harder speaker-independent task.
  • Proposed method: uPIT extends PIT with an utterance-level criterion that aligns frames from each speaker to one output stream.Deep LSTM RNNs minimize utterance-level separation error, removing the need for additional speaker tracing or very large contexts.
  • Proposed method: uPIT models require no clustering or attractor estimation during inference, unlike DPCL and DANet.The associated training cost is negligible relative to RNN computation and zero during inference.
  • Results: Experiments on WSJ0 and Danish two- and three-talker mixtures found favorable comparisons with established methods and generalization to unseen speakers and languages.A single uPIT model also handled both two-speaker and three-speaker mixtures.
  • Problem formulation: Monaural separation estimates individual source signals from a linearly mixed single-microphone signal, typically through time-frequency representations.The paper targets reverberated source signals when the observations are reverberated.
  • Problem formulation: STFT-based systems estimate source spectra, then use inverse DFT and overlap-add to reconstruct time-domain signals.In typical setups, the mixed-speech phase is reused because phase estimation remains unresolved.

III. MASKS AND TRAINING CRITERIA

The paper extends masks developed for single-talker speech enhancement to multi-talker speech separation, using them as intermediate targets for estimating source magnitude spectra.

  • III. MASKS AND TRAINING CRITERIA: Mask-based separation estimates masks as an intermediate step toward estimating the magnitude spectra of individual sources.The paper applies three established single-talker speech-enhancement masks to multi-talker separation.

A. Ideal Ratio Mask

The Ideal Ratio Mask provides a commonly used mask target and an upper performance bound, but it depends on source information unavailable during separation.

  • A. Ideal Ratio Mask: The Ideal Ratio Mask is constrained so source masks sum to one across all time-frequency units.A softmax activation can satisfy this constraint.
  • A. Ideal Ratio Mask: The Ideal Ratio Mask reaches highest SDR with mixed-speech phase when all sources share the same phase, an assumption generally invalid.This limits its interpretation as a practical achievable target.
  • A. Ideal Ratio Mask: The Ideal Ratio Mask is not a desirable practical target because source magnitudes are unknown during separation.The paper nevertheless reports it as an upper performance bound.

B. Ideal Amplitude Mask

The Ideal Amplitude Mask can recover source magnitudes from mixture magnitudes, but its ideal phase assumption is generally unmet and its values may exceed one.

  • B. Ideal Amplitude Mask: The Ideal Amplitude Mask is defined as the source magnitude divided by the mixture magnitude.It is also called the FFT-mask or Amplitude Mask when estimated by a deep-learning model.
  • B. Ideal Amplitude Mask: Ideal Amplitude Masks can construct exact source magnitudes from mixture magnitudes, but highest SDR requires source and mixture phases to match.That phase condition is not satisfied in most cases.
  • B. Ideal Amplitude Mask: Ideal Amplitude Masks range from zero to infinity, although most time-frequency units empirically lie between zero and one.Softmax, sigmoid, and ReLU are therefore possible output activations.

C. Ideal Phase Sensitive Mask

The Ideal Phase Sensitive Mask incorporates phase differences between each source and the mixture, addressing a limitation of magnitude-only masks. Its non-negative variant clips empirically near-zero negative values before mask estimation.

  • IPSM accounts for phase differences between source and mixture, unlike IRM and IAM, which can yield suboptimal reconstruction when mixture phase is reused.
  • IPSM sums to one because its phase-correcting term uses the source–mixture phase difference.
  • IPSM is smaller than IAM when the phase difference between mixed speech and a source is large.
  • Approximately 20% of IPSMs are negative, but these values are usually very close to zero.
  • INPSM addresses near-zero negative IPSMs by defining a non-negative phase-sensitive target.
  • Softmax, Sigmoid, tanh, and ReLU can estimate IPSM and INPSM targets; a learned IPSM is called a PSM.

D. Training Criterion

The training criterion optimizes estimated masks or reconstructed magnitudes against source-dependent targets, while phase-sensitive targets incorporate mixture–source phase differences. The conventional model maps mixture features to masks for each talker, but mask-target training has important limitations.

  • Mask-based separation estimates source magnitudes by multiplying predicted masks with the mixture magnitude spectrum.
  • Mask-target MSE is ill-defined in silence and does not directly measure reconstructed-source error.
  • Recent methods instead minimize MSE between estimated and true magnitudes, making silence segments irrelevant to mask-estimation accuracy.
  • With PSMs, training targets are replaced by phase-discounted targets, and the IPSM is the task’s achievable upper bound under the corresponding cost function.
  • The conventional model feeds J mixture-feature frames into a DNN, CNN, or LSTM RNN to generate M mask frames for each talker.
  • For two talkers, the model reconstructs each source by multiplying its predicted mask elementwise with the mixture representation.

B. The Label Permutation Problem

The label permutation problem arises because output masks lack a known speaker ordering, causing frame-level assignments to become ambiguous during training and inference. PIT makes training permutation-invariant, but frame-level PIT can still require speaker tracing across output meta-frames.

  • The Label Permutation Problem: Unknown output-mask ordering makes the training error assignment ambiguous.The model estimates masks simultaneously from the same mixture, so the correspondence between references and outputs is not known in advance.
  • The Label Permutation Problem: Constant output assignments fail when training includes many speakers of both genders.Gender-based conventions can work for simple female–male mixtures but do not generalize to mixed-speaker training sets.
  • The Label Permutation Problem: Prior speaker-independent separation systems perform poorly because of the label permutation problem.The problem is also called label ambiguity in prior work.
  • Permutation Invariant Training: PIT treats reference signals as an unordered set, producing the same training result regardless of reference ordering.It computes pairwise mean-squared errors, evaluates candidate permutations, and optimizes the lowest permutation loss.
  • Permutation Invariant Training: Frame-level PIT keeps permutations consistent within each output meta-frame but may change them across meta-frames.Reconstructing speakers therefore requires either assuming a fixed permutation or adding a speaker-tracing algorithm.

V. UTTERANCE-LEVEL PIT

The paper presents uPIT to solve the tracing problem by selecting one permutation for an entire utterance rather than recomputing it for each meta-frame. It uses recurrent models suited to variable-length signals and evaluates separation quality with SDR and PESQ.

  • V. UTTERANCE-LEVEL PIT: Adjacent-meta-frame tracing can compound one early assignment error across all later frames.A separate tracing step also complicates the overall model.
  • V. UTTERANCE-LEVEL PIT: uPIT extends frame-level PIT with an utterance-level cost function to solve both tracing and label permutation problems.The method selects the permutation minimizing utterance-level separation error.
  • V. UTTERANCE-LEVEL PIT: uPIT applies the minimum-error permutation to every frame in an utterance.This avoids the changing-permutation issue of original PIT, whose optimal permutation is computed separately for each output meta-frame.
  • V. UTTERANCE-LEVEL PIT: Deep LSTM and bidirectional LSTM RNNs are used with uPIT because utterances vary in length and separation requires long-range dependencies.Unlike the original PIT setup, uPIT uses N-unit input and output layers rather than explicitly adding contextual frames to those layers.
  • V. UTTERANCE-LEVEL PIT: The evaluation measures improvement in Signal-to-Distortion Ratio and Perceptual Evaluation of Speech Quality.Both SDR and PESQ are widely used metrics for speech-enhancement evaluation in multi-talker separation tasks.

A. Datasets

The experiments evaluate PIT and uPIT on two- and three-talker mixtures across WSJ0 and Danish datasets, using recurrent models for utterance-level training. Results show that PIT generalizes to unseen speakers, while uPIT improves default-assignment separation and benefits from phase-aware criteria and extended training.

  • Datasets: uPIT was evaluated on WSJ0-2mix, WSJ0-3mix, and Danish-2mix using 129-dimensional STFT magnitude spectra.The spectra used 8 kHz sampling, 32 ms frames, and 16 ms frame shifts.
  • Datasets: The Danish-2mix open-condition test set uses speakers unseen during training, while validation supports hyperparameter selection and closed-condition evaluation.Training and validation use 45 male and 45 female speakers each; the open-condition test set uses another 45 male and 45 female speakers.
  • PIT experiments: 7.5 dB SDR improvement is achieved by frame-level PIT with default assignment on two-talker mixtures.Reducing output-window size improves optimal-assignment performance but makes speaker tracing more important because permutations change more frequently.
  • PIT experiments: PIT generalizes well to unseen speakers because open- and closed-condition performance is very close.The comparison is reported for WSJ0-2mix experiments with different frame-level PIT configurations.
  • uPIT experiments: uPIT experiments use LSTM and BLSTM RNNs because utterance-level training exploits long-range signal dependencies.The evaluated models use three recurrent layers, while training uses dropout and utterance-level minibatches.

1) uPIT Training Progress:

uPIT training steadily reduces both training and validation error while improving default-assignment separation over original PIT. The gains arise from the utterance-level criterion, with phase-aware training and two-stage modeling providing additional improvements.

  • uPIT Training Progress: Both training and validation MSE steadily decrease during uPIT training, indicating that uPIT effectively solves the label permutation problem.The progress is shown for BLSTM training on two-talker mixtures using the utterance-level cost.
  • uPIT Performance for Different Setups: 9.4 dB SDR improvement is achieved on both closed- and open-condition sets with uPIT default assignment.This compares favorably with 7.6 dB on closed condition and 7.5 dB on open condition from deep CNNs combined with PIT.
  • uPIT Performance for Different Setups: The 9.4 dB result comes from the utterance-level training criterion rather than BLSTM architecture alone.PIT-BLSTM has much worse default-assignment results despite achieving the best optimal-assignment results among the compared models.
  • uPIT Performance for Different Setups: PSM and NPSM training criteria improve SDR over the amplitude-mask criterion, indicating a benefit from including phase information.Phase information is used explicitly by PSM and NPSM and implicitly through the cosine term in the uPIT objective.
  • Two-stage Models and Reduced Dropout Rate: Two-stage models consistently outperform the corresponding reduced-dropout single-stage models, reaching 10 dB SDR improvement overall.The second-stage model is trained with the original input features and the first-stage estimated mask.

4) Opposite Gender vs. Same Gender.:

Across gender, language, and benchmark comparisons, uPIT achieves strong separation performance while avoiding the clustering or attractor-estimation stages required by competing methods.

  • Opposite Gender vs. Same Gender.: uPIT achieves much higher SDR improvement for opposite-gender than same-gender WSJ0-2mix speech, although gender is not explicitly used.The results indicate that same-gender mixtures are harder to separate.
  • Multi-Language Models: A model trained on English WSJ0-2mix generalizes well to unseen Danish speech, with SDR close to ideal ratio mask values.The PESQ gap relative to the ideal ratio mask is larger than the SDR gap.
  • Summary of Multiple 2-Speaker Separation Techniques: uPIT improves on PIT and achieves comparable performance with DPCL+, DPCL++, and DANet on WSJ0-2mix.Unlike these competing systems, uPIT and PIT require neither clustering nor attractor estimation during inference.
  • Training Behavior: Increasing model size decreases both training and validation MSE on WSJ0-3mix.The validation MSE remains slightly higher than the training MSE.
  • Three-Talker Separation: On WSJ0-3mix, larger and further-trained uPIT configurations achieve higher SDR than DPCL++ and similar SDR to DANet without curriculum training.The basic 896-unit uPIT-BLSTM already compares favorably with DPCL++.

E. Combined Two- and Three-Talker Speech Separation

Three-speaker uPIT models can operate on both two- and three-speaker mixtures, including mixtures with an unknown active-speaker count. This flexibility preserves strong performance while avoiding prior knowledge of the number of speakers at test time.

  • Combined Two- and Three-Talker Speech Separation: Three-speaker uPIT-BLSTM models were evaluated on both WSJ0-2mix and WSJ0-3mix datasets.The two-speaker training condition used a third silent channel, and active streams were identified by energy during evaluation.
  • Combined Two- and Three-Talker Speech Separation: uPIT-BLSTM achieves good, slightly worse performance than corresponding specialized two- and three-speaker models.The comparison is reported for models trained across both mixture types.
  • Combined Two- and Three-Talker Speech Separation: The uPIT-BLSTM-ST model outperforms corresponding specialized models, indicating that one model can handle varying and unknown numbers of speakers.This removes the need for a priori speaker-count knowledge at test time required by DPCL++ and DANet.
  • Output-Stream Behavior: In 2999 of 3000 WSJ0-2mix test mixtures, output streams one and two were the most energetic, supporting a constant output-mask permutation.Those streams averaged approximately 33 dB more energy than the silent channel; a three-speaker example showed the third stream 63 dB lower than the active streams.
  • Conclusion and Discussion: uPIT handles the label permutation problem, generalizes across speakers and languages, and does not require prior knowledge of speaker count.The conclusion also reports better performance for bidirectional than unidirectional LSTMs and for phase-sensitive than amplitude masks.
  • Conclusion and Discussion: uPIT is algorithmically simpler than DPCL and DANet while achieving performance on par with DPCL and comparable to DANet.DPCL and DANet use separate embedding and clustering stages during inference.
Loading 1703.06284v2…