PaperlayerView exact revision on arXiv

Source-linked AI summary

Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation

Yuzhou Liu, DeLiang Wang

arXiv:1904.11148v1cs.SDcs.LGeess.AS

TL;DR

Talker-independent monaural speaker separation remains difficult because output-speaker pairing is nontrivial and frame-level separation can suffer assignment errors. The paper uses deep CASA to separate spectra frame by frame and then track speakers sequentially, reducing assignment errors and improving separation metrics.

  • Problem

    Talker-independent training makes output-speaker pairing nontrivial, while monaural speaker separation remains challenging for machines.

  • Method

    Deep CASA divides separation into simultaneous frame-level grouping with permutation-invariant training and sequential speaker grouping using embeddings and a clustering network.

  • Results

    Deep CASA cuts frame assignment error by half compared with uPIT models, while joint optimization adds 0.1 dB ∆SDR, 0.02 PESQ, and 0.3% ESTOI.

  • Takeaways & Limitations

    Separating frame-level separation from speaker tracking improves both objectives relative to uPIT-based systems.

Abstract

from arXiv · show

We address talker-independent monaural speaker separation from the perspectives of deep learning and computational auditory scene analysis (CASA). Specifically, we decompose the multi-speaker separation task into the stages of simultaneous grouping and sequential grouping. Simultaneous grouping is first performed in each time frame by separating the spectra of different speakers with a permutation-invariantly trained neural network. In the second stage, the frame-level separated spectra are sequentially grouped to different speakers by a clustering network. The proposed deep CASA approach optimizes frame-level separation and speaker tracking in turn, and produces excellent results for both objectives. Experimental results on the benchmark WSJ0-2mix database show that the new approach achieves the state-of-the-art results with a modest model size.

I. INTRODUCTION

Monaural speaker separation remains difficult for machines, especially with competing speakers and unknown talkers. The paper combines CASA-inspired grouping with deep learning, separating frame-level spectra before tracking speakers over time.

  • Monaural speaker separation targets competing speakers recorded by one microphone and supports applications including speech recognition, speaker identification, and hearing aids.
  • CASA decomposes separation into simultaneous grouping of overlapping time-frequency units and sequential grouping of segments into auditory streams.
  • Talker-dependent neural separation improves intelligibility but does not generalize to untrained speakers.
  • Frame-level PIT improves local separation but can swap speaker assignments across frames, while deep clustering avoids permutation ambiguity through affinity-matrix training.
  • The proposed deep CASA approach improves frame-level separation and speaker tracking by optimizing simultaneous and sequential grouping in turn.
  • The approach improves on uPIT and the preliminary system in both frame-level separation and speaker tracking.

II. MONAURAL SPEAKER SEPARATION AND PERMUTATION INVARIANT TRAINING

Monaural separation estimates multiple speech signals from one mixture, commonly in the time-frequency domain. The central training challenge is assigning unordered output streams to target speakers without creating conflicting gradients.

  • Monaural speaker separation estimates independent speech signals from a single-channel mixture, with this work focusing on two concurrent speakers.
  • Time-frequency systems use mixture STFT features to predict a mask for each speaker and multiply those masks by the mixture to estimate sources.
  • Estimated magnitude spectra can be combined with noisy mixture phase and resynthesized into separated waveforms using inverse STFT.
  • Phase-sensitive approximation accounts for errors introduced by noisy phase by targeting |Xc(t, f)| ⊙cos(φc(t, f)).
  • Talker-independent training creates a permutation problem because inconsistent output-speaker pairings can generate conflicting gradients and prevent convergence.

B. Permutation Invariant Training

Permutation-invariant training addresses unordered speaker outputs by selecting the pairing that minimizes loss. Frame-level and utterance-level variants trade off local separation quality against stable speaker assignment.

  • Frame-level PIT selects the lowest-loss speaker permutation independently for each frame.
  • tPIT separates speakers well at the frame level, but output assignments may swap frequently across frames.
  • uPIT fixes output-speaker pairing across an utterance using the permutation with minimum utterance-level loss.
  • uPIT preserves default output assignment better than tPIT but underperforms tPIT after optimal reassignment and tracks same-gender speakers poorly.

III. DEEP CASA APPROACH TO MONAURAL SPEAKER SEPARATION

Deep CASA divides monaural separation into frame-level simultaneous grouping and temporal sequential grouping. It combines tPIT-based spectral separation with a second-stage mechanism that organizes estimates by speaker while addressing phase reconstruction.

  • The deep CASA framework first separates speakers’ spectral components frame by frame, then streams estimates belonging to each speaker across time.
  • The baseline simultaneous-grouping system uses mixture magnitude STFT input, a BLSTM, and tPIT training before passing frame-level estimates onward.
  • Complex ratio masking reconstructs sources in the complex STFT domain, and complex approximation compares reconstructed and clean sources using real and imaginary components.
  • The framework organizes frame-level estimates by minimum loss, evaluates candidate domains for organization, and then computes utterance-level SNR after inverse STFT.

3) Convolutional neural networks for simultaneous grouping:

The proposed Dense-UNet uses an encoder–decoder CNN with dense blocks, skip connections, and frequency mapping to improve simultaneous grouping for speech separation.

  • Dense-UNet is proposed for simultaneous grouping, motivated by DenseNet and UNet successes in music source separation.
  • The UNet alternates convolutional and downsampling layers in its encoder with convolutional and upsampling layers in its decoder.Skip connections connect corresponding encoder and decoder levels to preserve raw information.
  • The network accepts magnitude or complex STFT inputs, masks, and outputs.
  • Dense-UNet replaces standard convolutional layers with densely connected CNN blocks whose layers use feed-forward connections.The architecture alternates nine dense blocks with four downsampling and four upsampling layers.
  • A frequency mapping layer projects frequency-dependent CNN outputs into an organized space with fewer parameters.One CNN layer in each dense block is replaced by this frequency mapping layer.

1) Baseline system:

The baseline sequential grouping stage tracks speakers by embedding frame-level separated spectra and clustering frames according to their optimal speaker assignments.

  • Sequential grouping clusters frame-level spectral estimates across time to extend deep clustering from T-F embeddings to speaker tracking.
  • The system embeds each frame into a D-dimensional unit-length vector and trains against a two-dimensional indicator of the optimal output assignment.The indicator is determined by which output-speaker pairing minimizes the tPIT loss for that frame.
  • The permutation-independent objective brings embeddings for frames with the same assignment closer and separates embeddings with different assignments.
  • Training weights frames by the normalized absolute difference between their two assignment losses.The weighting emphasizes high-energy frames where the two separated outputs differ substantially.
  • At inference, K-means clusters frame embeddings into two groups, outputs are organized by cluster labels, and iSTFT reconstructs time-domain signals.

2) Temporal convolutional networks for sequential grouping:

The sequential grouping network uses a TCN with dilated convolutions for long temporal context and introduces dropDilation to regularize its temporal connections.

  • TCNs replace RNNs in sequential grouping and use stacked dilated convolutions to provide very long memory.The study adopts a TCN similar to TasNet.
  • The TCN preprocesses features with a dense CNN, 1 × 1 convolution, and layer normalization before exponentially dilated convolutional blocks.Dilation factors increase as 2^0, 2^1, ..., 2^(M−1) to exploit large temporal contexts.
  • Overfitting is a concern because sequence models can memorize training patterns and become trapped in local minima.
  • DropDilation randomly drops dilated connections in depthwise dilated convolutions while preserving direct connections.The method is designed to regularize the TCN while retaining local information.
  • Evaluation uses WSJ0-2mix with 30 hours of training data, 10 hours of validation data, and a 5-hour open-condition test set.
  • Performance is measured with ΔSDR, PESQ, ESTOI, and ΔSI-SNR for separation, quality, intelligibility, and comparison with competing systems.

B. Models

The experiments compare BLSTM and Dense-UNet models for simultaneous grouping, and BLSTM and TCN models for sequential grouping under specified training configurations.

  • Simultaneous grouping models: Simultaneous grouping evaluates BLSTM and Dense-UNet models.
  • Simultaneous grouping models: The simultaneous-grouping BLSTM has three BLSTM layers with 896×2 units each, while Dense-UNet uses dense blocks with K = 64 and L = 5.
  • Simultaneous grouping models: Magnitude-STFT inputs use ELU outputs for phase-sensitive mask estimation, whereas complex objectives use real and imaginary STFT inputs with linear outputs.
  • Simultaneous grouping models: Both simultaneous-grouping networks use Adam, dropout regularization, validation-based learning-rate adjustment, and early stopping.
  • Sequential grouping models: Sequential grouping evaluates BLSTM and TCN models trained on top of a tuned simultaneous-grouping model.
  • Sequential grouping models: The TCN uses a maximum dilation factor of 2^6 = 64, a theoretical receptive field of 8.128 seconds, B = 256, H = 512, and dropDilation with p = 0.7.
  • Sequential grouping models: Both sequential-grouping models use dense CNN preprocessing, D = 40 embedding dimensions, Adam optimization, learning-rate adjustment, and early stopping.

3) One stage uPIT models:

The evaluated one-stage uPIT models use SNR objectives with a uPIT training criterion, following the specified training recipes.

  • 3) One stage uPIT models:: The study evaluates a Dense-UNet and a TCN trained with SNR objectives and the uPIT training criterion JuP IT −SNR.Other training recipes follow those described earlier in the paper.

C. Results and Comparisons

The experiments compare deep CASA with tPIT and uPIT across frame-level separation, speaker assignment, sequential grouping, assignment errors, and gender conditions. Deep CASA combines strong frame separation with improved tracking, while performance varies with objectives, architectures, and speaker gender.

  • Simultaneous grouping: 4.7 million trainable parameters accompany Dense-UNet’s substantial performance gain over BLSTM in simultaneous grouping.Frequency mapping adds 0.3 dB ∆SDR, 0.1 PESQ, and 0.8% ESTOI while reducing parameters by 0.9 million.
  • Simultaneous grouping: The complex STFT objective substantially improves separation, while the SNR objective further outperforms the CA objective.The resulting tPIT Dense-UNet trained with JtP IT −SNR is adopted for subsequent simultaneous-grouping evaluations.
  • tPIT and uPIT comparison: uPIT improves default output assignment over tPIT but has significantly worse optimal-assignment performance because it does not optimize frame-level loss.For a same-gender mixture, uPIT also makes several default-assignment mistakes and exhibits mixed frequency patterns in some frames.
  • Sequential grouping: TCN substantially outperforms BLSTM with around 8 million parameters, and dropDilation adds a 0.5 dB ∆SDR gain over conventional dropout.uPIT Dense-UNet and uPIT TCN underperform the deep CASA systems, including after optimal reassignment for uPIT Dense-UNet.
  • Overall comparison: Deep CASA cuts frame assignment error by half compared with uPIT models while achieving better frame-level separation and speaker tracking in turn.Across gender combinations, both systems perform better on male-female mixtures than same-gender mixtures, with the largest gap for female-female mixtures.
  • Objective analysis: SNR-trained tPIT Dense-UNet can produce striped time-frequency patterns because neighboring frames overlap by 75%, complicating default-output organization.Sequential grouping labels nearly match optimal labels in speech-dominant frames, but organizing the outputs can still yield substantially different magnitude STFTs.

V. CONCLUDING REMARKS

The deep CASA approach separates talkers through simultaneous grouping followed by sequential grouping, optimizing frame-level separation and speaker tracking in turn. It achieves state-of-the-art results on WSJ0-2mix with a modest model size while offering lower clustering complexity and extension to more speakers.

  • Deep CASA approach: Simultaneous grouping separates two speakers at the frame level, then sequential grouping streams the separated spectra into two sources.The two-stage design corresponds to the proposed deep CASA pipeline.
  • Experimental results: The proposed algorithm achieves state-of-the-art results on WSJ0-2mix with a modest model size.
  • Comparison with deep clustering: Deep CASA operates on frame-level outputs and reduces clustering complexity from O(FT) in deep clustering to O(T).The sequential stage can use magnitude STFTs, complex masks, or time-domain signals.
  • Comparison with deep clustering: Unlike deep clustering, the approach combines clustering-based speaker tracking with frame-level separation in a divide-and-conquer framework.The paper motivates this split because deep clustering is not better than ratio masking for frame-level separation.
  • Extension: Although formulated for two speakers, deep CASA can be extended to three or more by adding output layers and assigning frame-level embeddings with constrained K-means.
Loading 1904.11148v1…