PaperlayerView exact revision on arXiv

Source-linked AI summary

SpEx: Multi-Scale Time Domain Speaker Extraction Network

Chenglin Xu, Wei Rao, Eng Siong Chng, Haizhou Li

arXiv:2004.08326v1eess.AScs.CLcs.SD

TL;DR

Speaker extraction seeks to focus on an unseen target speaker in multi-talker environments, but frequency-domain methods face phase-estimation difficulty. SpEx addresses this with an end-to-end time-domain architecture using speaker conditioning, multi-scale embeddings, and joint training, and significantly outperforms frequency-domain counterparts.

  • Problem

    Speaker extraction seeks to estimate a mask for a new speaker unseen during training in multi-talker environments, supporting applications such as hearing aids and speech recognition.

  • Method

    SpEx uses speaker and speech encoders, a speaker extractor, and a speech decoder in an end-to-end time-domain system with multi-scale encoding and joint multi-task training.

  • Results

    SpEx significantly outperforms frequency-domain counterparts; its multi-scale configuration reaches 13.9dB SDR, 13.3dB SI-SDR, and 3.00 PESQ.

  • Takeaways & Limitations

    The reference speech enables top-down voluntary focus on the target speaker, particularly when speakers are pre-registered for applications such as speaker verification.

Abstract

from arXiv · show

Speaker extraction aims to mimic humans' selective auditory attention by extracting a target speaker's voice from a multi-talker environment. It is common to perform the extraction in frequency-domain, and reconstruct the time-domain signal from the extracted magnitude and estimated phase spectra. However, such an approach is adversely affected by the inherent difficulty of phase estimation. Inspired by Conv-TasNet, we propose a time-domain speaker extraction network (SpEx) that converts the mixture speech into multi-scale embedding coefficients instead of decomposing the speech signal into magnitude and phase spectra. In this way, we avoid phase estimation. The SpEx network consists of four network components, namely speaker encoder, speech encoder, speaker extractor, and speech decoder. Specifically, the speech encoder converts the mixture speech into multi-scale embedding coefficients, the speaker encoder learns to represent the target speaker with a speaker embedding. The speaker extractor takes the multi-scale embedding coefficients and target speaker embedding as input and estimates a receptive mask. Finally, the speech decoder reconstructs the target speaker's speech from the masked embedding coefficients. We also propose a multi-task learning framework and a multi-scale embedding implementation. Experimental results show that the proposed SpEx achieves 37.3%, 37.7% and 15.0% relative improvements over the best baseline in terms of signal-to-distortion ratio (SDR), scale-invariant SDR (SI-SDR), and perceptual evaluation of speech quality (PESQ) under an open evaluation condition.

I. INTRODUCTION

Speaker extraction focuses on a specified speaker in mixtures with unknown or varying interference, using reference speech to direct top-down attention. SpEx introduces a time-domain, jointly trained, multi-scale approach that avoids phase estimation.

  • Motivation: Frequency-domain extraction commonly relies on magnitude and phase spectra, while phase estimation remains difficult.Prior approaches use spectro-temporal masking with frequency-domain representations.
  • Motivation: Speaker extraction uses reference speech from an unseen target speaker to direct attention and avoid requiring the mixture’s speaker count or resolving global permutation ambiguity.This setting supports applications including speaker verification, hearing aids, speech recognition, diarization, and voice surveillance.
  • Contributions: SpEx extends Conv-TasNet to speaker extraction with an end-to-end time-domain architecture.Its contributions include a speaker encoder, multi-task learning, and multi-scale encoding and decoding.
  • Contributions: The speaker encoder emulates top-down voluntary auditory focus by using the target speaker’s reference speech.The paper jointly trains this component within SpEx, unlike the independently trained speaker encoder in earlier work.

II. TIME-DOMAIN SPEAKER EXTRACTION NETWORK

The proposed network encodes mixture and reference speech separately, uses the target-speaker representation to estimate a mask, and decodes the masked representation into target speech. Its multi-task training jointly optimizes speaker characterization and extraction.

  • Network overview: A general speaker extraction network contains a speaker encoder, speech encoder, speaker extractor, and speech decoder.The speaker encoder represents reference speech; the speech encoder represents mixture speech; the extractor estimates a target mask; and the decoder reconstructs target speech.
  • Extraction pipeline: The speaker extractor combines mixture embedding coefficients with the target speaker embedding to estimate a receptive mask for target-speaker reconstruction.The speech decoder reconstructs the time-domain target signal from the masked representation.
  • Time-domain architecture: SpEx replaces STFT-based speech encoding with a trainable time-domain encoder that produces spectrum-like embedding coefficients.This time-domain design avoids the phase-estimation issue associated with frequency-domain reconstruction.
  • Multi-task learning: The speaker encoder is jointly optimized with cross-entropy speaker classification and signal reconstruction losses.This multi-task objective combines speaker discrimination with extraction-oriented reconstruction.

2) Speech Encoder:

The speech encoder replaces Fourier magnitude–phase decomposition with trainable time-domain, multi-scale embedding coefficients. These embeddings are normalized, processed through TCN-based extraction, and masked using target-speaker conditioning.

  • Time-domain encoding avoids separately handling real and imaginary components and uses trainable rather than predefined sine/cosine basis functions.
  • Three parallel 1-D CNNs encode mixture speech at different temporal resolutions into multi-scale embeddings.
  • Varying filter lengths provide complementary temporal resolutions, with short windows resolving higher frequencies and long windows resolving lower frequencies.
  • The speaker extractor conditions TCN processing on a repeatedly concatenated speaker embedding to estimate a receptive mask for each embedding scale.
  • Sigmoid outputs constrain each scale-specific mask to [0, 1] and match the encoded representation dimensions before element-wise masking.

4) Speech Decoder:

SpEx decodes each masked multi-scale response into a time-domain signal using learned de-convolutional bases. Training combines reconstruction objectives across scales, while inference evaluates individual and weighted combined outputs.

  • The decoder reconstructs time-domain signals s1, s2, and s3 from masked responses using learned bases at three scales.
  • Multi-scale SI-SDR loss weights reconstruction quality from all three decoded signals against the clean speech target.
  • The reconstructed signals are compared with the clean target using SI-SDR, with zero-mean normalization ensuring scale invariance.
  • At inference, SpEx evaluates s1, s2, and s3 separately and also forms a weighted sum, while the multi-scale loss is training-only.

C. Multi-task Learning

SpEx jointly trains its four components using signal reconstruction and speaker classification objectives. This extends Conv-TasNet’s reconstruction-focused pipeline with speaker-conditioned, multi-task optimization.

  • The speech encoder, speaker extractor, and speech decoder minimize multi-scale SI-SDR loss, while the speaker encoder also minimizes speaker-classification cross-entropy.
  • The classification objective uses true speaker labels and predicted speaker probabilities across Ns speakers.
  • The total loss is a weighted sum of reconstruction loss J1 and classification loss J2.
  • Unlike Conv-TasNet’s reconstruction-only training, SpEx includes a speaker encoder and conditions extraction on a target speaker.

1) Top-down voluntary focus:

SpEx uses a reference speech signal to focus extraction on a specified speaker, addressing unknown speaker counts and permutation ambiguity. Its evaluation uses simulated speaker-independent mixtures with varied reference durations and mixture sizes.

  • 1) Top-down voluntary focus:: A speaker encoder provides top-down voluntary focus, enabling SpEx to single out one voice instead of estimating masks for all competing speakers.
  • 1) Top-down voluntary focus:: Multi-task learning trains the speaker encoder with both signal reconstruction and speaker classification objectives.
  • 1) Top-down voluntary focus:: The multi-scale design is motivated by speech’s rich temporal structure and the expectation that multiple resolutions capture it in greater detail.
  • 1) Top-down voluntary focus:: The databases contain 20,000 training, 5,000 development, and 3,000 test utterances for both two-speaker and three-speaker mixtures.
  • 1) Top-down voluntary focus:: Open-condition evaluation excludes test speakers from training and development, and reference speech is not used in mixture construction.
  • 1) Top-down voluntary focus:: The target speaker is the first selected utterance, and run-time extraction is conditioned on reference speech from that speaker.
  • 1) Top-down voluntary focus:: Under the Random condition, test reference duration averages 7.3s with a 2.7s standard deviation and ranges from 1.6s to 19.6s.
  • 1) Top-down voluntary focus:: Three-speaker experiments compare training on two-speaker mixtures, three-speaker mixtures, or both.

B. Speaker Encoder

The speaker encoder represents reference speech as a target-speaker embedding that guides extraction, while SpEx jointly integrates speaker characterization with time-domain processing.

  • B. Speaker Encoder: The speaker encoder converts reference speech into a speaker embedding that directs the extractor toward the target speaker.This implements top-down voluntary attention using a reference utterance from the target speaker.
  • B. Speaker Encoder: SpEx trains the speaker encoder as an integral component of the extraction architecture during training and inference.The approach contrasts this jointly integrated encoder with an independently trained i-vector extractor.
  • B. Speaker Encoder: The i-vector baseline uses 60-dimensional MFCC, energy, and derivative features extracted from 25ms windows with 10ms shifts.Its UBM and total variability matrix are trained on clean single-speaker speech.
  • B. Speaker Encoder: A BLSTM-based speaker encoder captures temporal information from reference speech and produces a speaker embedding through nonlinear and mean-pooling layers.The BLSTM has 256 cells in each forward and backward direction, followed by 256-node and 400-node layers.
  • B. Speaker Encoder: The speech encoder uses three parallel 1-D convolution branches to produce multi-scale embedding coefficients at short, middle, and long temporal resolutions.The filter lengths cover 20, 80, and 160 samples, corresponding to 2.5ms, 10ms, and 20ms at 8kHz.

2) Speaker Extractor:

The speaker extractor uses target-speaker information to mask mixture representations, and time-domain processing outperforms the frequency-domain counterpart in the reported comparison.

  • 2) Speaker Extractor:: The speaker extractor estimates a target-speaker mask from mixture features and the reference-derived speaker embedding.The masked representation is passed to the speech decoder to reconstruct the target speech.
  • 2) Speaker Extractor:: The multi-scale design applies separate masking and reconstruction paths to embedding coefficients at different temporal resolutions.Three convolutional scales are processed, with each scale producing a modulated response for decoding.
  • 2) Speaker Extractor:: The benchmark compares SpEx with four Speaker Beam Frontend baselines representing frequency-domain speaker extraction methods.The baselines progressively use IBM, magnitude-spectrum, temporal-spectrum, and BLSTM speaker-encoder variants.
  • 2) Speaker Extractor:: SDR, SI-SDR, and PESQ are used to evaluate extraction quality, with SI-SDR included for its suitability for single-channel extraction.An A/B preference test additionally assesses perceived quality and intelligibility.
  • 2) Speaker Extractor:: Time-domain systems consistently outperform the frequency-domain counterpart, including when they use fewer or roughly the same number of parameters.The reported advantage is attributed to trainable encoder-decoder representations that avoid phase estimation.

2) Single-scale vs. Multi-scale:

Multi-scale embedding and joint speaker-encoder training improve SpEx over single-scale and i-vector configurations, while the final system outperforms competitive baselines.

  • 2) Single-scale vs. Multi-scale:: Multi-scale embedding uses short, middle, and long convolution filters to provide multiple time-frequency resolutions for extraction.The speaker extractor estimates a mask at each scale, and the decoder reconstructs a signal from each modulated response.
  • 2) Single-scale vs. Multi-scale:: 13.9dB SDR, 13.3dB SI-SDR, and 3.00 PESQ are achieved by the best multi-scale configuration with α=0.10 and β=0.10.Relative to the single-scale system, this improves SDR by 6.1%, SI-SDR by 7.3%, and PESQ by 2.7%.
  • 2) Single-scale vs. Multi-scale:: The best multi-scale performance uses the short-window output s1, which also reduces evaluation-time parameters compared with weighted output summation.Training can still optimize multiple reconstructed outputs through the weighted multi-scale SI-SDR loss.
  • 3) I-vector vs. Speaker Embedding:: Joint speaker-encoder optimization combines speaker-classification cross-entropy with multi-scale SI-SDR training.This replaces the independently extracted i-vector with an encoder trained alongside the extraction pipeline.
  • 3) I-vector vs. Speaker Embedding:: 8.6% SDR, 9.8% SI-SDR, and 4.7% PESQ relative improvements are obtained over the i-vector-based system with multi-task learning.The best reported SDR and SI-SDR are 15.1dB and 14.6dB when the cross-entropy sub-loss weight is 0.2.
  • 4) Benchmark against the Baselines:: 37.3%, 37.7%, and 15.0% relative improvements over SBF-MTSAL-Concat are reported for SDR, SI-SDR, and PESQ under the open condition.The comparison includes SpEx and four competitive baseline systems.
  • 4) Benchmark against the Baselines:: SpEx avoids inexact phase estimation by operating on embedding coefficients rather than decomposed magnitude and phase spectra.Its stacked dilated depth-wise separable convolutions also capture long-range dependencies without recurrent connections.
  • 4) Benchmark against the Baselines:: Figure 5 reports a female-female mixture example in which SpEx recovers and purifies the target signal better than the competitive baselines.The comparison is made using log magnitude spectra and the clean target speech.

5) Different Gender vs. Same Gender:

Extraction is easier for different-gender mixtures than same-gender mixtures, but SpEx retains substantial gains over the strongest baseline in both conditions and across tested SNR groups.

  • 5) Different Gender vs. Same Gender:: Different-gender mixtures consistently outperform same-gender mixtures in the reported objective evaluations.The paper relates this pattern to greater voice differences between male and female speakers.
  • 5) Different Gender vs. Same Gender:: 34.9% and 40.9% relative SDR improvements over SBF-MTSAL-Concat are reported for different-gender and same-gender conditions, respectively.The corresponding PESQ improvements are 15.2% and 15.0%.
  • 6) Mixture with Different SNR:: 52.9%, 39.6%, and 30.4% relative SDR improvements over SBF-MTSAL-Concat are obtained for [0, 1)dB, [1, 3)dB, and [3, 5)dB SNR groups.Higher-SNR test data performs better than lower-SNR data.
  • 6) Mixture with Different SNR:: The simulated database limits SNR evaluation to 0dB–5dB, motivating future investigation across a wider range from -10dB to 20dB.This is the stated scope boundary for the reported SNR experiments.

7) Subjective Evaluation:

Subjective evaluation favored SpEx over the best SBF-MTSAL-Concat baseline, while additional experiments examined how reference-speech duration affects extraction. SpEx also achieved comparable performance to Conv-TasNet on WSJ0-2mix.

  • Subjective Evaluation: 73.5% of listeners preferred SpEx, versus 11.9% for SBF-MTSAL-Concat, with p < 0.05 significance.The test used 20 listening pairs and 13 subjects, who judged quality and intelligibility.
  • Reference Duration: Longer test reference speech generally improved speaker extraction performance.Performance improved as test reference duration increased from 7.5s to 15s, 30s, and 60s.
  • Reference Duration: Fixing reference duration at 15s during training and development slightly reduced performance relative to the Random condition.Longer test reference speech still consistently helped.
  • Comparison with Speech Separation: SpEx achieved comparable performance to Conv-TasNet with the same TCN architecture on WSJ0-2mix.The comparison used the same optimization scheme for the re-implemented Conv-TasNet.
  • Comparison with Speech Separation: SpEx performance depends strongly on speaker-embedding quality, and its proposed speaker encoder outperformed an i-vector encoder.The authors identify evaluation on speaker databases larger than WSJ0-2mix’s 101 speakers as future work.

B. Experiments on Three-Speaker Mixture

On mixtures with more speakers, SpEx was evaluated under different training conditions and reference durations, with performance favoring fewer simultaneous speakers and longer references. The discussion also identifies extensions for adverse acoustic conditions and alternative attention inputs.

  • Experiments on Three-Speaker Mixture: Two-speaker mixtures consistently outperformed three-speaker mixtures across all three training conditions.The conditions used only two-speaker data, only three-speaker data, or combined two- and three-speaker data.
  • Experiments on Three-Speaker Mixture: Longer reference speech continued to improve performance in the multi-speaker experiments.The authors attribute this pattern to longer references producing better speaker embeddings.
  • Discussions and Conclusions: SpEx is intended for pre-registered target speakers, including speaker-verification scenarios where enrollment identifies the target.The network uses reference speech to form a top-down voluntary focus.
  • Discussions and Conclusions: The time-domain framework, multi-scale encoding and decoding, and multi-task learning are identified as contributors to improvement over frequency-domain counterparts.The authors report that SpEx significantly outperformed those counterparts in their experiments.
  • Discussions and Conclusions: Multichannel inputs are proposed as a future extension for reverberation and multiple interference speakers.The authors also suggest replacing the TCN block with a dual-path RNN for improved speech quality.
  • Discussions and Conclusions: Audio-visual speaker encoding is proposed to strengthen top-down voluntary focus.The speaker encoder mechanism is described as enabling this extension.
  • Discussions and Conclusions: EEG or ECoG signals are proposed as alternatives to reference speech for decoding the attended speaker’s speech.This direction is framed around assistive devices such as hearing aids.
Loading 2004.08326v1…