Source-linked AI summary
SpEx+: A Complete Time Domain Speaker Extraction Network
Meng Ge, Chenglin Xu, Longbiao Wang, Eng Siong Chng, Jianwu Dang, Haizhou Li
TL;DR
Speaker extraction must isolate a target voice from a mixture using reference speech, but prior SpEx combined time-domain extraction with frequency-domain speaker embeddings, creating a mismatch. SpEx+ uses weight-shared speech encoders for both inputs, and experiments report significant improvement, especially for same-gender mixtures.
Problem
Prior SpEx combined time-domain speech encoding for extraction with frequency-domain MFCC speaker embeddings, creating a latent-feature and analysis-window mismatch.
Method
SpEx+ uses identical weight-shared speech encoders to represent mixture and reference speech in a common latent feature space.
Results
SpEx+ significantly outperforms previous state-of-the-art TseNet and SpEx, with relative SI-SDR improvements of 23.6% and 9.1%, respectively.
Takeaways & Limitations
SpEx+ is a complete time-domain speaker extraction network whose significant performance improvement is especially pronounced under same-gender mixture conditions.
Abstract
from arXiv · showhide
Speaker extraction aims to extract the target speech signal from a multi-talker environment given a target speaker's reference speech. We recently proposed a time-domain solution, SpEx, that avoids the phase estimation in frequency-domain approaches. Unfortunately, SpEx is not fully a time-domain solution since it performs time-domain speech encoding for speaker extraction, while taking frequency-domain speaker embedding as the reference. The size of the analysis window for time-domain and the size for frequency-domain input are also different. Such mismatch has an adverse effect on the system performance. To eliminate such mismatch, we propose a complete time-domain speaker extraction solution, that is called SpEx+. Specifically, we tie the weights of two identical speech encoder networks, one for the encoder-extractor-decoder pipeline, another as part of the speaker encoder. Experiments show that the SpEx+ achieves 0.8dB and 2.1dB SDR improvement over the state-of-the-art SpEx baseline, under different and same gender conditions on WSJ0-2mix-extr database respectively.
1. Introduction
Speaker extraction isolates a target voice using reference speech, avoiding separation’s global permutation ambiguity but requiring that reference. SpEx+ addresses a latent-feature mismatch in prior SpEx by sharing speech-encoder weights between mixture and reference processing.
- Real-world speech is often corrupted by background noise or speaker interference, motivating front-end processing for speech verification and recognition.
- Speaker extraction uses a target speaker’s reference speech to extract that voice, avoiding global permutation ambiguity and not requiring the mixture’s speaker count.
- Frequency-domain speaker extraction faces phase estimation during reconstruction, while SpEx combines time-domain extraction with MFCC-based speaker embeddings.
- SpEx’s speech encoder and MFCC-based speaker encoder can represent inputs in mismatched latent feature spaces, limiting system efficiency and effectiveness.
- SpEx+ shares identical speech-encoder structures and weights so mixture and reference speech are represented in a uniform latent feature space.
2. SpEx+ Architecture
SpEx+ combines twin weight-shared speech encoders with speaker encoding, extraction, and multi-scale time-domain decoding. Its architecture uses multi-scale representations, speaker-conditioned TCN masking, and joint speech-reconstruction and speaker-classification training.
- Twin speech encoder: SpEx+ uses two identical weight-shared speech encoders to map mixture and reference speech into a common latent feature space.The encoders run separately because they process different inputs, forming the twin speech encoders.
- Twin speech encoder: The speech encoder uses parallel 1-D CNNs with three filter lengths to produce multi-scale embeddings at different temporal resolutions.A shared stride of L1/2 is used when concatenating embeddings across scales, with N filters per CNN.
- Speaker encoder: The speaker encoder transforms reference-speech embeddings through a 1-D CNN, ResNet blocks, mean pooling, and projection into a fixed-dimensional speaker embedding.Residual blocks reduce the representation time series by three through 1-D max pooling and suppress silence.
- Speaker extractor and decoder: The speaker extractor estimates target-speaker masks across three scales using speaker-conditioned TCN stacks with B = 8 blocks and R = 4 repetitions.Only the first TCN block in each stack receives the speaker embedding; dilation grows exponentially across blocks.
- Speaker extractor and decoder: The decoder element-wise multiplies masks with mixture embeddings and reconstructs modulated responses into time-domain signals at multiple scales.The extracted signal s1(t) is selected as the ultimate output during runtime inference.
- Multi-task learning: SpEx+ is trained with multi-scale SI-SDR reconstruction loss combined with cross-entropy speaker-classification loss.The joint objective optimizes output speech quality and discriminative reference-speaker embeddings.
3. Experiments and Discussion
Experiments evaluate SpEx+ on simulated WSJ0-based speaker-extraction tasks, comparing baselines, gender conditions, reference durations, and tied versus untied encoders. SpEx+ improves over prior systems and benefits from longer reference speech.
- Dataset: The WSJ0-2mix-extr1 database contains 101 speakers, with 20,000 training, 5,000 development, and 3,000 open-condition test utterances.Test speakers were unseen during training, and mixtures were generated at relative SNRs from 0 to 5 dB for training and development.
- Dataset: The reference speech is a different randomly selected utterance from the target speaker, while the other mixture speaker is treated as interference.The reference utterance provides the speaker embedding used to characterize the target speaker.
- Comparative evaluation: SpEx+ is compared with baseline systems using SDR, SI-SDR, and PESQ under open-condition evaluation.The baseline implementation includes modified SpEx and TseNet systems for comparison.
- Comparative evaluation: 23.6% and 9.1% relative SI-SDR improvements are reported over TseNet and SpEx, respectively.The paper attributes the improvements mainly to SpEx+'s multi-scale time-domain speaker encoder.
- Reference duration: Longer reference speech duration always leads to better performance on WSJ0-2mix.Table 3 reports SDRi and SI-SDR for different reference-speech durations.
- Comparative evaluation: 13.7% and 4.8% relative SI-SDR improvements are reported over Conv-TasNet for speech separation and SpEx for speaker extraction, respectively.SpEx+ also retains advantages for unknown speaker counts and global permutation ambiguity compared with group speech-separation methods.
4. Conclusions
SpEx+ is a complete time-domain speaker extraction network using shared multi-scale twin speech encoders and a time-domain speaker encoder. Experiments report significant improvements, especially for same-gender mixtures.
- Conclusions: SpEx+ uses a multi-scale twin speech encoder to map mixture and reference speech into the same latent feature space.Its time-domain speaker encoder produces an embedding that characterizes the target speaker and guides extraction.
- Conclusions: Experiments show significant performance improvement for SpEx+, especially under same-gender mixture conditions.