Source-linked AI summary
Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam
Marc Delcroix, Tsubasa Ochiai, Katerina Zmolikova, Keisuke Kinoshita, Naohiro Tawara, Tomohiro Nakatani, Shoko Araki
TL;DR
The paper addresses SpeakerBeam’s difficulty discriminating target speakers with similar voice characteristics. It introduces a time-domain SpeakerBeam, spatial-feature integration, and an auxiliary speaker-identification loss, and reports improved target extraction, especially for same-gender mixtures, while outperforming TasNet-based extraction.
Problem
SpeakerBeam struggles to discriminate target speakers from interferers with similar voice characteristics, particularly in same-gender mixtures.
Method
The paper develops time-domain SpeakerBeam and adds spatial features, internal spatial combination, and an auxiliary speaker-identification loss.
Results
Experiments show improved target speech extraction, especially for same-gender mixtures, and performance that outperforms a TasNet-based target-selection system.
Takeaways & Limitations
Time-domain processing, spatial information, speaker-identification supervision, and more training speakers improve SpeakerBeam’s target-speaker discrimination.
Takeaways & Limitations
Future work is needed to combine the techniques for more challenging noisy and reverberant mixtures and to investigate alternative spatial integrations and SI-losses.
Abstract
from arXiv · showhide
Target speech extraction, which extracts a single target source in a mixture given clues about the target speaker, has attracted increasing attention. We have recently proposed SpeakerBeam, which exploits an adaptation utterance of the target speaker to extract his/her voice characteristics that are then used to guide a neural network towards extracting speech of that speaker. SpeakerBeam presents a practical alternative to speech separation as it enables tracking speech of a target speaker across utterances, and achieves promising speech extraction performance. However, it sometimes fails when speakers have similar voice characteristics, such as in same-gender mixtures, because it is difficult to discriminate the target speaker from the interfering speakers. In this paper, we investigate strategies for improving the speaker discrimination capability of SpeakerBeam. First, we propose a time-domain implementation of SpeakerBeam similar to that proposed for a time-domain audio separation network (TasNet), which has achieved state-of-the-art performance for speech separation. Besides, we investigate (1) the use of spatial features to better discriminate speakers when microphone array recordings are available, (2) adding an auxiliary speaker identification loss for helping to learn more discriminative voice characteristics. We show experimentally that these strategies greatly improve speech extraction performance, especially for same-gender mixtures, and outperform TasNet in terms of target speech extraction.
1. INTRODUCTION
The paper improves SpeakerBeam’s ability to discriminate and extract a target speaker, especially when speakers have similar voice characteristics. It introduces a time-domain architecture, spatial-feature integration, and an auxiliary speaker-identification loss, with experiments showing stronger extraction performance.
- Motivation: SpeakerBeam addresses speech separation’s source-count and permutation-ambiguity limitations by extracting a speaker identified through an auxiliary clue.The original method uses an adaptation utterance to derive a speaker embedding that guides target extraction.
- Motivation: Similar voice characteristics create a substantial performance gap between same-gender and different-gender mixtures.The difficulty arises because the target speaker is harder to discriminate from interfering speakers.
- Proposed strategies: TD-SpeakerBeam replaces frequency-domain processing with time-domain mixture inputs and directly outputs the estimated target waveform.Its auxiliary network also uses a convolutional architecture to produce richer speaker embeddings.
- Proposed strategies: Spatial features are added to improve speaker discrimination from microphone-array recordings, including an internal-combination design within SpeakerBeam.The paper motivates spatial information as complementary to the speaker embedding used for target selection.
- Proposed strategies: A multi-task loss combines speech reconstruction with speaker identification to encourage more discriminative speaker embeddings.The auxiliary SI-loss is intended to improve identification of the target speaker during extraction.
- Results: Experiments show TD-SpeakerBeam improves extraction and outperforms a TasNet-based selection system, while spatial features and SI-loss further help, especially for same-gender mixtures.The reported evaluations cover two datasets and include comparisons with TasNet separation followed by x-vector target selection.
2. PROPOSED TIME-DOMAIN SPEAKERBEAM
TD-SpeakerBeam adapts the SpeakerBeam framework to time-domain signals and incorporates speaker embeddings, spatial features, and a multi-task loss to improve target-speaker extraction.
- 2.1. TD-SpeakerBeam: TD-SpeakerBeam processes mixture and adaptation utterances as time-domain signals and directly outputs the estimated target speech.Its extraction network follows Conv-TasNet, while the auxiliary network computes the target speaker embedding.
- 2.1. TD-SpeakerBeam: The extraction network uses a single target-speech output and a multiplicative adaptation layer driven by the target speaker embedding.The adaptation layer is inserted between the first and second convolution blocks.
- 2.2. Spatial features: IPD features encode spatial information from two microphones and are represented using cosine and sine features aligned to the extraction network.The IPD features are computed from STFT coefficients and upsampled when their analysis settings differ from the extraction encoder.
- 2.2. Spatial features: Internal IPD combination inserts processed spatial features after the adaptation layer, allowing speaker selection to use spectral information while upper layers exploit spatial information.This differs from input combination, which concatenates spatial features with the encoder output before the initial convolutional processing.
- 2.2. Spatial features: The study limits spatial processing to additional features for the extraction network and excludes beamforming from its investigation.Beamforming is identified as relevant to ASR applications but outside this paper’s scope.
- 2.3. multi-task learning with additional SI-loss: The multi-task objective combines target-speech SiSNR reconstruction with cross-entropy speaker identification loss over the target speaker embedding.The loss uses the target speaker label, a projection matrix, softmax classification, and a scaling parameter α.
3. RELATED PRIOR WORK
The paper compares direct target speech extraction with a pipeline that separates the mixture and then selects the target speaker.
- 3. RELATED PRIOR WORK: The experimental baseline combines TasNet speech separation with x-vector-based target speaker selection.This provides a time-domain separation-based comparison for target speech extraction.
- 3. RELATED PRIOR WORK: Prior work also explored fixed beamformers with attention followed by refinement using FD-SpeakerBeam.That alternative is discussed as related multi-channel source-separation work.
4. EXPERIMENTS
Experiments on multi-channel WSJ0-2mix and single-channel CSJ-2mix evaluate spatial features, SI-loss, and training-speaker scale for target speech extraction. TD-SpeakerBeam improves over frequency-domain SpeakerBeam and TasNet-based baselines, with especially strong gains for same-gender mixtures.
- Datasets: Experiments used MC-WSJ0-2mix for spatial-feature investigations and CSJ-2mix for SI-loss and training-speaker analyses.MC-WSJ0-2mix has 101 training speakers and two used microphone channels, whereas CSJ-2mix has 937 training speakers.
- Results with IPD features using MC-WSJ0-2mix: TD-SpeakerBeam outperformed all systems except TasNet with oracle speaker selection on MC-WSJ0-2mix.Its same- versus different-gender performance gap was smaller than FD-SpeakerBeam’s.
- Results with IPD features using MC-WSJ0-2mix: Internal IPD combination improved TD-SpeakerBeam performance by up to 1 dB and surpassed input IPD combination.TasNet with input IPD features also improved especially for same-gender mixtures, while internal IPD combination was omitted for TasNet because it performed worse.
- Results with the SI-loss on CSJ-2mix: TD-SpeakerBeam achieved much better performance than FD-SpeakerBeam and TasNet with or without oracle speaker selection on CSJ-2mix.Adding SI-loss provided a further consistent improvement of up to 1 dB.
- Results with the SI-loss on CSJ-2mix: TD-SpeakerBeam with or without SI-loss greatly reduced processing failures, while SI-loss produced more cases with high SDR improvement.The histogram summarizes SDR improvements for female-female, male-male, and female-male mixtures.
- Results with the SI-loss on CSJ-2mix: Increasing training speakers greatly improved SpeakerBeam performance but had little effect on TasNet, especially benefiting same-gender mixtures.SI-loss consistently improved performance with more than 100 speakers, while female-female results remained lower partly because fewer female speakers were available.
5. CONCLUSION
The paper improves SpeakerBeam’s target-speaker discrimination through time-domain implementation and additional strategies. The authors identify combining these techniques for noisy and reverberant mixtures as future work.
- Time-domain implementation greatly improved SpeakerBeam’s target speech extraction performance.
- Spatial information, an auxiliary SI-loss, and more training speakers further reduced the performance gap between same-gender and different-gender mixtures.
- Future work will combine these techniques for more challenging noisy and reverberant mixtures and investigate alternative spatial-integration and SI-loss approaches.