Source-linked AI summary
TristouNet: Triplet Loss for Speaker Turn Embedding
Hervé Bredin
TL;DR
Speaker comparison needs representations that place same-speaker speech closer than different-speaker speech while supporting direct sequence comparison. TristouNet learns such embeddings with recurrent networks and triplet loss under Euclidean distance, and it improves short-turn comparison and speaker change detection over established approaches. The paper reports stronger speaker change detection coverage, while noting that complete-system diarization impact and variable-length generalization remain unquantified.
Problem
Speaker comparison requires a representation and comparison function that make same-speaker speech closer than different-speaker speech, while existing representations may require separately designed comparison functions.
Method
TristouNet trains recurrent neural networks on anchor-positive-negative sequence triplets with triplet loss to learn a shared embedding compared using Euclidean distance.
Results
TristouNet outperforms BIC- and divergence-based approaches for short speech-turn segmentation and improves speaker change detection coverage.
Takeaways & Limitations
The embedding provides a direct Euclidean comparison approach for short speech-turn comparison and speaker change detection.
Takeaways & Limitations
The impact on a complete speaker diarization system and generalization to variable-length sequences were not quantified.
Abstract
from arXiv · showhide
TristouNet is a neural network architecture based on Long Short-Term Memory recurrent networks, meant to project speech sequences into a fixed-dimensional euclidean space. Thanks to the triplet loss paradigm used for training, the resulting sequence embeddings can be compared directly with the euclidean distance, for speaker comparison purposes. Experiments on short (between 500ms and 5s) speech turn comparison and speaker change detection show that TristouNet brings significant improvements over the current state-of-the-art techniques for both tasks.
1. INTRODUCTION
Speaker comparison can be framed as learning representations in which same-speaker speech is closer than different-speaker speech. TristouNet addresses this by learning a sequence embedding for Euclidean comparison and improves short-turn segmentation and clustering approaches.
- Problem: Speaker verification, identification, and diarization all require a representation function and comparison function that distinguish same-speaker from different-speaker speech.The desired relation is d(f(xa), f(xp)) < d(f(xa), f(xn)).
- Context: The i-vector approach is described as the de facto standard for speaker-recognition representation and the state of the art for speaker diarization.Its sensitivity to sequence duration motivates using other methods for short speech turns before clustering.
- Approach: The paper fixes the comparison function as Euclidean distance and learns the representation function with a recurrent-neural-network speech sequence embedding.This reverses the usual i-vector setup, which treats the representation as given and focuses on designing the comparison function.
- Results: The proposed embedding outperforms BIC and divergence-based approaches for short speech-turn segmentation and clustering, improving speaker change detection.These approaches are commonly used for turns shorter than 5 seconds.
2. RELATED WORK
Earlier neural speaker-recognition methods either used representations not directly optimized for their comparison function or classified sequence pairs. This work instead optimizes a shared sequence embedding directly with triplet loss and uses LSTMs for speaker comparison.
- Prior neural approaches: Earlier deep-network approaches used hidden-layer activations as speaker representations while relying on separately designed GMM- or HMM-based comparison functions.Their internal representations were not optimized directly for the later comparison function.
- Proposed distinction: The proposed approach directly optimizes representation function f for a fixed Euclidean distance, rather than learning an intermediate representation for a separate comparator.This is the paper’s stated distinction from earlier neural speaker-recognition methods.
- Triplet formulation: Compared with a prior pairwise system trained with binary cross-entropy, this work uses sequence triplets and triplet loss to optimize the shared embedding directly.The prior system reported a 35% error rate on 500ms speech-sequence pairs.
- Architecture: The paper uses recurrent neural networks, specifically LSTMs, for speaker comparison and describes this as their first use for speaker turn embedding.LSTMs had previously been successful in automatic speech recognition and speaker adaptation.
3. TRIPLET LOSS FOR SEQUENCE EMBEDDING
The section introduces triplet-loss training and TristouNet, which embeds speech sequences for speaker comparison. Training emphasizes separating same-speaker and different-speaker sequences while the architecture uses bidirectional LSTMs and average pooling.
- Triplet loss: Triplet training uses anchor, positive, and negative sequences, embedding all three before optimizing their relative distances.The anchor and positive come from the same speaker, while the negative comes from a different speaker.
- Triplet loss: The loss adds a safety margin α and focuses optimization on triplets that violate the separation constraint.Triplets satisfying the constraint contribute nothing to the loss, so sampling them would slow training.
- Triplet loss: Hard-negative sampling selects n sequences per speaker, forms Nn(n −1)/2 anchor-positive pairs, and chooses negatives producing ∆τ + α > 0.Sampling is repeated after each epoch.
- TristouNet sequence embedding: TristouNet uses two LSTMs processing the feature sequence forward and backward, then average-pools and concatenates their outputs into a 2.d1-dimensional vector.The two recurrent networks each have d1 units.
4. EXPERIMENTS
Experiments evaluate TristouNet on speaker comparison and speaker change detection using ETAPE data, with comparisons against Gaussian divergence and BIC. The embedding improves coverage while maintaining comparable purity in speaker change detection.
- Data and setup: The ETAPE TV subset provides 13.8 hours of training data from 184 speakers and 4.2 hours of development data from 61 speakers.The corpus contains French television news, debate, and entertainment recordings.
- Data and setup: The same/different experiment classifies pairs of speech sequences as same-speaker or different-speaker using thresholded distances.It uses 100 randomly extracted sequences for each of the 61 development speakers and compares Gaussian divergence, BIC, and the proposed embedding.
- Speaker change detection: Speaker change detection compares distances between left and right 2s sliding windows every 100ms, then applies peak detection and final thresholding.The final threshold removes small peaks and retains large peaks as hypothesized speaker changes.
- Additional analyses: The experiments include distance distributions, training loss and equal error rate across epochs, and same/different results across sequence durations.Embeddings are also evaluated after 50 training epochs for varying sequence durations, and the experiments are reproducible through released Python code.
- Speaker change detection: Figure 6 varies the final threshold to evaluate speaker change detection performance on the ETAPE development set.Performance is assessed using coverage and purity, which capture complementary segmentation behavior.
- Results: At 94.4% purity, embedding-based detection reaches 55% coverage, versus 48% for both BIC and divergence.The embedding does not improve the best achievable purity, but it substantially improves coverage and yields hypothesized turns that are 15% longer on average at that purity.
5. PERSPECTIVE
The paper identifies important open questions about the broader impact and generalization of its speaker change detection results. It also points to extensions involving variable-length sequences, speaker recognition, and larger architectures.
- Open questions: The impact of the speaker change detection improvement on a complete diarization system has yet to be quantified.The unquantified system includes speech activity detection and speech turn clustering.
- Future work: Future work should evaluate generalization to variable-length sequences and application to speaker recognition.Variable-length sequences are supported by the system but were not tested in the reported experiments.
- Future work: The paper also proposes investigating deeper or wider neural networks and replacing triplet loss with center loss.These are presented as possible research directions rather than evaluated contributions.