Source-linked AI summary
Deep Speaker: an End-to-End Neural Speaker Embedding System
Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, Zhenyao Zhu
TL;DR
Speaker recognition systems face limitations from separately optimized components and mismatched frame-level training versus utterance-level testing. Deep Speaker learns utterance embeddings on a hypersphere using neural feature extractors, pooling, normalization, cosine-based triplet loss, and softmax pretraining. Across text-independent evaluations, it improves over DNN-based i-vector systems and supports transfer between Mandarin and English.
Problem
DNN-based speaker recognition can suffer from independently optimized stages and a mismatch between frame-level training labels and utterance-level test predictions.
Method
Deep Speaker uses ResCNN or GRU feature extractors, mean pooling, length normalization, cosine-based triplet loss, global hard-negative selection, and softmax pretraining.
Results
Deep Speaker significantly improves text-independent recognition over DNN-based i-vector systems, including roughly 50% relative EER reduction on Mandarin UIDs and 30% on English MTurk.
Takeaways & Limitations
The system leverages large training datasets and transfers across Mandarin and English, including speaker-recognition settings with limited data.
Takeaways & Limitations
Future work targets reducing model size, CPU requirements, and long training times.
Abstract
from arXiv · showhide
We present Deep Speaker, a neural speaker embedding system that maps utterances to a hypersphere where speaker similarity is measured by cosine similarity. The embeddings generated by Deep Speaker can be used for many tasks, including speaker identification, verification, and clustering. We experiment with ResCNN and GRU architectures to extract the acoustic features, then mean pool to produce utterance-level speaker embeddings, and train using triplet loss based on cosine similarity. Experiments on three distinct datasets suggest that Deep Speaker outperforms a DNN-based i-vector baseline. For example, Deep Speaker reduces the verification equal error rate by 50% (relatively) and improves the identification accuracy by 60% (relatively) on a text-independent dataset. We also present results that suggest adapting from a model trained with Mandarin can improve accuracy for English speaker recognition.
1 Introduction
Speaker recognition seeks to identify or verify speakers from audio, but conventional and DNN-based systems have training–testing mismatches and independently optimized stages. Deep Speaker extends end-to-end embeddings with multiple architectures and distance-based training, and experiments show strong gains over DNN i-vector systems.
- Speaker verification tests a claimed identity, while speaker identification classifies an unknown voice among known speakers.
- Conventional i-vector systems separately collect statistics, extract embeddings, and classify speakers with PLDA.DNN acoustic models can replace the GMM-UBM when producing sufficient statistics.
- DNN-based systems face independently optimized stages and a mismatch between frame-level training labels and utterance-level test predictions.
- Deep Speaker extracts frame-level features, pools and length-normalizes them into utterance embeddings, and trains with triplet loss on same- and different-speaker pairs.Softmax pretraining improves performance.
- Cosine similarity is optimized directly through globally selected hard negatives, rather than a PLDA-like loss restricted to minibatches.The authors report faster training convergence from global hard-negative selection.
2 Related Work
Related work includes traditional i-vector systems, DNN-assisted i-vector extraction, and end-to-end neural speaker recognition. Deep Speaker differs through architectures balancing inference time and depth while evaluating both text-dependent and text-independent tasks.
- Traditional systems model speaker variability with i-vectors and classify using cosine similarity or PLDA variants.
- DNN-based alternatives replace parts of the i-vector pipeline or classify speakers from frame-level acoustic features.
- End-to-end neural systems jointly optimize speaker-recognition components, using LSTM outputs or pooled network representations for verification.
- Deep Speaker uses different architectures to balance inference time with model depth and evaluates text-dependent and text-independent recognition.The paper also studies dataset size, softmax pretraining, model fusion, and cross-language adaptation.
3 Deep Speaker
Deep Speaker extracts frame-level acoustic features with ResCNN or GRU architectures, pools them into normalized utterance embeddings, and trains cosine-similarity-based triplet loss. The system improves training and evaluation through hard-negative selection and softmax pre-training.
- Residual CNN: Residual CNNs use stacked residual blocks with convolutional layers and identity skip connections to extract acoustic features.Three ResBlocks are stacked, with 3 × 3 convolutions and identity mappings; channel increases use 5 × 5 convolutions with 2 × 2 stride.
- GRU network: The GRU model reduces time and frequency dimensionality with convolution before three forward-only GRU layers, followed by the same pooling and normalization stages.Each GRU layer has 1024 units, and the convolution reduces dimensionality for faster recurrent computation.
- Speaker embedding architecture: ResCNN and GRU networks extract frame-level features, while average pooling, affine projection, and length normalization produce 512-dimensional utterance embeddings.The two architectures have similar parameter counts of 23M–24M, supporting performance comparisons.
- Triplet loss and selection: Triplet loss compares an anchor, same-speaker positive, and different-speaker negative, enforcing a minimum cosine-similarity margin between positive and negative pairs.Embeddings are unit-normalized, and cosine similarity directly represents utterance similarity.
- Triplet loss and selection: 48.97% higher probability of finding an effective negative sample results when scanning four rather than one GPU, while training time increases by 5.47%.Hard negatives become harder to find as training progresses, motivating searches across GPUs.
4 Experimental Setup
Experiments evaluate Deep Speaker and a DNN-based i-vector baseline across Mandarin and English datasets, using text-independent and text-dependent recognition trials. The evaluation measures verification and identification with cosine-similarity trials and tests architectures, pretraining, training scale, and enrollment size.
- Datasets: Experiments use three datasets spanning Mandarin and English, with text-independent and text-dependent speaker-recognition tasks.UIDs and XiaoDu contain Mandarin speech; MTurk contains English speech.
- Datasets: UIDs and MTurk are text-independent datasets, whereas XiaoDu is text-dependent and consists of wake-word utterances.
- Evaluation: Evaluation samples one anchor-positive and 99 anchor-negative trials per anchor utterance, computing cosine similarity for verification and identification.EER measures verification and ACC measures identification; evaluations are not split by gender.
- Baseline: The DNN i-vector baseline uses acoustic features, a seven-layer DNN, senone posteriors, and i-vector extraction for speaker recognition.Its inputs include 40-dimensional Fbank and 20-dimensional MFCC features with appended delta and acceleration features.
- Deep Speaker training: Deep Speaker models use 64-dimensional normalized Fbank features and are trained in softmax-pretraining and triplet-loss fine-tuning stages.Training uses synchronous SGD with momentum and separate minibatch sizes for pretraining and fine-tuning.
5.1 Speaker-independent Experiments on UIDs
On the Mandarin text-independent UIDs task, Deep Speaker substantially improves over the DNN i-vector baseline, with performance shaped by training strategy, architecture, fusion, data scale, and enrollment count.
- Baseline comparison: 50–80% relative EER reduction and 60–70% relative ACC improvement distinguish Deep Speaker from the DNN i-vector baseline.These results use Train50k for training and Eva200 for testing.
- Softmax Pre-training: Softmax plus triplet-loss training performs best, followed by triplet-only and softmax-only training.The result supports combining softmax pretraining with triplet-loss fine-tuning.
- Softmax Pre-training: 63.62% and 17.10% relative EER reductions distinguish ResCNN with combined training from softmax-only and triplet-only models, respectively.The same model also reduces identification error by 47.53% and 31.33% against those two alternatives.
- Architecture comparison: After triplet-loss training, ResCNN has 19.49% lower EER and 10.88% lower error than GRU, reversing the softmax-only architecture ordering.GRU is faster, processing 0.44 minibatches per second versus 0.23 for ResCNN.
- System Combination: Score fusion of ResCNN and GRU yields 7.17% lower EER and 13.37% lower error than the best single system.Both embedding fusion and score fusion improve the single-system baselines, with score fusion performing best.
- Amount of Training Data: Using the full training dataset reduces ResCNN identification error by 17.94% and EER by 21.65% relative to using around one-fifth as much data.For GRU, the corresponding reductions are 15.16% and 13.88%.
- Enrollment Utterance Count: Increasing enrollment utterances from one to five decreases EER and increases ACC.Enrollment embeddings are averaged before constructing recognition trials.
5.2 Text-dependent Experiments on XiaoDu
On the text-dependent XiaoDu task, training only on XiaoDu favors the DNN i-vector baseline, but pretraining Deep Speaker on large text-independent data and fine-tuning it reverses that comparison.
- Task and data scope: The XiaoDu evaluation uses text-dependent wake-word recognition, while Train50k is a text-independent Mandarin training dataset.
- Text-dependent training: The DNN i-vector baseline performs best when models are trained only on the XiaoDu dataset.The authors suggest XiaoDu may be too small for complex Deep Speaker models and that its constrained phonetic variability benefits factor-analysis extraction.
- Text-dependent training: Deep Speaker models trained only on Train50k perform slightly better than models trained only on XiaoDu.The authors attribute this result to the larger amount of training data.
- Fine-tuning: Fine-tuned ResCNN and GRU reduce EER relative to the DNN i-vector system by 16.76% and 18.24%, respectively, with similar ACC.Fine-tuning the traditional DNN i-vector system does not significantly improve performance.
5.3 Text-independent Experiments on MTurk
On the text-independent MTurk task, Deep Speaker systems outperform the DNN i-vector baseline, and models trained on Mandarin transfer to English recognition. Fine-tuning further improves cross-language performance.
- 12.11% and 9.79% EER reductions were achieved by ResCNN and GRU, respectively, versus the DNN i-vector system.
- 15.02% and 19.38% error reductions were achieved by ResCNN and GRU, respectively, versus the DNN i-vector system.
- Mandarin Train50k models performed fairly well on English speaker classification without fine-tuning.
- 25% EER and 35% error reductions were obtained by fine-tuned models compared with non-fine-tuned models.
5.4 Time Span Experiments on UIDs
Recognition performance declines as the time between enrollment and testing increases. ResCNN nevertheless achieves the best performance among systems at the same time span.
- Performance decreases as the enrollment–test time span increases across all systems.The experiment evaluates time spans including less than one week, less than one month, and less than three months.
- ResCNN achieves the best performance for the same enrollment–test time span.
6 Conclusion
Deep Speaker is an end-to-end speaker embedding system that maps utterances to a cosine-similarity hypersphere using ResCNN or GRU features and triplet learning. It improves recognition relative to a DNN-based i-vector approach and supports transfer across languages and task settings, while future work targets efficiency and error analysis.
- Deep Speaker maps speaker utterances to a hypersphere where cosine similarity measures speaker similarity.
- ResCNN and GRU extract frame-level acoustic features, while triplet loss based on cosine similarity trains the embeddings.
- Roughly 50% relative EER and 60% error reductions were reported on the Mandarin text-independent speaker recognition dataset versus the DNN-based i-vector approach.
- Deep Speaker representations transfer across languages and support speaker recognition on small datasets for text-independent and text-dependent tasks.
- Future work targets error-case analysis, smaller models, lower CPU requirements, and shorter training times.