Source-linked AI summary
End-to-End Attention based Text-Dependent Speaker Verification
Shi-Xiong Zhang, Zhuo Chen, Yong Zhao, Jinyu Li, Yifan Gong
TL;DR
Text-dependent speaker verification benefits from phonetic constraints and short enrollment utterances, but prior systems commonly aggregate frame-level representations with equal weighting. This paper proposes an end-to-end CNN-attention system that learns speaker- and phonetic-informed frame weights and reports effectiveness on the Windows 10 “Hey Cortana” task.
Problem
Prior speaker-verification systems use phonetic or speaker-discriminative representations, while frame-level features are commonly combined by equal weighting; text-dependent verification seeks robust decisions from short enrollment utterances.
Method
The system uses speaker-discriminative CNNs for noise-robust frame features, attention based on speaker and phonetic information, and joint end-to-end training that imitates evaluation and selects confusing impostors.
Results
The end-to-end system outperforms GMM and i-vector systems by 25% and 9%, respectively, while the learned-weight attention network performs best among compared attention mechanisms.
Takeaways & Limitations
The proposed CNN-attention approach demonstrates effective end-to-end text-dependent speaker verification on the Windows 10 “Hey Cortana” task.
Takeaways & Limitations
The attention model is evaluated here on text-dependent speaker verification, while stronger performance for text-independent tasks is presented as expected future work.
Abstract
from arXiv · showhide
A new type of End-to-End system for text-dependent speaker verification is presented in this paper. Previously, using the phonetically discriminative/speaker discriminative DNNs as feature extractors for speaker verification has shown promising results. The extracted frame-level (DNN bottleneck, posterior or d-vector) features are equally weighted and aggregated to compute an utterance-level speaker representation (d-vector or i-vector). In this work we use speaker discriminative CNNs to extract the noise-robust frame-level features. These features are smartly combined to form an utterance-level speaker vector through an attention mechanism. The proposed attention model takes the speaker discriminative information and the phonetic information to learn the weights. The whole system, including the CNN and attention model, is joint optimized using an end-to-end criterion. The training algorithm imitates exactly the evaluation process --- directly mapping a test utterance and a few target speaker utterances into a single verification score. The algorithm can automatically select the most similar impostor for each target speaker to train the network. We demonstrated the effectiveness of the proposed end-to-end system on Windows $10$ "Hey Cortana" speaker verification task.
1. INTRODUCTION
The paper addresses text-dependent speaker verification for fixed phrases such as “Hey Cortana,” where phonetic constraints support robust verification from short utterances. It proposes an end-to-end system that combines CNN speaker features with attention-guided aggregation and challenging-impostor training.
- Text-dependent speaker verification uses the same phrases for enrollment and recognition, unlike text-independent verification.
- Phonetic knowledge enables robust verification with very short enrollment utterances.
- Earlier systems extracted frame-level speaker or phonetic features, equally weighted them, and aggregated them into utterance-level representations.
- The proposed system uses speaker-discriminative CNNs, attention-based feature combination, and joint end-to-end optimization.
- Its training algorithm selects the most competing impostor for each target speaker and evaluates the Windows 10 “Hey Cortana” task.
2. END-TO-END SPEAKER VERIFICATION
The system jointly learns CNN and attention components, then freezes them to create enrollment speaker models and score test utterances against claimed speakers. Training imitates evaluation by optimizing verification scores on target and challenging-impostor pairs.
- 2.1. End-to-End architecture: The architecture learns a CNN and attention network during training, while enrollment and verification use the resulting fixed models.
- 2.1. End-to-End architecture: Enrollment converts several speaker utterances into supervectors and averages them into a speaker model.
- 2.1. End-to-End architecture: Verification computes similarity between a test representation and the enrolled speaker model, then compares the score with a threshold.
- 2.2. End-to-End training: The end-to-end criterion jointly trains CNN, attention, and logistic-regression parameters using similarity scores for test–speaker pairs.
- 2.2. End-to-End training: Training imitates evaluation by mapping a test utterance and several target utterances directly to one verification score.
- 2.2. End-to-End training: Mini-batches use target speakers, acceptance utterances, and the most similar impostors to emphasize challenging speaker pairs.
3. NEURAL NETS FOR SPEAKER VERIFICATION
The paper reviews phonetic- and speaker-discriminative neural networks for speaker verification, then proposes a deep CNN for extracting speaker-discriminative information. It contrasts frame-level and utterance-level representations and motivates CNN-based features for both text-dependent and text-independent tasks.
- 3.1. Learning Phonetic Representation: Phonetic-discriminative DNNs classify speech frames by phoneme or senone and provide bottleneck or posterior features for speaker verification.These features help align different speakers in a common phonetic space before backend scoring.
- 3.2. Learning Speaker Representation: Speaker-discriminative DNNs classify frames or utterances by speaker identity to extract speaker information such as d-vectors.Frame-level speaker features can be averaged into an utterance-level speaker supervector.
- 3.2.3. cnn-vectors: Unlike the RNN-vector approach, the proposed CNN is presented as suitable for both text-dependent and text-independent speaker verification.The paper motivates this choice by prior results showing deep CNNs outperforming LSTMs in many speech-recognition tasks.
- 3.2.3. cnn-vectors: The proposed deep CNN uses a VGG-style architecture with time-frequency convolutions, pooling, linear projection, batch normalization, and asymmetric temporal context.Its input context spans 30 historical frames and 5 future frames to control latency.
4. ATTENTION MECHANISM
The attention mechanism replaces uniform averaging of CNN frame representations with context-dependent weighting. Proposed networks use phonetic information for alignment and, in the advanced version, learn additional weights from phonetic context and bottleneck features.
- 4. ATTENTION MECHANISM: The attention network learns how to combine CNN frame-level speaker representations instead of simply averaging them.It uses phonetic context information to select and combine frame features into an utterance-level representation.
- 4. ATTENTION MECHANISM: A general attention model computes a weighted mean of frame representations, with weights determined by each frame’s relevance to a context vector.The model maps h1:T and context c to a summary vector f.
- 4.1. attention network with posterior weights: The first proposed attention network uses DNN posterior features as context so phonetic alignment supplies the frame-combination weights.The resulting speaker supervector stacks speaker features in the corresponding phonetic space using posterior probabilities.
- 4.1. attention network with posterior weights: Posterior-weighted aggregation compares speaker vectors within corresponding phonetic spaces rather than uniformly across all frames.For example, when a phoneme posterior is one, only the associated 64-dimensional speaker block is nonzero.
- 4.2. attention network with learned weights: The advanced attention network combines posterior and bottleneck phonetic features to learn frame weights in addition to alignment.Its softmax weights represent the density of speaker information in each frame, and the CNN, attention model, and logistic regression can be jointly trained.
- 4.2. attention network with learned weights: The paper applies the attention model to text-dependent speaker verification, while suggesting the phonetic-space mapping could also support text-independent tasks.The text-independent extension is presented as a prospective scope rather than an evaluated result in this section.
5. EXPERIMENTS AND RESULTS
The experiments evaluate the proposed system on Windows 10 “Hey Cortana” speaker verification data and compare end-to-end configurations with conventional baselines. The end-to-end system improves over GMM and i-vector systems, while learned attention performs best among the tested attention mechanisms.
- Experimental Setup: The evaluation uses about 60k utterances from 3k target speakers and 3k impostors, with segments lasting approximately 65–110 frames.The enrollment set includes six utterances.
- Experimental Setup: The systems use 38-dimensional MFCC features, while the end-to-end CNN uses zero padding, one-step convolutional strides, and 2 × 2 max pooling with two-step strides.The CNN training selects six utterances to emulate the Windows 10 enrollment scenario.
- System Comparisons: The end-to-end system outperforms the GMM and i-vector systems by 25% and 9%, respectively, in EER comparisons.Table 1 compares GMM-UBM, i-vector/PLDA, and end-to-end systems in EER%.
- Model Comparisons: The learned-weight attention network performs best among the tested attention mechanisms, and the proposed CNN outperforms the parameter-matched DNN by 6%.The CNN gain over LSTMs is reported as very small for this text-dependent task.
- Training Strategy: The experiments also assess the speaker vector pool used to select the most similar impostor during end-to-end training.This comparison is reported in Table 3.
6. CONCLUSION
The paper presents an end-to-end text-dependent speaker verification system combining speaker-discriminative CNN features with attention-based utterance representations. It jointly trains the CNN and attention models and selects similar impostors during training, demonstrating effectiveness on the Windows 10 “Hey Cortana” task.
- Conclusion: The system uses a speaker-discriminative CNN to extract frame-level features and an attention network to generate an utterance-level speaker representation.The attention model uses speaker-discriminative and phonetic information to learn attention weights.
- Conclusion: The CNN and attention models are jointly learned with an end-to-end training algorithm that selects the most similar impostors for each target speaker.The training procedure is designed around the verification task.
- Conclusion: The proposed system is shown to be effective on the Windows 10 “Hey Cortana” speaker verification task.