Source-linked AI summary
End-to-End Text-Dependent Speaker Verification
Georg Heigold, Ignacio Moreno, Samy Bengio, Noam Shazeer
TL;DR
Speaker verification systems traditionally use loosely connected components, motivating a more direct approach for compact text-dependent systems. This paper jointly maps a test utterance and a few enrollment utterances to a verification score within one neural architecture. On the internal “Ok Google” benchmark, the approach improved the best small-footprint DNN baseline from over 3% to 2% equal error rate.
Problem
Existing speaker-verification approaches break the task into more tractable but loosely connected subproblems, while small-footprint systems need a more direct deep-learning model.
Method
The proposed neural architecture directly maps a test utterance and a few enrollment utterances to a score while jointly optimizing the speaker representation and speaker model with the verification loss.
Results
Over 3% to 2% equal error rate: the approach improved the best small-footprint DNN baseline on the internal “Ok Google” benchmark.
Takeaways & Limitations
The approach provides a compact, simplified verification system requiring fewer additional concepts and heuristics.
Takeaways & Limitations
The baseline softmax approach requires candidate sampling to scale to many speaker labels and may require runtime score normalization and dropout for comparable error rates.
Abstract
from arXiv · showhide
In this paper we present a data-driven, integrated approach to speaker verification, which maps a test utterance and a few reference utterances directly to a single score for verification and jointly optimizes the system's components using the same evaluation protocol and metric as at test time. Such an approach will result in simple and efficient systems, requiring little domain-specific knowledge and making few model assumptions. We implement the idea by formulating the problem as a single neural network architecture, including the estimation of a speaker model on only a few utterances, and evaluate it on our internal "Ok Google" benchmark for text-dependent speaker verification. The proposed approach appears to be very effective for big data applications like ours that require highly accurate, easy-to-maintain systems with a small footprint.
1. Introduction
The paper proposes an end-to-end approach for text-dependent speaker verification that maps a test utterance and a few enrollment utterances directly to a verification score. It targets compact, simple systems and evaluates the approach on the global “Ok Google” password.
- Application setting: The global password is the short phrase “Ok Google,” approximately 0.6 seconds long.The constrained lexicon is intended to compensate for phonetic variability and facilitates combination with keyword spotting and voice search systems.
- Proposed approach: End-to-end verification directly maps a test utterance and a few utterances used to build a speaker model to one verification score.The components are jointly optimized with a verification-based loss following the standard speaker verification protocol.
- Contributions: The architecture includes estimation of a speaker model from only a few utterances.
- Empirical evaluation: The evaluation compares frame- and utterance-level representations and feedforward and recurrent neural networks.
- Scope: The paper focuses on text-dependent verification for small-footprint systems, while noting that the approach could also be used for text-independent verification.
2. Speaker Verification Protocol
The standard speaker-verification protocol consists of training a speaker representation, enrolling a few utterances into a speaker model, and evaluating a score against a threshold. Errors are measured through false rejects, false accepts, and equal error rate.
- Training: Training learns a speaker representation from utterances for use with a simple scoring function.Representations may differ by model type, representation level, and training loss; i-vectors and d-vectors summarize frame-level information.
- Enrollment: Enrollment uses a few speaker utterances to estimate a speaker model, commonly by averaging their i-vectors or d-vectors.
- Evaluation: Evaluation compares the score for a test utterance and speaker model with a predefined threshold, accepting above and rejecting below it.The two error types are false reject and false accept.
- Scoring: Cosine similarity can score an evaluation utterance by comparing its representation f(X) with the speaker model m_spk.
- Scoring: PLDA is an alternative refined, data-driven scoring approach.
3. D-Vector Baseline Approach
The d-vector baseline derives utterance-level speaker representations with a DNN, averages enrollment representations into a speaker model, and scores test utterances by cosine similarity. Its frame-derived context and softmax training objective create limitations relative to direct verification optimization.
- D-vector extraction: D-vectors represent utterances using a DNN with a locally-connected layer and fully connected layers, trained with softmax loss.The final hidden-layer activations are averaged across utterance frames to produce one d-vector.
- Enrollment and scoring: Each speaker model averages the d-vectors from the enrollment utterances, while verification uses cosine similarity with the test utterance’s d-vector.The same fixed DNN produces representations for enrollment and test utterances.
- Limitations: The baseline’s frame-window representations provide limited context, and its softmax loss does not follow the standard verification protocol.These mismatches motivate heuristics and score normalization during evaluation.
- Limitations: Alternative systems also contain loosely connected components or stages that are not directly optimized for verification.The paper cites GMM-UBM, i-vector, PLDA, and frame-based models as examples of these concerns.
4. End-To-End Speaker Verification
The proposed architecture jointly trains utterance representations, estimates a speaker model from a few enrollment utterances, and verifies a test utterance with cosine similarity and logistic regression. It directly optimizes the evaluation metric under the standard protocol and supports variable enrollment counts.
- Architecture: The end-to-end system combines representation training, speaker-model enrollment, and evaluation in one network architecture with a verification-based loss.The architecture follows the standard speaker-verification stages while optimizing them jointly.
- Speaker representations: The system uses DNN or LSTM networks to produce utterance-level speaker representations, with the LSTM connected to the loss only through its final output.This yields a single representation for each utterance rather than multiple LSTM outputs.
- Enrollment: The speaker model averages a small number of enrollment representations, while training samples only a few same-speaker utterances to match enrollment conditions.A weight indicates whether each available utterance should contribute, allowing a variable number of enrollment utterances.
- Verification: Verification computes cosine similarity between the test representation and speaker model, then applies logistic regression to estimate acceptance or rejection.The learned verification threshold is −b/w.
- Advantages: The architecture directly optimizes the evaluation metric using consistent speaker models and does not depend on the number of training speakers or a minimum number of utterances per speaker.The authors associate this design with avoiding heuristics and postprocessing such as score normalization.
5. Experimental Evaluation
The evaluation tests end-to-end speaker verification on the internal “Ok Google” benchmark, comparing representations, losses, speaker-model sizes, and network architectures. Results favor utterance-level and end-to-end approaches, with recurrent models gaining accuracy at higher computational cost.
- 5.1. Data Sets & Basic Setup: The evaluation uses anonymized “Ok Google” voice-search utterances, multistyle noise augmentation, and real enrollment and evaluation data.Training data include added car and cafeteria noise at varied SNRs and simulated microphone distances; enrollment and evaluation use only real data.
- 5.2. Frame-Level vs. Utterance-Level Representation: Utterance-level representations outperform frame-level representations by 30%, while score normalization provides up to a 20% relative improvement for either approach.The comparison uses a DNN trained with softmax and 50% dropout in the linear layer.
- 5.3. Softmax vs. End-to-End Loss: On the larger 22M-speaker training set, end-to-end loss clearly outperforms softmax, while t-normalization benefits softmax but not the other losses.The softmax layer uses candidate sampling to scale to 80k speaker labels.
- 5.4. Feedforward vs. Recurrent Neural Networks: Speaker-model size has a broad optimum around 5 utterances, yielding 2.04% EER versus 2.25% for a model size of 1.The optimum is close to the enrollment set’s true average model size of 6 utterances.
- 5.4. Feedforward vs. Recurrent Neural Networks: The best DNN improves 10% relative over the small-footprint DNN, while the LSTM adds another 30% gain but requires about ten times more operations.The LSTM and DNN have comparable parameter counts, but the LSTM’s computational complexity is substantially higher.
6. Summary & Conclusion
The proposed end-to-end system directly maps utterances to verification scores while jointly optimizing speaker representations and models. On the internal “Ok Google” benchmark, it improved error rates, with recurrent modeling achieving the best reported result at higher runtime cost.
- The end-to-end approach directly maps utterances to a verification score and jointly optimizes speaker representations and speaker models with the training and evaluation loss.
- Over 3% to 2% equal error rate: the approach improved the best small-footprint DNN baseline on the internal “Ok Google” benchmark.The paper assumes sufficient training data for this improvement.
- Most of the improvement came from utterance-level rather than frame-level modeling.
- 1.4% equal error rate was achieved with a recurrent neural network instead of a simple deep neural network, but with higher computational runtime cost.
- 4.7% equal error rate was obtained by a reasonable but not fully state-of-the-art i-vector/PLDA system.
- The authors conclude that more comparative studies are needed despite the demonstrated results.