PaperlayerView exact revision on arXiv

Source-linked AI summary

Speaker Recognition Based on Deep Learning: An Overview

Zhongxin Bai, Xiao-Lei Zhang

arXiv:2012.00931v2eess.AS

TL;DR

Speaker recognition has advanced rapidly with deep learning, but comprehensive coverage of its major subtasks and challenges remains limited. This paper surveys deep-learning methods for feature extraction, verification, identification, diarization, and robustness, finding that deep embedding became the prevailing direction after x-vector performance improvements. The survey also identifies fixed speaker capacity and overlap handling as important diarization boundaries.

  • Problem

    Comprehensive reviews of recent deep-learning speaker recognition across major subtasks and research topics are lacking.

  • Method

    The paper synthesizes deep-learning approaches for speaker features, diarization, and robust recognition, including domain adaptation and speech enhancement.

  • Results

    Deep embedding became the trend of speaker recognition after x-vector performance improved, while DNN-UBM/i-vector achieved more than 20% relative EER reduction over the GMM-UBM/i-vector baseline.

  • Takeaways & Limitations

    The survey organizes the field around speaker feature extraction, diarization, and robust recognition while connecting verification, identification, and diarization.

  • Takeaways & Limitations

    End-to-end diarization cannot handle test conversations with more speakers than the maximum represented during training, creating a fixed speaker capacity issue.

Abstract

from arXiv · show

Speaker recognition is a task of identifying persons from their voices. Recently, deep learning has dramatically revolutionized speaker recognition. However, there is lack of comprehensive reviews on the exciting progress. In this paper, we review several major subtasks of speaker recognition, including speaker verification, identification, diarization, and robust speaker recognition, with a focus on deep-learning-based methods. Because the major advantage of deep learning over conventional methods is its representation ability, which is able to produce highly abstract embedding features from utterances, we first pay close attention to deep-learning-based speaker feature extraction, including the inputs, network structures, temporal pooling strategies, and objective functions respectively, which are the fundamental components of many speaker recognition subtasks. Then, we make an overview of speaker diarization, with an emphasis of recent supervised, end-to-end, and online diarization. Finally, we survey robust speaker recognition from the perspectives of domain adaptation and speech enhancement, which are two major approaches of dealing with domain mismatch and noise problems. Popular and recently released corpora are listed at the end of the paper.

1. Introduction

Speaker recognition identifies people from voice and supports authentication, forensics, surveillance, retrieval, and multi-speaker speech recognition. This survey addresses the rapid growth of deep-learning methods by covering major subtasks and core feature-extraction choices.

  • Automatic speaker recognition uses vocal traits such as vocal-tract shape, larynx size, accent, and rhythm to identify speakers computationally.Applications include device authentication, transaction security, forensics, surveillance, and audio-based information retrieval.
  • Deep learning has substantially advanced speaker recognition, including performance in wild environments.
  • The survey covers speaker verification, identification, diarization, and robust speaker recognition.Robust recognition is treated through domain adaptation and speech enhancement.
  • Deep speaker feature extraction is reviewed through inputs, network structures, temporal pooling strategies, and objective functions.These components also underpin other speaker-recognition subtasks.
  • The overview differs from earlier reviews by focusing on recent deep-learning techniques across a broader set of subtasks and topics.It explicitly includes speaker verification, diarization, and domain adaptation perspectives not comprehensively covered in the cited prior overviews.

2. Overview and scope

The survey organizes speaker recognition into verification, identification, diarization, and robust recognition, while distinguishing their task assumptions and computational frameworks. It emphasizes feature extraction, similarity scoring, diarization methods, and robustness to noise and domain mismatch.

  • 2. Overview and scope: The overview covers speaker verification, identification, diarization, and robust recognition, with the first three illustrated in Figure 1.Robust recognition addresses noise and domain mismatch.
  • Speaker verification: Stage-wise speaker verification transforms an utterance into a high-dimensional feature vector, then compares enrollment and test features against a threshold.End-to-end verification instead takes paired utterances and directly produces their similarity score.
  • Speaker identification: Speaker identification determines a test speaker from K enrolled speakers, distinguishing closed-set cases from open-set cases.Real-world systems generally need to enroll and test new speakers dynamically.
  • Speaker identification: Speaker verification is a special case of open-set speaker identification with K = 1, so their fundamental techniques can be similar.
  • Speaker diarization: Diarization partitions a conversation into single-speaker segments and differs from verification through changing speaker counts, no enrollment, and overlapped speech.The survey emphasizes these distinguishing properties in its diarization coverage.
  • Robust speaker recognition: The survey also includes benchmark corpora and reviews speech enhancement and domain adaptation as approaches to robust recognition.

3. Speaker feature extraction with DNN/i-vector

This section reviews two main deep-learning enhancements to the i-vector framework: DNN-UBM/i-vector and DNN-BNF/i-vector. Across reported comparisons, these approaches generally improve over GMM-UBM/i-vector, though results depend on conditions and source studies.

  • From GMM/i-vector to DNN/i-vector: The traditional GMM/i-vector system models MFCC frames with a GMM-UBM and represents each speech segment with an i-vector.The i-vector is the MAP estimate of a segment-specific latent vector derived from accumulated sufficient statistics.
  • Overview: The DNN/i-vector improvements comprise two main streams: DNN-UBM/i-vector and DNN-BNF/i-vector.DNN-UBM replaces GMM-UBM posteriors, while DNN-BNF supplies bottleneck features to factor analysis.
  • DNN-UBM/i-vector: DNN-UBM/i-vector uses ASR-DNN senone posteriors in place of GMM-UBM posteriors to extract i-vectors.The DNN aligns frames with senones and obtains posterior probabilities from its softmax output.
  • DNN-UBM/i-vector: 30% relative EER reduction was reported for DNN-UBM/i-vector over GMM/i-vector on the telephone condition of the 2012 NIST SRE.The review attributes this improvement to the DNN acoustic model’s representation ability.
  • DNN-BNF/i-vector: DNN-BNF/i-vector extracts compact bottleneck-layer features for factor analysis, with models commonly trained to discriminate senones or phonemes.Reported analyses found that phonetic recognition accuracy does not necessarily predict speaker recognition accuracy.
  • Discussion: Across selected literature comparisons, DNN-UBM/i-vector achieved more than 20% relative EER reduction over GMM-UBM/i-vector, while DNN-BNF outperformed MFCC inputs.A supervised GMM-UBM also achieved 20% relative improvement on NIST SRE10, but test conditions can change the conclusions.

4. Speaker feature extraction with deep embedding

This section traces deep speaker embeddings from d-vector to x-vector and organizes their design around inputs, architectures, pooling, and objectives. The surveyed evidence links expanded training data and augmentation to improved x-vector performance.

  • Deep embedding evolution: D-vector labels training frames with utterance-level speaker identities and averages last-hidden-layer frame embeddings into an utterance representation.Its test-stage representation assumes a development-set embedding space can generalize to unseen speakers.
  • Deep embedding evolution: X-vector replaces frame-by-frame labels with utterance-level labels and statistical aggregation of frame embeddings.Time-delay layers produce frame embeddings, while pooling concatenates their mean and standard deviation.
  • Taxonomy and trends: Deep speaker embeddings are discriminative, supervised DNN-based vector representations, unlike the traditional GMM-UBM/i-vector framework.The paper distinguishes deep embedding as a new framework rather than a hybrid DNN/i-vector framework.
  • Experimental evidence: 14% and 25% relative EER reductions result from fusing d-vector and i-vector systems in clean and noisy conditions, respectively.The comparison is against the i-vector system.
  • Experimental evidence: With enlarged training data and data augmentation, x-vector achieves significant performance improvement over the GMM-UBM/i-vector baseline.The surveyed discussion identifies data augmentation as important for x-vector performance.
  • Taxonomy and trends: The review identifies network inputs, network structures, temporal pooling, and objective functions as four key components of deep embedding techniques.These components support the paper’s taxonomy of existing algorithms.

5. Deep embedding: network structures and inputs

Deep speaker feature extractors combine flexible acoustic inputs with diverse neural architectures. The surveyed evidence shows that architecture choices can materially affect EER, although gains are not consistent across datasets.

  • Inputs and architectures: Network designs span convolutional, recurrent, fully connected, attention, and hybrid layers with varied activation functions and topologies.The review emphasizes that input and structure are jointly designed case by case.
  • Representative structures: TDNN is a one-dimensional time-axis convolutional extractor used by x-vector, while E-TDNN widens temporal context and interleaves affine layers.LSTM insertions into TDNN are used to capture temporal information beyond local patterns.
  • Representative structures: ResNet applies two-dimensional convolutions across time and frequency and is used directly or modified as a speaker-feature backbone.Other surveyed architectures include VGGNet, Inception-resnet-v1, BERT, Transformer, LSTM, and GRU models.
  • Inputs and architectures: Deep embedding systems use raw waveforms or time-frequency features such as spectrograms, Mel-filterbanks, and MFCCs.Common architecture families include DNNs, RNNs/LSTMs, and CNNs.
  • Performance discussion: More than 40% relative EER reduction is achieved by F-TDNN over original TDNN on the SITW dataset.The paper notes that this promotion is not consistent across all datasets.
  • Performance discussion: Delta and double-delta features help statistical models but are not very effective in convolutional and time-delay neural networks.The paper suggests neural networks intrinsically capture frame-to-frame time dependency.
  • Open issues: Raw-wave and efficient models remain underexplored, especially for preserving potentially useful phase information and serving computation-limited edge or mobile devices.The review identifies model size and inference efficiency as areas needing further study.

6. Deep embedding: Temporal pooling layers

Temporal pooling bridges frame-level features and utterance-level embeddings. The review covers average, statistical, attentive, multi-head, multi-resolution, vector-based, and Baum-Welch-informed pooling designs.

  • Pooling foundations: Temporal pooling transforms a sequence of frame-level speaker features into an utterance-level representation.The pooling layer is explicitly described as the bridge between frame-level and utterance-level hidden layers.
  • Pooling foundations: Average pooling is the most common pooling function, while statistics pooling concatenates the mean and standard deviation of frame features.Statistics pooling therefore represents each utterance using both first- and second-order summary information.
  • Attention pooling: Self-attention weights frames because equal contribution is not guaranteed when frames differ in speaker-discriminative information.The review presents weighted means and standard deviations under a general self-attention framework.
  • Attention pooling: The general attention framework scores each frame, normalizes scores with softmax, and can use multiple heads to attend across representation subspaces.The scoring function uses learnable parameters, and multi-head attention concatenates head-specific statistics.
  • Attention pooling: Attentive pooling variants include single-head weighted means, attentive statistics, sub-vector multi-head pooling, and Baum-Welch statistics attention.The Baum-Welch variant incorporates utterance-frame relationships through statistics-derived keys.
  • Multi-resolution pooling: Multi-resolution multi-head pooling uses temperature parameters to control attention-weight sharpness, with larger temperature producing lower resolution.Global multi-head pooling concatenates weighted means, producing an output in R^(K d2).
  • Other pooling variants: Vector-based attentive pooling assigns vectorial rather than scalar attention weights to each frame-level vector.The review contrasts this with methods that apply one scalar weight per frame-level vector.

6.4. NetVLAD & GhostVLAD pooling

NetVLAD aggregates frame-level descriptors through trainable soft clustering and residual encoding. GhostVLAD excludes selected ghost clusters so undesirable speech sections contribute less to the final representation.

  • NetVLAD: NetVLAD can be interpreted as trainable discriminative clustering that softly assigns frame descriptors to clusters and encodes residuals.The layer aggregates features across time before producing the output representation.
  • NetVLAD: NetVLAD converts a three-dimensional tensor into N one-dimensional descriptors, where N equals the product of its first two dimensions.The tensor dimensions depend on speech length, spectrum-frequency bins, and convolution-kernel count.
  • NetVLAD: The layer uses M learned clusters and assignment weights, then normalizes cluster residuals column-wise before vectorization.The resulting vector is concatenated across clusters and undergoes final ℓ2-normalization.
  • GhostVLAD: GhostVLAD omits ghost clusters from the final concatenation, so noisy and undesirable speech sections are down-weighted in normal VLAD clusters.The excluded clusters do not contribute to the final representation.

6.5. Learnable dictionary encoding pooling

Learnable dictionary encoding pooling models frame-level features with a learned dictionary and aggregates them relative to dictionary centers.

  • 6.5. Learnable dictionary encoding pooling: LDE pooling models the distribution of frame-level features H using a learnable dictionary of component centers.The dictionary contains centers o_m, with m ranging from 1 to M.
  • 6.5. Learnable dictionary encoding pooling: Each dictionary center uses a learnable smoothing factor τ_m to assign weights to frame-level features.
  • 6.5. Learnable dictionary encoding pooling: The pooling layer produces an aggregated output for each dictionary center, with a simplified form introduced to facilitate derivation.

6.6. Spatial pyramid pooling

Spatial pyramid pooling converts variable-length utterances into fixed-length feature vectors while retaining spatial information from frame-level feature maps.

  • 6.6. Spatial pyramid pooling: Spatial pyramid pooling directly produces fixed-length feature vectors from variable-length utterances in a CNN-based network.
  • 6.6. Spatial pyramid pooling: The layer divides feature maps into 1 × 1, 2 × 2, and 3 × 3 patches, then average-pools each patch.
  • 6.6. Spatial pyramid pooling: Pooling each local patch preserves spatial information from the last frame-level feature maps.
  • 6.6. Spatial pyramid pooling: A later approach extracted embeddings from the patches with a parameter-sharing LDE layer instead of average pooling.

6.7. Other temporal pooling functions

Other temporal pooling methods capture speaker characteristics through alternative aggregation strategies, but no single pooling function is universally best across settings.

  • 6.7. Other temporal pooling functions: Cross-convolutional-layer pooling captures first-order statistics for modeling long-term speaker characteristics.
  • 6.7. Other temporal pooling functions: Total variability pooling, LSTM-based utterance representations, and multiscale aggregation provide additional alternatives to single-scale pooling.
  • 6.7. Other temporal pooling functions: Table 9 reports literature comparisons of temporal pooling functions, but results across rows are not comparable.
  • 6.7. Other temporal pooling functions: Learnable pooling functions generally achieve better or at least competitive results than average or statistical pooling, with higher computational complexity.
  • 6.7. Other temporal pooling functions: The best pooling function depends on the dataset, network structure, and loss function.

7. Deep embedding: Classification-based objective functions

Deep speaker embeddings commonly use classification-based objectives, while Softmax variants and regularizers improve feature discriminability by shaping angular structure and reducing within-class variation.

  • 7. Deep embedding: Classification-based objective functions: Deep-embedding speaker recognition commonly formulates training as multiclass classification over speakers using a final fully connected layer and cross-entropy-based Softmax loss.The training mini-batch contains input vectors, speaker labels, output weights, and biases.
  • 7. Deep embedding: Classification-based objective functions: Standard Softmax maximizes between-class distance but lacks an explicit constraint on minimizing within-class variance.
  • 7.1. The variants of softmax loss: ASoftmax introduces an integer angular margin, while AMSoftmax and AAMSoftmax use additive cosine or angular margins.The ASoftmax margin is limited to a positive integer, making it insufficiently flexible.
  • 7.1. The variants of softmax loss: Compared with Softmax, ASoftmax, AMSoftmax, and AAMSoftmax produce angularly distributed features and quantitatively control decision boundaries through cosine margins.These margins target reduced within-class variance and align with cosine-similarity back ends.
  • 7.2. Regularization for Softmax loss and its variants: Regularizers include feature normalization, hyperspherical-energy constraints, Gaussian priors, center loss, and ring loss.Center loss explicitly minimizes within-class variance, while hyperspherical energy enlarges inter-class separability.
  • 7.3. Multi-task learning for deep embedding: Multitask learning can produce positive results by adding phonetic information to frame-level learning, although phonetic information may be unhelpful for utterance-level embeddings.
  • 7. Deep embedding: Classification-based objective functions: Loss-function design strongly affects performance, and nearly all state-of-the-art deep embedding systems replace traditional Softmax with variants, especially AMSoftmax and AAMSoftmax.Softmax variants and regularizers can also be combined.

8. End-to-end speaker verification: Verification-based objective functions

End-to-end speaker verification directly scores utterance pairs, so its objective functions must learn effective losses, similarity measures, and informative training trials. Verification-based methods can achieve competitive performance with classification-based systems while avoiding output layers that scale with the number of training speakers, but they are harder to train.

  • End-to-end speaker verification: End-to-end speaker verification directly produces a similarity score for a pair of utterances, differing from deep embedding mainly through its objective function.The reviewed systems may still extract utterance-level embeddings for independent back-ends during testing.
  • Core design issues: Verification-based objectives address loss design, utterance-pair similarity measurement, and selection or construction of informative training trials.The number of possible trials grows at least quadratically with the number of training utterances, and many trials are uninformative.
  • Pairwise losses: Pairwise losses include binary cross-entropy, contrastive loss, and learned similarity functions based on cosine similarity, PLDA-like metrics, or joint vectors.Binary cross-entropy commonly balances positive and negative trials because negative trials usually greatly outnumber positive trials.
  • Quadruplet loss: pAUC-based loss directly optimizes the DET tradeoff, addresses class imbalance, selects difficult quadruplets, and approximates dataset-level pAUC within mini-batches.Experiments reported that a batch size no larger than 512 can provide a good approximation.
  • Discussion: Verification-based output layers remain small regardless of training-speaker count, whereas classification output parameters grow linearly and can reach 25.6 million for 50,000 speakers with 512 hidden units.Verification-based systems therefore avoid this parameter-scaling weakness, but require more difficult trial construction and selection during training.
  • Discussion: Recently, verification-based systems have achieved competitive performance with classification-based systems, although their optimization can be unstable when informative trials change dynamically.Class-center learning is cited as one way to alleviate this training weakness.

9. Speaker diarization

The survey covers stage-wise, end-to-end, online, and multimodal diarization, emphasizing how deep learning is integrated into segmentation, embedding, clustering, and refinement. Stage-wise systems are modular and flexible but struggle with independent optimization, unsupervised clustering, and overlapping speech.

  • Overview: Speaker diarization technologies are reviewed as stage-wise, end-to-end, online, and multimodal approaches, with the latter three identified as emerging directions.Stage-wise diarization has been studied for a longer period than the other reviewed directions.
  • Stage-wise diarization: Stage-wise diarization typically combines voice activity detection, speech segmentation, speaker feature extraction, and speaker clustering, with optional re-segmentation.The modules are independent, and the re-segmentation module refines boundaries after clustering.
  • Speech segmentation: Segmentation uses either uniform sliding windows or speaker change detection based on adjacent-segment representations, distances, thresholds, or learned predictors.A common uniform setting is a 1.5-second window with 0.75-second overlap.
  • Speaker feature extraction: Diarization feature extraction maps speech segments to embeddings using i-vectors, DNN-UBM/i-vectors, x-vectors, or other deep embedding extractors.Some methods also incorporate acoustic conditions such as microphone distance or channel information.
  • Speaker clustering: Speaker clustering groups segment embeddings using similarity measures such as cosine or PLDA-based scores, with recent work exploring semi-supervised and supervised learning.Supervised sequence labels can assign positive integers according to each speaker’s first appearance.
  • Speech overlap: Stage-wise systems often assume one active speaker, but speech overlap frequently hinders diarization performance and has motivated overlap detection and overlap-aware re-segmentation.An LSTM-based approach formulates overlap detection as sequence labeling before assigning overlap regions to two speakers.
  • End-to-end diarization: End-to-end diarization with multi-label classification has a fixed speaker capacity and cannot handle test conversations containing more speakers than seen in training.This makes it less flexible than stage-wise methods, which can handle any number of test speakers.

10. Robust speaker recognition

Robust speaker recognition addresses domain mismatch and noise through domain adaptation, speech enhancement, and data augmentation. The survey emphasizes unsupervised adaptation, including adversarial methods that seek domain-invariant yet speaker-discriminative features, alongside multi-condition training for noisy environments.

  • Overview: Recognition in the wild introduces domain mismatch and noise, motivating domain adaptation, speech enhancement, and data augmentation methods.The survey organizes robust speaker recognition around these three approaches.
  • Domain adaptation: Low-resource applications can reuse auxiliary labeled data, but differences in language, phonemes, and recording equipment create domain shifts that hinder effectiveness.The paper focuses on unsupervised domain adaptation because target-domain labels are unavailable and the setting is technically challenging.
  • Domain adaptation: Deep-learning-based domain adaptation is categorized as discrepancy-based, adversarial-training-based, or reconstruction-based.Adversarial adaptation minimizes an approximate domain discrepancy through a domain discriminator.
  • Adversarial adaptation: The unified adversarial framework alternately trains a domain discriminator to predict source versus target domains and a target mapping to make those predictions incorrect.At convergence, the mapped source and target distributions are expected to become approximately equal while retaining speaker discrimination.
  • DANN: DANN uses an encoder, speaker classifier, and domain discriminator; after training, only the encoder remains to extract domain-invariant and speaker-discriminative features.The encoder, classifier, and discriminator are optimized in alternating steps using source labels and source/target domain data.
  • Applications: Adversarial architectures have been applied to channel-, language-, phoneme-, and noise-invariant speaker features.The survey cites both i-vector and x-vector implementations among these applications.

11. Datasets

The survey catalogs established and recently developed speaker-recognition datasets and challenges across wild, office, far-field, noisy, conversational, and multimodal conditions. The collection includes resources for verification, identification, recognition, and diarization.

  • Dataset overview: Table 12 summarizes common and recently developed speaker-recognition databases and challenges, including SV, SI, and SR resources.“Wild” denotes unconstrained audio, while “quite” denotes indoor recordings in a typical office environment.
  • Established resources: NIST SRE is described as the largest and most popular speaker-recognition challenge, while VoxCeleb is collected automatically from open-source media using visual and audio processing.VoxCeleb uses active-speaker verification and CNN-based facial recognition to confirm speaker identity.
  • Recording conditions: SITW benchmarks text-independent recognition on single- and multi-speaker audio acquired under unconstrained conditions, whereas RSR2015 uses six mobile devices in an office setting.RedDots uses a mobile app for offline voice recording and later upload.
  • Noise and auxiliary data: VOICES contains speech mixed with television, music, or babble noise in furnished rooms, and LibriSpeech provides speaker identity labels despite being designed for speech recognition.These characteristics make both resources relevant to robustness or speaker-recognition research beyond their original purposes.
  • Recently developed datasets: CN-CELEB and BookTubeSpeech use automated media pipelines, while Hi-MIA targets far-field scenarios with microphone arrays and a close-talking reference microphone.CN-CELEB adds human checking after automated segment extraction.
  • Challenges and multimodal corpora: FFSVC20 focuses on far-field speaker verification with distributed microphone arrays under noisy real-world conditions, and DIHARD targets diarization robustness across equipment, noise, and conversational domains.AMI provides 100 hours of multimodal meeting recordings from synchronized close-talking, far-field, video, and room devices.

12. Conclusions and discussions

The paper organizes deep-learning-based speaker recognition around feature extraction, diarization, and robustness, while identifying unresolved challenges in training, real-world deployment, and adaptation.

  • The overview relates speaker verification, identification, and diarization, then organizes the field around feature extraction, diarization, and robust recognition.These core issues provide the paper’s framework for analyzing shared difficulties and methods.
  • Speaker feature extraction: Deep speaker embedding methods are reviewed through their inputs, network structures, temporal pooling strategies, and loss functions.The review also examines training-sample construction for end-to-end speaker-verification losses.
  • Speaker diarization: Diarization coverage includes stage-wise, supervised end-to-end, online, and multimodal approaches.The paper emphasizes several modern diarization settings within its broader overview.
  • Robust speaker recognition: Robust speaker recognition is surveyed through deep-learning-based domain adaptation and speech preprocessing for domain mismatch and background noise.The paper also summarizes popular and recently developed datasets.
  • Open problems: Open problems remain in network training, loss functions, real-world diarization, and domain adaptation.The paper notes that many feature-extraction systems still require handcrafted acoustic inputs and that state-of-the-art models have many parameters.
Loading 2012.00931v2…