Contrastive Predictive Coding Based Feature for Automatic Speaker Verification

1.1.1 Speaker Identification v.s. Verification

• Automatic speaker recognition identifies or verifies speaker-related information, with speaker identification framed as closed-set and verification as open-set.Source: §1.1 · p. 12, Figure 1.2 · p. 14, §1.1.1 · p. 14

1.1 Introduction

• Speaker information is embedded in speech but can be corrupted by channel effects such as environmental and recording noise.Source: §1.1 · p. 12

1.1.2 General Processing Pipeline

• The recognition pipeline comprises feature processing, clustering, summarization, and backend processing.Features include MFCC, FilterBank, PLP, and bottleneck representations; clustering can use GMM, summarization produces fixed-length utterance features, and backend processing scores decisions with SVM, cosine similarity, or PLDA.Source: §1.1.2 · pp. 14–15, Figure 1.3 · p. 15

1.1.3 Metrics

• EER measures the equilibrium point between false alarm and false negative rates, and this thesis adopts it as its evaluation metric.Source: §1.1.3 · p. 16

1.1.4 Challenges

• Speaker recognition is difficult despite outputting only once per sequence: it optimizes a Sequence-to-One mapping, has very little data, and faces major channel effects.Source: §1.1.4 · p. 16

1.1.5 Applications

• Automatic speaker recognition techniques can transfer to language recognition, age estimation, emotion classification, and spoofing-attack detection.Source: §1.1.5 · p. 17

1.2 Adapted Gaussian Mixture Models (GMM-UBM)

• GMM-UBM was a dominant 1990s speaker-verification approach that trains a speaker-independent UBM and adapts it into speaker-specific models.The approach exploits large unlabeled datasets despite limited speaker enrollment data.Source: §1.2 · p. 17, §1.2.2 · p. 19

1.2.1 Likelihood Ratio Detector

• Speaker verification compares the likelihood of an utterance under a speaker-dependent GMM with its likelihood under the speaker-independent UBM.The resulting likelihood-ratio detector accepts the speaker hypothesis when the speaker-model likelihood is larger.Source: §1.2.1 · pp. 17–18

1.2.2 UBM

• The UBM represents shared speech characteristics with a large GMM trained on broad data, while speaker-dependent characteristics are obtained through adaptation.This decomposition motivates using abundant unlabeled data for the UBM and short enrollment data for speaker adaptation.Source: §1.2.2 · p. 18

1.2.3 MAP Estimation

• MAP estimation linearly adapts UBM mixture weights, means, and variances using enrollment statistics, although common practice adapts only the means.The illustrated example keeps mixture variances fixed.Source: §1.2.3 · p. 19, Figure 1.5 · p. 20

1.3 Joint Factor Analysis (JFA)

• JFA addresses the huge supervector dimension by splitting supervector M into speaker-independent, speaker-dependent, channel-dependent, and residual components; with F = 39 and C = 1024, M is almost 40,000-dimensional.The extracted vector y is much lower-dimensional than GMM-UBM’s M.Source: §1.3 · pp. 20–21

1.4 Front-End Factor Analysis (i-vectors)

• i-vectors modify JFA into a fixed-length summarization using one low-rank total-variability matrix that models both speaker and channel variability.The extracted i-vector represents the speaker and can be processed with backends such as PLDA.Source: §1.4 · p. 21

1.5 Robust DNN Embeddings (x-vectors)

• x-vectors use a supervised DNN trained for speaker classification, with outputs after statistical pooling serving as speaker embeddings.The framework combines clustering and summarization, and uses data augmentation with noise and reverberation because DNN training requires substantial data.Source: §1.5 · p. 22
• x-vectors do not necessarily outperform i-vectors when data and computational resources are limited.This qualifies the robustness claims associated with x-vector systems.Source: §1.5 · p. 23

1.6 Learnable Dictionary Encoding (LDE)

• LDE combines clustering, summarization, and backend processing into an end-to-end framework, avoiding a separately trained PLDA backend with Angular Softmax losses.It uses a learnable dictionary-style pooling layer to produce a fixed-dimensional speaker representation.Source: §1.6 · pp. 23–24, Figure 1.8 · p. 24
• LDE employs ResNet34 and places only a fully connected classification layer after pooling, unlike the deeper post-pooling structure described for x-vectors.The comparison highlights a different backbone and post-pooling design.Source: §1.6 · pp. 23–24

2.1 Introduction

• The thesis surveys Log-spectrogram, Log-Filterbank, MFCC, and PLP, noting that x-vectors and LDE use Log-Filterbank while several conventional systems use MFCC.The feature-processing discussion establishes MFCC as the experimental baseline.Source: §2.1 · p. 25

2.2 Mel-Frequency Cepstral Coefficients (MFCC)

• MFCC extraction applies STFT, Mel filtering, logarithms, DCT, and coefficient selection to decorrelate features and reduce dimensionality.Compared with Log-Spectrogram and Log-Filterbank, MFCC has fewer dimensions and less visible structure in the stated comparison.Source: §2.2 · p. 26

2.3 MFCC Details

• MFCCs are extracted from recordings at 16 kHz using 25 ms frames, 10 ms shifts, 40 Mel filters, 24 cepstral coefficients, and delta features.First- and second-order derivatives (deltas and double-deltas) are computed during UBM training; the configuration is mostly based on the Kaldi toolkit (povey2011kaldi).Source: §2.3 · p. 26, Table 2.1 · p. 27

3.1 Introduction

• Predictive coding proposes that current and past states contain information about future states, motivating representations that extract global structure from noisy speech.CPC connects predictive coding with representation learning and aims to learn noise-invariant representations.Source: §3.1 · p. 28

3.2 Predictive Coding in Neuroscience

• Predictive coding models generate hypotheses, compute residual errors, and propagate those errors hierarchically, allowing learned representations to retain input properties.The framework emphasizes feedback alongside feedforward transmission, while hierarchical models pass residual error forward to the next level.Source: §3.2 · pp. 29, 31

3.3.1 Connection to Predictive Coding

• CPC seeks slow-varying, high-level features by predicting future observations from past context, separating global information from low-level noise through mutual-information estimation.Unlike L1 or MSE prediction losses that require a decoder D, CPC models mutual information directly using noise contrastive estimation.Source: §3.3.1 · p. 31

3.3.2 Mutual Information

• Mutual information I(X; Y) is introduced as the information shared between two random variables and formalized using entropy and conditional entropy.The section defines I(X; Y), H(X), and H(X | Y), then begins an algebraic proof using joint and conditional probabilities.Source: §3.3.2 · pp. 33–34
• Mutual information measures how well a representation captures global information, being high for relevant remembered developments and low for minor ones.The discussion also considers the independent-variable case, where conditional-entropy identities simplify the mutual-information analysis.Source: §3.3.2 · pp. 35–36

3.3.3 Noise-Contrastive Estimation (NCE)

• NCE estimates unnormalized parametric probability densities by treating the intractable partition function as an additional parameter.This addresses the high-dimensional intractability of computing Z(α), where maximum likelihood requires normalization.Source: §3.3.3 · pp. 37–38

3.3.3.1 Density Estimation in a Supervised Setting

• In supervised NCE, samples from an unknown data density are contrasted with samples from known noise and classified using logistic regression to estimate a density ratio.Optimizing the resulting cross-entropy loss yields the desired density estimate.Source: §3.3.3.1 · pp. 38, 39, 41

3.3.3.2 The NCE Estimator

• The NCE estimator is off by a scaling constant, as established in the supervised formulation.The estimator is introduced after the general NCE setup and inherits this scaling issue from Section 3.3.3.1.Source: §3.3.3.2 · p. 41

3.4.1 Single Autoregressive Model

• CPC encodes raw utterance waveforms into a latent space and trains an RNN with NCE to learn global speaker information.The latent representation is learned from raw waveform input, with the recurrent model trained using the NCE loss.Source: §3.4.1 · p. 42

3.4.1.1 NCE Loss

• The NCE loss extends current-frame density-ratio estimation to predict future frames up to T steps, making the objective predictive-coding-based.The modified loss adds density ratios for future frames fi(ui+t, si) rather than using only the current frame.Source: §3.4.1.1 · p. 44

3.4.1.2 Connection to Mutual Information

• Minimizing NCE makes the density ratio converge to p(ui|si)/p(ui) and maximizes mutual information, with I(ui+t, si) >= log N − Lopt.The derivation identifies log N − Lopt as a lower bound on mutual information.Source: §3.4.1.2 · pp. 46–47

3.4.2 Shared Encoder Approach

• The shared encoder approach uses two jointly optimized autoregressive models in one latent space, processing forward and backward sequences and concatenating their context vectors for speaker verification.It addresses inaccurate early unidirectional-RNN context vectors while avoiding the contradiction between bidirectional sequence access and future-frame prediction.Source: §3.4.2 · pp. 49–50

3.4.3 Detailed Implementation

• Most implementation details conform to the original CPC model (oord2018representation).The section states this conformity without providing further implementation specifications.Source: §3.4.3 · p. 50
• The encoder takes raw waveform without Voice Activity Detection or Mean Variance Normalization; each iteration randomly extracts a 1.28-second (20480-data-point) segment, using a five-layer 1-dimensional CNN with 160-fold downsampling.Source: §3.4.3 · pp. 50–51
• The autoregressive component is a GRU, whose context vector serves as the CPC feature for downstream tasks.The described GRU has 256 hidden dimensions before the reduced-dimension variants are explored.Source: §3.4.3 · p. 51
• Three CPC models vary the GRU hidden dimension because the standard 256-dimensional feature would take weeks to train in speaker-verification systems.The variants are CDCK2, CDCK5, and CDCK6; CDCK2 and CDCK5 use a single autoregressive model, whereas CDCK6 uses a shared approach.Source: §3.4.3 · p. 51
• NCE negatives are drawn from other utterances in the same batch, with future prediction timestep k=12 and batch size B=64 for all CPC models.Selecting other batch samples enables the negatives to be computed in one forward pass.Source: §3.4.3 · p. 51

• CPC features are evaluated for their effectiveness in speaker verification and their integration into a standard speaker-verification system.Source: §3.5 · p. 52
• The CPC model is trained on training data and produces frame-level representations for downstream speaker-verification processing.Source: §3.5 · p. 52
• Utterance-level representations are obtained either by temporal averaging across frames or with an additional i-vector summarization system.Source: §3.5 · p. 52
• The resulting representations are mean- and length-normalized, reduced with Linear Discriminant Analysis, and scored using a PLDA model before EER computation.Source: §3.5 · p. 52
• Figures 3.6–3.8 illustrate the CPC implementation and the training and testing pipelines for the speaker-verification system.Source: Figure 3.6 · p. 53, Figure 3.7 · p. 54, Figure 3.8 · p. 54

• LibriSpeech is a 1000-hour speech dataset derived from LibriVox audiobooks and contains male and female speakers reading book chapters.Source: §4.1 · p. 55
• The corpus is recorded at 16 kHz and partitioned into seven subsets summarized in Figure 4.1.Source: §4.1 · p. 55
• The experiments train CPC models on train-clean-100, train-clean-360, and train-clean-500.Source: §4.1 · p. 55
• Dev-other and dev-test are used for validation and CPC model selection before reporting speaker-verification results.Source: §4.1 · p. 55

• Each trial records an enrollment ID, test ID, and target/nontarget label indicating whether the speakers match.Source: §4.2 · pp. 55–56
• Table 4.1 provides three example speaker-verification trials, while Figure 4.1 summarizes the underlying LibriSpeech subsets.Source: Table 4.1 · p. 56, Figure 4.1 · p. 56
• 1,716,019 trials are included in the first trial list, which randomly splits LibriSpeech recordings into enrollment and test halves.Source: §4.2 · p. 56
• The two trial lists differ in chapter-overlap constraints: the second prevents overlap among chapters spoken by the same speaker, unlike the first.Source: §4.2 · p. 56
• The trial lists are made available for download, and Table 4.2 summarizes CPC model training.Source: §4.2 · p. 57, Table 4.2 · p. 57

• On the development set, NCE losses decreased while future-frame positive-sample prediction accuracies increased over training iterations for CDCK2, CDCK5, and CDCK6.The NCE loss L was averaged over all future prediction timesteps 1, 2, .., k.Source: §4.3 · pp. 57–58
• The CDCK2 model recorded EERs of 5.887 and 13.48, corresponding to relative improvements of 36% and 18%, respectively.Source: §4.3 · p. 61
• CDCK6 achieved lower NCE loss and higher prediction accuracy during training than CDCK2, but its features performed worse than CDCK2 for speaker verification.Source: §4.3 · pp. 57, 61
• After PCA, CPC feature dimensions were reduced to at most 60 before i-vector summarization.PCA was selected as a linear transform that avoids introducing extra nonlinearity.Source: §4.3 · p. 62
• CPC features with i-vectors achieved up to 11% relative EER improvement over the MFCC baseline on the first trial list, but improvements were smaller than with average pooling.On the second trial list, MFCC with i-vectors outperformed CPC with i-vectors.Source: §4.3 · p. 62, Table 4.5 · p. 63
• MFCC-CPC fusion with i-vectors achieved 34% and 17% relative improvements over MFCC i-vectors on the two trial lists.The features were fused by concatenating the MFCC and CPC vectors before i-vector extraction.Source: §4.3 · p. 63

4.4 Feature Visualizations

• CPC and MFCC features differ substantially in structure and magnitude across the two visualized utterances.The comparison includes MFCC, CPC, and PCA-transformed CPC features from CDCK5-24.Source: §4.4 · pp. 63–64, Figure 4.8 · p. 64, Figure 4.9 · p. 65
• Several CPC feature bins remain within a small range over time, indicating learned global information that persists across time.Source: §4.4 · p. 64

4.5 Speaker Verificaiton DET Curves

• MFCC-CPC fusion reduced miss and false-alarm probabilities relative to the baseline on both trial lists.Source: §4.5 · p. 65, Figure 4.10 · p. 66, Figure 4.11 · p. 66
• CPC i-vectors attained lower miss and false-alarm probabilities than MFCC i-vectors only on the first trial list.Source: §4.5 · p. 67, Figure 4.12 · p. 67, Figure 4.13 · p. 68

5.1 CPC as an Alternative Feature for Speaker Verification

• CPC was explored as an unsupervised learned feature and achieved competitive speaker-verification accuracy on the LibriSpeech corpus.Source: §5.1 · p. 69

5.2 i-vectors is not an Ideal Summarization Method for CPC

• i-vectors did not show a clear CPC advantage over MFCC, unlike average pooling, and were judged non-ideal for summarizing CPC frame-level features.Source: §5.2 · p. 70
• The discussion attributes this limitation partly to the possibility that CPC features are not multivariate-Gaussian distributed, an assumption imposed by i-vectors.The text proposes x-vectors as a summarization method that does not assume an input distribution.Source: §5.2 · pp. 69–70

5.3 CPC Complements MFCC for i-vectors Speaker Verification

• CPC and MFCC fusion improved over both MFCC-only and CPC-only i-vector systems, supporting complementarity on LibriSpeech.Source: §5.3 · p. 70
• Whether CPC complements MFCC across all speech data remains unresolved and is left for future work.Source: §5.3 · p. 70

5.4.1 Density Estimation Methods

• Future work proposes comparing NCE with alternative density-estimation methods such as importance sampling.Source: §5.4.1 · p. 71

5.4.2 SRE16

• The reported LibriSpeech experiments use relatively clean speech and manually created trial lists, preventing comparison with other speaker-verification systems.Source: §5.4.2 · p. 71

5.4.3 CPC x-vectors

• Planned extensions include CPC experiments with x-vectors, language identification, and domain adaptation for speaker recognition.The text notes signs that CPC may not generalize well to unseen conditions.Source: §5.4.3 · p. 72, §5.4.4 · p. 72, §5.4.5 · p. 73

5.4.4 Language Identification

• CPC is hypothesized to learn language information in addition to speaker information because it captures global speech information.Source: §5.4.4 · p. 72