Source-linked AI summary
Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System
Weicheng Cai, Jinkun Chen, Ming Li
TL;DR
The paper addresses how to build utterance-level speaker and language recognition systems that handle variable-length speech and produce discriminative representations for open-set verification. It develops a unified end-to-end framework that compares pooling and encoding layers and introduces discriminative loss functions. Experiments on VoxCeleb and NIST LRE 07 report significant improvements from suitable encoding layers and loss functions.
Problem
Speaker and language recognition need robust utterance-level representations from variable-length speech, while open-set verification requires discriminative large-margin features.
Method
The paper develops a unified end-to-end system using CNN feature extraction, temporal pooling or learnable dictionary encoding, and center or A-Softmax loss for open-set verification.
Results
Suitable encoding layers and loss functions significantly improve end-to-end performance across VoxCeleb speaker recognition and NIST LRE 07 language recognition tasks.
Takeaways & Limitations
Self-attentive pooling and learnable dictionary encoding outperform temporal average pooling, while discriminative losses generally outperform softmax loss.
Takeaways & Limitations
The comparison focuses on systems that do not require additional transcribed speech data or an extra DNN acoustic model.
Abstract
from arXiv · showhide
In this paper, we explore the encoding/pooling layer and loss function in the end-to-end speaker and language recognition system. First, a unified and interpretable end-to-end system for both speaker and language recognition is developed. It accepts variable-length input and produces an utterance level result. In the end-to-end system, the encoding layer plays a role in aggregating the variable-length input sequence into an utterance level representation. Besides the basic temporal average pooling, we introduce a self-attentive pooling layer and a learnable dictionary encoding layer to get the utterance level representation. In terms of loss function for open-set speaker verification, to get more discriminative speaker embedding, center loss and angular softmax loss is introduced in the end-to-end system. Experimental results on Voxceleb and NIST LRE 07 datasets show that the performance of end-to-end learning system could be significantly improved by the proposed encoding layer and loss function.
1. Introduction
Language and speaker recognition require robust utterance-level representations from variable-length speech, with open-set verification additionally requiring discriminative feature spaces. The paper therefore develops a unified end-to-end system and investigates its encoding layers and loss functions.
- Motivation: Language and text-independent speaker recognition are utterance-level sequence-to-one tasks that seek time-invariant representations from variable-length input.Unlike speech recognition, they retrieve information about an entire utterance rather than specific word content.
- Conventional systems: The conventional pipeline uses local feature descriptors, a dictionary, and vector encoding to aggregate variable-length sequences into utterance-level vectors.Examples include Fbank, MFCC, PLP, SDC, bottleneck, PPP, and tandem features, with dictionaries learned by K-means, GMM, or DNNs.
- End-to-end motivation: End-to-end learning offers an alternative to hand-crafted i-vector pipelines by placing a general encoding layer above a convolutional neural network.The learnable dictionary encoding layer connects conventional GMM Supervector processing with end-to-end neural networks.
- Recognition settings: Closed-set identification is treated as classification, whereas open-set speaker verification is treated as metric learning requiring discriminative large-margin features.In open-set verification, testing identities are not restricted to identities seen during training.
- Paper objective: The paper builds a unified end-to-end system for language and speaker recognition and focuses on improving it through encoding layers and loss functions.The pipeline includes input data, frame-level feature extraction, encoding, loss, and similarity-metric modules.
2. End-to-End System Overview
The end-to-end framework processes speech of arbitrary duration by extracting frame-level features and aggregating them into a fixed-dimensional utterance representation. A similarity metric is added specifically for open-set verification.
- Framework: The framework accepts variable-length speech segments and produces an utterance-level result.This accommodates testing speech whose duration is not known in advance.
- Output and verification: The utterance representation is passed through a fully connected layer and output layer, with an additional similarity metric for open-set verification.The similarity metric is specifically designated for the open-set verification task.
- Frame-level feature extraction: A convolutional neural network extracts high-level local patterns from input features such as log mel-filterbank energies.The resulting frame-level representations retain temporal order before aggregation.
- Encoding: The encoding layer aggregates temporally ordered frame representations into a fixed-dimensional utterance-level representation.This aggregation resolves the mismatch between variable-length input sequences and utterance-level outputs.
3. Encoding layer
The encoding layer converts variable-length frame sequences into fixed-dimensional utterance-level representations. The paper compares temporal average pooling, self-attentive pooling, and learnable dictionary encoding as alternative aggregation mechanisms.
- Temporal average pooling: Temporal average pooling equally aggregates CNN-extracted features across time.It is the basic pooling approach described for end-to-end architectures.
- Self-attentive pooling: Self-attentive pooling weights frames according to their importance for classification before aggregating them into an utterance representation.A learned context vector and softmax-derived weights identify informative frames.
- Self-attentive pooling: The self-attention mechanism is self-contained and does not require extra guiding source information.This distinguishes the described architecture from an earlier attention-based recurrent approach that used language-category pretraining.
- Learnable dictionary encoding: Learnable dictionary encoding combines dictionary learning and vector encoding into one end-to-end trainable layer.It imitates the GMM Supervector mechanism while learning directly from the loss function.
- Learnable dictionary encoding: The LDE layer transforms a D × L temporal feature sequence into a length-independent D × C utterance-level representation.It assigns features to learnable dictionary components, aggregates residual vectors, and concatenates the resulting encoder outputs.
4. Loss function
The paper replaces explicitly separated front-end and back-end processing with unified end-to-end losses, addressing the need for discriminative embeddings in open-set speaker verification. It examines softmax, center, and angular softmax objectives for closed-set identification and open-set verification.
- 4.1. Loss function for closed-set identification: Closed-set identification jointly learns the front-end feature extractor and back-end classifier under a unified softmax loss.The classifier uses the penultimate representation, target label, and final-layer weights and bias.
- 4.2. Loss function for open-set verification: Open-set speaker verification requires embeddings that are both separable and discriminative because testing identities cannot all be pre-collected for training.Consequently, basic label-prediction softmax is not always applicable to this setting.
- 4.2. Loss function for open-set verification: Pairwise contrastive and triplet losses formulate open-set verification as metric learning but require non-trivial pair or triplet mining.The mining procedure is described as time-consuming and performance-sensitive.
- 4.2.1. Center loss: Center loss learns one center per class and penalizes distances between features and their corresponding centers while preserving inter-class separability through joint softmax supervision.The scalar λ balances center loss against softmax loss, and λ = 0 recovers conventional softmax.
- 4.2.2. Angular Softmax loss: Angular softmax learns an angular margin with integer m controlling its size; m = 1 yields modified softmax, while m ≥ 2 imposes stronger angular classification requirements.Its geometric formulation produces an angular distance metric on a hypersphere and separates features by angular margins.
5. Experiments
Experiments evaluate unified end-to-end systems for speaker and language recognition on VoxCeleb and NIST LRE 07, comparing encoding layers, losses, and similarity metrics. The results favor LDE encoding and discriminative losses, with task-dependent gains over TAP and i-vector baselines.
- Datasets and tasks: Experiments use VoxCeleb for speaker verification and identification, and NIST LRE 07 for closed-set language detection across 3-, 10-, and 30-second durations.VoxCeleb verification uses Cdet and EER; identification reports top-1 and top-5 accuracy, while LRE 07 uses Cavg and EER.
- Overall comparison: End-to-end learning systems significantly outperform the conventional i-vector approach on both speaker-recognition and language-recognition tasks.The comparison covers the verification and identification results in Tables 2–4.
- Encoding layers: Both SAP and LDE outperform TAP, while LDE also surpasses SAP across the reported experiments.These comparisons concern the encoding layer used to form utterance-level representations.
- Losses and similarity metrics: Discriminative center loss and A-Softmax generally outperform softmax, whereas PLDA provides little or negative gain for end-to-end systems trained with discriminative losses.PLDA reduces error in the conventional i-vector system but can worsen end-to-end results.
- Reported gains: 25%, 45%, and 63% relative error reductions are achieved by CNN-LDE versus CNN-TAP on NIST LRE 07 3s, 10s, and 30s tasks.CNN-LDE also achieves 12% relative error reduction for VoxCeleb speaker identification; LDE-A-Softmax verification reaches Cdet 0.441 and EER 4.56%.
6. Conclusions
The paper develops a unified, interpretable end-to-end system for speaker and language recognition that accepts variable-length input and produces utterance-level results. It improves performance by using suitable encoding layers and loss functions, including SAP, LDE, center loss, and A-Softmax.
- Conclusions: A unified end-to-end system is developed for speaker and language recognition, accepting variable-length input and producing utterance-level results.The system investigates encoding layers and loss functions within one pipeline.
- Conclusions: SAP and LDE replace basic temporal average pooling to produce utterance-level representations from variable-length sequences.The paper introduces both self-attentive pooling and learnable dictionary encoding.
- Conclusions: Center loss and A-Softmax loss are introduced to obtain more discriminative speaker embeddings for open-set speaker verification.These losses address the speaker-embedding component of the end-to-end system.
- Conclusions: Experimental results show that suitable encoding layers and loss functions significantly improve end-to-end system performance.The reported experiments use VoxCeleb and NIST LRE 07 datasets.