Source-linked AI summary
Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification
Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, Michael Zeng
TL;DR
Large-scale labeled data for ASV is difficult to collect, motivating investigation of speech representations learned from large-scale unlabeled data. The paper evaluates these representations with ECAPA-TDNN, using learnable weighted averages across hidden layers, and finds they outperform handcrafted FBank features, with lower layers capturing more speaker-related information.
Problem
Large-scale labeled ASV data is difficult to collect because speaker information raises privacy concerns, while existing unsupervised speaker-verification methods may assume one speaker per utterance.
Method
The study evaluates representations from pre-trained models with different learning methods, model sizes, and training datasets, feeding either final-layer or learnably weighted all-hidden-layer representations into ECAPA-TDNN.
Results
46% relative improvement over FBank was observed for the weighted-average hidden representation, which outperformed the handcrafted feature in the experiments.
Takeaways & Limitations
Learnable layer weighting helps leverage speaker-related information across the pre-trained model, with lower layers capturing more speaker-related information than higher layers.
Abstract
from arXiv · showhide
The speech representations learned from large-scale unlabeled data have shown better generalizability than those from supervised learning and thus attract a lot of interest to be applied for various downstream tasks. In this paper, we explore the limits of speech representations learned by different self-supervised objectives and datasets for automatic speaker verification (ASV), especially with a well-recognized SOTA ASV model, ECAPA-TDNN [1], as a downstream model. The representations from all hidden layers of the pre-trained model are firstly averaged with learnable weights and then fed into the ECAPA-TDNN as input features. The experimental results on Voxceleb dataset show that the weighted average representation is significantly superior to FBank, a conventional handcrafted feature for ASV. Our best single system achieves 0.537%, 0.569%, and 1.180% equal error rate (EER) on the three official trials of VoxCeleb1, separately. Accordingly, the ensemble system with three pre-trained models can further improve the EER to 0.479%, 0.536% and 1.023%. Among the three evaluation trials, our best system outperforms the winner system [2] of the VoxCeleb Speaker Recognition Challenge 2021 (VoxSRC2021) on the VoxCeleb1-E trial.
1. INTRODUCTION
This paper investigates whether speech representations learned from large-scale unlabeled data can improve a state-of-the-art speaker verification model on VoxCeleb. It uses weighted representations from all hidden layers and evaluates models trained with different objectives, sizes, and datasets.
- Large-scale labeled data improves ASV but is difficult to collect because speaker information raises privacy concerns.
- Speech representations from Wav2Vec 2.0 and HuBERT reduce reliance on the single-speaker-per-utterance assumption used by earlier unsupervised speaker-verification methods.
- The paper is the first reported attempt to use large-scale unlabeled speech representations to improve ECAPA-TDNN on VoxCeleb.
- Weighted averaging of representations from all hidden layers is used instead of relying only on the pre-trained model’s final layer.
- The study compares pre-trained models across learning methods, model sizes, and large-scale training datasets.
- Learnable weights provide a layer-wise analysis of speaker information embedded in pre-trained models.
2. RELATED WORK
Prior ASV work relied on handcrafted acoustic features, task-specific feature learning, and self-supervised speech representations. Self-supervised pre-training uses unlabeled speech to produce representations that can support downstream tasks with limited labeled data.
- FBank and MFCC are common handcrafted acoustic features that characterize speech in the frequency domain.
- Feature engineering for handcrafted features includes delta features that capture temporal dynamics.
- Self-supervised speech representation learning is commonly grouped into reconstruction, predictive, and related unlabeled-data objectives.
- The paper’s Figure 1 presents how representations from a pre-trained model are leveraged for downstream processing.
3. METHODS
The method feeds representations from self-supervised speech models into ECAPA-TDNN for speaker verification. It learns a weighted combination of hidden layers and uses staged optimization before fine-tuning the full system.
- 3.1. Pre-train for Representation Learning: Wav2Vec 2.0, HuBERT, and UniSpeech-SAT provide representations learned with contrastive, masked-label prediction, and utterance-wise speaker-aware objectives.
- 3.1. Pre-train for Representation Learning: The pre-trained models encode waveform samples with a CNN and transform the resulting feature sequence into layer-specific hidden states.
- 3.2. Downstream Speaker Verification Model: ECAPA-TDNN is selected as the downstream speaker-verification model to push performance beyond earlier x-vector-based systems.
- 3.2.2. Explore Speaker Information in Pre-trained Model: The weighted-layer design addresses the possibility that final-layer representations emphasize the pre-training objective rather than speaker information.
- 3.2.2. Explore Speaker Information in Pre-trained Model: Learnable weights combine hidden states from every pre-trained-model layer into frame representations that replace FBank inputs to ECAPA-TDNN.
- 3.2.2. Explore Speaker Information in Pre-trained Model: Training first updates ECAPA-TDNN and hidden-state weights with the pre-trained model fixed, then fine-tunes all parameters.
4. EXPERIMENTAL SETUP
The experiments evaluate pre-trained speech representations and ECAPA-TDNN using Voxceleb1 and Voxceleb2, with handcrafted Fbank baselines, augmentation, and staged fine-tuning procedures.
- Datasets and features: Voxceleb1 and Voxceleb2 provide the training and evaluation datasets, with Vox1-O, Vox1-E, and Vox1-H as official trials.The baseline uses 40-dimensional Fbank features with 25 ms windows and 10 ms frame shifts, without VAD.
- Pre-training data: UniSpeech-SAT models use 90k hours from Librivox, VoxPopuli, and Gigaspeech, while the layer count records only transformer layers.The table caption specifies 60k, 24k, and 10k hours for the three datasets, respectively.
- Models: UniSpeech-SAT, HuBERT, and Wav2vec2.0 pre-trained models are compared with a small ECAPA-TDNN downstream model.UniSpeech-SAT explicitly models speaker information during pre-training and uses additional public or synthesized data.
- Training: All models use Additive Angular Margin Loss with margin 0.2 and randomly sampled 3s training segments.The two-stage pipeline fixes the pre-trained model for 10 epochs, then fine-tunes all parameters for 5 epochs.
- Scoring and calibration: Cosine similarity, adaptive s-norm, a 600-speaker imposter cohort, and quality-aware score calibration are used during evaluation.The calibration model is trained on 30k randomly generated trials from the Voxceleb2 test set.
5. EVALUATION RESULTS
The results show that pre-trained representations outperform Fbank, especially when hidden layers are weighted, and that larger models, fine-tuning, and score-level strategies further improve ASV performance.
- Handcrafted-feature comparison: 46% relative improvement over Fbank is achieved by weighted averaging hidden representations on Vox1-O before augmentation.Both last-layer and all-hidden-layer representations outperform Fbank, while augmentation further improves every result.
- Model and training comparisons: Fine-tuning the pre-trained model improves results beyond using it as a fixed feature extractor, with larger gains on Vox2 dev than Vox1 dev.Wav2vec2.0 Large (XLSR) and UniSpeech-SAT Large outperform HuBERT Large after fine-tuning.
- Model and training comparisons: ∼30% relative EER improvement on all three Voxceleb1 trials is achieved by the best pre-trained representation model compared with Fbank.These experiments use the larger Voxceleb2 development dataset and compare pre-trained models and training strategies.
- Challenge comparison: The best system exceeds the VoxSRC2021 winner on the Voxceleb1-E trial after large-margin fine-tuning and quality-aware score calibration.The cited challenge scores are Vox1-O: 0.461, Vox1-E: 0.634, and Vox1-H: 0.993.
- Layer-wise analysis: Lower pre-trained-model layers receive more discriminative weight than higher layers for ASV.The paper relates this pattern to pre-training objectives that are more closely tied to speech recognition than speaker verification.
6. CONCLUSION
The paper concludes that learnable weighted combinations of hidden representations improve ECAPA-TDNN speaker verification and reveal stronger speaker information in lower layers.
- Conclusion: Pre-trained representations from large-scale unlabeled data outperform handcrafted Fbank features for the evaluated speaker-verification systems.The conclusion attributes further gains to automatically learned weights over all hidden states.
- Conclusion: Lower layers capture more speaker-related information than higher layers in the evaluated pre-trained models.The authors identify a remaining performance gap on two evaluation sets relative to the VoxSRC2021 best system.
- Future work: The authors plan to incorporate the competing system’s more aggressive augmentation and dedicated training objectives to improve performance further.This is presented as future work rather than an evaluated result in the paper.