Source-linked AI summary
Exploring wav2vec 2.0 on speaker verification and language identification
Zhiyun Fan, Meng Li, Shiyu Zhou, Bo Xu
TL;DR
The paper asks whether wav2vec 2.0’s self-supervised representations can support speaker verification and language identification, where pre-training work is limited. It analyzes these representations and fine-tunes them for both tasks, obtaining 3.61% EER on VoxCeleb1 and unified multi-task modeling.
Problem
Pre-training methods have mostly targeted speech recognition, leaving their usefulness for speaker verification and language identification unclear.
Method
The paper visualizes pre-trained w2v-encoder representations, fine-tunes the model separately for speaker verification and language identification, and jointly fine-tunes one model for both tasks.
Results
3.61% EER on VoxCeleb1 and 12.02% EER under the 1 second condition plus 3.47% under the full-length condition on AP17-OLR are reported.
Takeaways & Limitations
Pre-trained wav2vec 2.0 representations contain speaker- and language-related characteristics, and multi-task learning enables unified modeling of both tasks with fewer parameters.
Abstract
from arXiv · showhide
Wav2vec 2.0 is a recently proposed self-supervised framework for speech representation learning. It follows a two-stage training process of pre-training and fine-tuning, and performs well in speech recognition tasks especially ultra-low resource cases. In this work, we attempt to extend self-supervised framework to speaker verification and language identification. First, we use some preliminary experiments to indicate that wav2vec 2.0 can capture the information about the speaker and language. Then we demonstrate the effectiveness of wav2vec 2.0 on the two tasks respectively. For speaker verification, we obtain a new state-of-the-art result, Equal Error Rate (EER) of 3.61% on the VoxCeleb1 dataset. For language identification, we obtain an EER of 12.02% on 1 second condition and an EER of 3.47% on full-length condition of the AP17-OLR dataset. Finally, we utilize one model to achieve the unified modeling by the multi-task learning for the two tasks.
1. INTRODUCTION
The paper extends wav2vec 2.0’s self-supervised pre-training beyond speech recognition to speaker verification and language identification. Preliminary analyses find speaker- and language-discriminative information in the representations, supporting fine-tuning for both tasks.
- Most speech self-supervised pre-training methods have been applied to speech recognition, while pre-training for speaker verification and language identification remains limited.
- Wav2vec 2.0 uses a two-stage pre-training and fine-tuning framework, with a CNN encoder, Transformer, quantization module, and contrastive objective.
- Pre-trained w2v-encoder context representations distinguish among speakers and languages even though wav2vec 2.0 pre-training is problem-agnostic.
- Lower-layer representations show stronger speaker and language distinguishability, matching information needed by speaker verification and language identification.
- 3.61% EER is obtained on VoxCeleb1, while language identification reaches 12.02% EER under the 1 second condition and 3.47% under the full-length condition on AP17-OLR.
- Multi-task learning fine-tunes one model for speaker verification and language identification simultaneously.
2. METHOD
The method pre-trains wav2vec 2.0 with masked contrastive prediction, then adds task-specific pooling and classification layers for downstream fine-tuning. A multi-task variant jointly optimizes speaker and language classification while balancing unequal data volumes and convergence rates.
- 2.1. Pre-training of wav2vec 2.0: Wav2vec 2.0 pre-training uses a CNN feature encoder, Transformer context network, and quantization module; its architecture largely matches fine-tuning except for quantization and extra output layers.
- 2.1. Pre-training of wav2vec 2.0: The CNN maps raw audio X to latent representations Z, while the Transformer masks sampled time steps and produces contextual representations C.
- 2.1. Pre-training of wav2vec 2.0: The quantization module discretizes Z into Q using two codebooks, each with 320 entries of size 128, selected through differentiable Gumbel softmax.
- 2.1. Pre-training of wav2vec 2.0: The pre-training loss combines an L2 penalty, contrastive loss against 100 distractors, and diversity loss; β = 10 and α = 0.1.
- 2.2. Fine-tuning: Downstream fine-tuning adds average pooling and a fully connected classifier, using cross-entropy and AM-softmax for speaker classification.
- 2.2. Fine-tuning: Multi-task fine-tuning uses shared pooling with parallel speaker and language classifiers, optimizing a weighted sum of their cross-entropy losses.
- 2.2. Fine-tuning: Equal-probability sampling balances speaker and language datasets, while development-set loss weighting addresses their inconsistent convergence speeds.
3. EXPERIMENTS
Experiments test whether wav2vec 2.0 retains speaker and language information, transferability to speaker verification and language identification, and unified multi-task modeling.
- Experimental design: The study uses speaker and language factors to evaluate whether problem-agnostic wav2vec 2.0 pre-training supports speaker verification and language identification.VoxCeleb1 is used for speaker verification, while AP17-OLR is used for language identification.
- Experimental design: The experiments use M-nofinetune, M-sv, M-lid, and M-multi to compare pretrained, task-specific, and multi-task configurations.M-nofinetune is pretrained on Librispeech; M-sv and M-lid are fine-tuned separately, while M-multi is fine-tuned on both datasets simultaneously.
- 3.3. Feasibility analysis: Pretrained representations show speaker and language distinguishability, with stronger distinguishability in lower Transformer layers.The representations are visualized with t-SNE across layer1, layer6, and layer12 for speakers and languages.
- 3.4. Speaker verification: Pretraining improves speaker verification substantially: M-sv reaches an EER of 3.61% versus 24.28% without pretraining on VoxCeleb1.M-sv also outperforms the listed baselines and is reported as a new state-of-the-art result on VoxCeleb1.
- 3.5. Language identification: M-lid outperforms training from scratch on both AP17-OLR duration conditions and shows a clear advantage over the released baselines, though it remains far from the best results.The paper reports that pretraining also alleviates serious overfitting observed without pretraining.
- 3.6. Multi-task system: Multi-task training slightly reduces performance relative to single-task training but achieves good results with fewer parameters for both tasks.The unified model is intended to simplify fine-tuning and reduce storage demand.
4. CONCLUSION
The paper finds wav2vec 2.0 feasible and effective for speaker verification and language identification, then uses multi-task learning for unified modeling of both tasks.
- Pre-trained wav2vec 2.0 features distinguish speakers and languages, with stronger distinctions in lower layers.
- The pre-trained model achieves competitive results on the VoxCeleb1 and AP17-OLR speaker verification and language identification datasets.
- Multi-task learning simplifies fine-tuning across the two tasks and reduces parameters through unified modeling.
- Future work will extend wav2vec 2.0 to more speech processing tasks using multi-task learning.