Source-linked AI summary
Fine-tuning wav2vec2 for speaker recognition
Nik Vaessen, David A. van Leeuwen
TL;DR
This paper investigates whether wav2vec2’s pretrained weights and output-sequence pooling can support speaker recognition. It adapts the framework with single-utterance and paired-utterance classifiers; w2v2-aam is the strongest wav2vec2 variant, though ECAPA-TDNN performs best across the test sets.
Problem
The paper asks whether pretrained wav2vec2 weights can support speaker recognition and how to pool its variable-length output into a fixed-size speaker embedding.
Source: §1 · p. 1Method
The authors adapt wav2vec2 with pooled single-utterance speaker classifiers using CE or AAM loss and a paired-utterance classification variant.
Source: §3.2 · p. 2, §3.2.1 · p. 2, §3.3 · p. 3Results
ECAPA-TDNN performs best on all test sets, while w2v2-aam is the best wav2vec2 variant and w2v2-ce and w2v2-aam improve on x-vector.
Source: §5.1 · p. 4Takeaways & Limitations
The authors conclude that speech-recognition pretraining weights are useful for speaker recognition, while paired-utterance classification did not perform well.
Source: §6 · p. 4The paired-utterance classification approach did not perform well.
Source: §6 · p. 4
Abstract
from arXiv · showhide
This paper explores applying the wav2vec2 framework to speaker recognition instead of speech recognition. We study the effectiveness of the pre-trained weights on the speaker recognition task, and how to pool the wav2vec2 output sequence into a fixed-length speaker embedding. To adapt the framework to speaker recognition, we propose a single-utterance classification variant with CE or AAM softmax loss, and an utterance-pair classification variant with BCE loss. Our best performing variant, w2v2-aam, achieves a 1.88% EER on the extended voxceleb1 test set compared to 1.69% EER with an ECAPA-TDNN baseline. Code is available at https://github.com/nikvaessen/w2v2-speaker.
1. INTRODUCTION
Building on wav2vec2’s success in speech recognition, this study asks whether its pretrained weights can transfer to speaker recognition and how to summarize its output sequence for that task.
1. INTRODUCTION
- The study extends the pretraining-and-fine-tuning approach used in NLP and recently applied to speech recognition with wav2vec2 to speaker recognition.The NLP examples cited are BERT, XLNet, and T5; wav2vec2 speech recognition is cited as.Source: §1 · p. 1
- Unlike speech recognition, which resembles token tagging, speaker recognition resembles single-sentence or sentence-pair classification and may benefit from a sequence-summary class token.The paper frames this analogy using BERT’s class-token role in classification.Source: §1 · p. 1
- The study tests whether pretrained wav2vec2 weights initialize speaker recognition effectively and whether pooling can replace a class token, including which pooling method works best.The authors also ask whether wav2vec2 can be adapted without the class token and next-sentence prediction task.Source: §1 · p. 1
2. RELATED WORK
Prior work applies wav2vec2 to speaker and language identification, emotion recognition, and a broader speech benchmark.
2. RELATED WORK
- Wav2vec2 has been used for speaker recognition and language identification in single-task and multitask settings.Source: §2 · p. 1
- For language identification, one study reports good performance across 25 languages but replaces raw waveforms with log-mel spectrogram inputs [6].Source: §2 · p. 1
- Emotion-recognition work uses either frozen wav2vec2 embeddings with a learnable downstream model or fine-tunes wav2vec2 with CTC loss on emotion-labeled phonetic units [8].Source: §2 · p. 1
- LeBenchmark uses wav2vec2 models across speech recognition, spoken language understanding, emotion recognition, and speech translation.Source: §2 · p. 1
3.1. The wav2vec2 architecture
The wav2vec2 fine-tuning architecture transforms raw audio into a sequence of representations, adds projected and positional information, and processes it with a transformer encoder.
3.1. The wav2vec2 architecture
- Figure 1 presents the wav2vec2 fine-tuning architecture, with shapes specified for a batch of B audio samples lasting 3 seconds.Source: §3.1 · p. 1, Figure 1 · p. 2
3.1.1. Feature extraction
- The feature extractor converts normalized raw audio into a sequence of 512-dimensional representations using seven 1D convolutional layers.Each output vector has a 20 ms receptive field.Source: §3.1.1 · p. 2
3.1.2. Projection, SpecAugment & positional embedding
- A shared layer projects each feature vector to 768 dimensions; masking and a grouped convolution create relative positional embeddings that expand the receptive field to 2.5 seconds.Layer normalization and dropout are applied after positional embeddings are added to the projected sequence.Source: §3.1.2 · p. 2
3.1.3. Transformer
- The encoder applies 12 transformer layers with 12-headed self-attention and feed-forward networks; LayerDrop can skip layers, and outputs retain local and potentially global information.Source: §3.1.3 · p. 2
3.2. Three wav2vec2 variants for speaker recognition
The paper adapts wav2vec2 to speaker recognition through single-utterance classification and paired-utterance scoring, rather than tokenwise CTC prediction.
3.2. Three wav2vec2 variants for speaker recognition
- The paper proposes two speaker-recognition adaptations to wav2vec2’s speech-recognition design, which labels output embeddings independently and uses CTC loss.Source: §3.2 · p. 2
3.2.1. Speaker recognition as single-utterance classification
- The single-utterance variants pool the sequence, classify speakers with a fully connected layer using CE or AAM loss, and score trials by cosine similarity between pooled embeddings.They are named w2v2-ce and w2v2-aam.Source: §3.2.1 · p. 2
3.2.2. Speaker recognition as utterance-pair classification
- The paired-utterance variant concatenates independently encoded utterances with start, separator, and end tokens, then uses the start-token output for logistic-regression scoring with BCE loss.Training uses batches containing 8 speakers, 4 utterances per speaker, and 16 same- and different-speaker pairs; this variant is w2v2-bce.Source: §3.2.2 · pp. 2–3
3.3. Pooling methods
The paper compares ways to compress wav2vec2’s variable-length output sequence into a fixed-size speaker embedding, including statistical pooling, positional selection, and a learned-sequence alternative.
3.3. Pooling methods
- Statistical pooling aggregates each embedding dimension over time using mean, max, mean&std, or quantiles.Mean&std doubles embedding dimensionality; quantile pooling expands each dimension five-fold using quantiles 0, 0.25, 0.5, 0.75, and 1.Source: §3.3 · p. 3
- Positional pooling selects the first, middle, last, or a random wav2vec2 output embedding as the speaker representation.Source: §3.3 · p. 3
- First&cls prepends an all-ones start token and uses its first output embedding, although the token has no meaningful prior.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments use VoxCeleb2 for training and VoxCeleb1 for evaluation, comparing wav2vec2 adaptations with speaker-recognition baselines under a shared computational budget. They also test pooling choices and ablate initialization, regularization, batch size, and learning-rate schedules.
4.1. Data
- Training uses VoxCeleb2 dev data, while evaluation uses the cleaned original, extended, and hard VoxCeleb1 test sets without speaker overlap.The validation set is based on about 2% of VoxCeleb2 dev data, with no recording overlap; it uses 5,000 same-speaker and 5,000 different-speaker trials.Source: §4.1 · p. 3
4.2. Computational budget and fair comparison
- Models are compared under a shared budget, using 3-second samples from 66 files per batch and 100,000 Adam iterations with a OneCycle schedule.No data augmentation is used; each network is trained with a batch size of 3.2M audio samples.Source: §4.2 · p. 3
4.3. Baseline systems
- The baseline comparison includes x-vector and ECAPA-TDNN alongside w2v2-ce, w2v2-aam, and w2v2-bce, with mean&std pooling for the CE and AAM variants.The wav2vec2 feature encoder is frozen; w2v2-aam and ECAPA-TDNN use AAM softmax loss.Source: §4.3 · p. 3
4.4. Variation in pooling
- The pooling comparison varies pooling methods only for w2v2-ce and w2v2-aam, using the baseline training settings and mean&std-tuned learning rate.Source: §4.4 · p. 3
4.5. Ablation study
- Ablations test feature-extractor freezing and pre-trained initialization, regularization, batch size, and learning-rate schedules.Schedules include constant rates, exponential decay, and a tri-stage schedule; the tri-stage schedule warms up, holds a constant rate, then decays.Source: §4.5 · pp. 3–4
5. RESULTS
ECAPA-TDNN performs best across the test sets, while w2v2-aam is the strongest wav2vec2 variant. First&cls leads the pooling comparison, and ablations indicate that pre-training, regularization, and warm-up learning-rate schedules matter for performance and stability.
5.1. Baseline comparison
- ECAPA-TDNN performs best on all test sets; w2v2-aam is the best wav2vec2 variant, both w2v2-ce and w2v2-aam improve on x-vector, and w2v2-bce performs worst.Source: §5.1 · p. 4
5.2. Pooling methods
- First&cls pooling performs best for both w2v2-ce and w2v2-aam, with a more pronounced advantage for w2v2-aam.Random pooling during evaluation degrades EER by 0.2 percentage points for networks trained with first&cls pooling, and ensembling different output embeddings does not improve performance.Source: §5.2 · p. 4
5.3. Ablation study
- Random initialization causes a large performance degradation, while the speech-recognition regularization settings also benefit speaker recognition fine-tuning.Freezing the feature extractor worsens performance but improves stability across seeds; increasing batch size beyond 3.2M samples does not improve performance, and warm-up schedules are critical for stable, good performance.Source: §5.3 · p. 4, Table 3 · p. 4
6. CONCLUSION AND FUTURE WORK
The paper concludes that wav2vec2 can be adapted to speaker recognition and that speech-recognition pre-trained weights are useful for this task. First&cls results motivate class-token pre-training, while paired-utterance classification remains a direction for further work.
6. CONCLUSION AND FUTURE WORK
- Wav2vec2 can be adapted to speaker recognition, and its speech-recognition pre-trained weights are useful for speaker-recognition fine-tuning.Source: §6 · p. 4
- Good results with first&cls pooling suggest that including a class token during pre-training is promising future work.Source: §6 · p. 4
- Paired-utterance classification did not perform well; future work could limit attention to the opposite utterance to simplify learning.Source: §6 · p. 4