Pushing the limits of raw waveform speaker recognition

1. Introduction

• Raw-waveform models trail handcrafted-feature counterparts: the latest cited model has 1.29% EER, versus under 1% for ECAPA-TDNN and variants.The paper proposes a new architecture to address this performance gap.Source: §1 · p. 1
• Prior speaker-recognition self-supervised works had not adopted end-to-end raw-waveform architectures, leaving that approach unexplored.Source: §1 · p. 1
• RawNet3 demonstrates EER under 1% on the VoxCeleb1 evaluation protocol and explores raw-waveform speaker verification with self-supervised learning.Source: §1 · p. 1
• The paper reports that its self-supervised model outperforms contrastive-based existing works and that self-supervised pre-training is effective in a semi-supervised scenario.Source: §1 · p. 1

2. Proposed architecture

• RawNet3 draws on RawNet2 and ECAPA-TDNN, processing waveforms into time-frequency representations with learned complex-valued analytic filterbanks.Source: §2 · p. 2
• Smaller filterbank stride improves performance but slows processing, while larger stride speeds processing at a performance cost.The default kernel length and stride are 251 and 48, generating approximately 15 ms windows and 3 ms shifts per frame.Source: §2 · p. 2
• Three residual backbone blocks concatenate their outputs and feed the sum of the first two block outputs into the third.Max pooling in the first two blocks reduces the unusually long raw-waveform sequences and is reported to mitigate overfitting.Source: §2 · p. 2
• The AFMS-Res2MP-block is based on Res2Net, uses RawNet2 AFMS instead of squeeze-excitation, and can apply max pooling before AFMS.Channel- and context-dependent statistic pooling replaces RawNet2’s uni-directional gated recurrent unit.Source: §2 · p. 2

3.1. Supervised learning: classification with AAM-softmax

• Supervised training treats speaker recognition as closed-set classification, then removes the classification head and uses the model to extract speaker embeddings.Source: §3.1 · p. 2
• AAM-softmax, also called ArcFace, enforces larger gaps between nearest speakers using a scale factor and margin.The loss uses the dot product between the L2-normalized speaker embedding and classification-head weights.Source: §3.1 · p. 2

3.2. Self-supervised learning: DINO

• DINO compares multiple views of one utterance with a teacher–student self-distillation setup; the teacher sees global views while the student sees global and local views.Only student parameters are directly updated; teacher parameters follow an exponential moving average of the student.Source: §3.2 · p. 2
• DINO sharpens teacher and student outputs and centers the teacher output to avoid representation collapse.The center is updated by exponential moving average, and the paper argues that balancing centering and sharpening permits training without architecture differentiation.Source: §3.2 · p. 2

4. Experiments

• Experiments cover supervised classification, DINO self-supervised training, and DINO pre-training followed by supervised fine-tuning.The semi-supervised setup pre-trains on VoxCeleb2 development data and fine-tunes on VoxCeleb1 development data.Source: §4 · p. 3

4.1. Dataset

• VoxCeleb1 and VoxCeleb2 contain speech collected in the wild; their development sets support supervised and self-supervised training, respectively.VoxCeleb1 has 1,211 development speakers and 40 evaluation speakers; VoxCeleb2 has 5,994 development speakers and 118 evaluation speakers.Source: §4.1 · p. 3, §4.2 · p. 3

4.2. Data configurations and metrics

• All experiments use the VoxCeleb1-O protocol; EER is primary, and minDCF is also reported.Tables cover DINO self-supervised results on VoxCeleb2 development data, fine-tuning on VoxCeleb1 development data, and supervised comparisons with recent literature.Source: §4.2 · p. 3, Table 2 · p. 3, Table 3 · p. 3

4.3. Implementation details

• Training uses PyTorch and four Nvidia V100 GPUs, with Adam and SGDR scheduling; supervised and DINO training use distinct schedules and augmentations.The supervised model trains for 40 epochs with three-second crops; self-supervised training runs for 80 epochs with two global and five local views.Source: §4.3 · p. 3

5.1. Supervised learning

• 1.05% EER for RawNet3 improves on RawNet2's 2.48% EER; excluding channel and context statistical pooling degrades EER to 1.45%.Excluding either logarithm or mean-subtraction normalisation also degrades performance.Source: §5.1 · p. 4
• 0.89% EER is reached with stride size 10; smaller filterbank strides improve performance at greater computational cost.Larger strides shorten the frame sequence and reduce computation, at the cost of performance.Source: §5.1 · p. 4

5.2. Self-supervised learning

• 5.74% EER is reported for DINO without ground-truth labels; logarithm and mean normalisation materially affect its performance.The paper relates this sensitivity to DINO's same-utterance, different-augmentation comparisons and potentially weaker channel-variation robustness.Source: §5.2 · p. 4
• 5.40% EER follows last-layer normalisation, while applying warm-up does not noticeably affect performance.Both hyperparameter changes behaved differently than anticipated.Source: §5.2 · p. 4

5.4. Comparison with recent literature

• Among the compared self-supervised models, DINO without negative pairs demonstrates the best performance.The five other works are described as contrastive-learning variants.Source: Table 5 · p. 4, §5.4 · p. 4

5.3. Semi-supervised learning

• 2.18% EER with DINO pre-training compares with 2.98% without it, a 25% relative performance improvement in the semi-supervised scenario.The scenario uses ground-truth labels only for the smaller VoxCeleb1 development set.Source: §5.3 · p. 4

5.4. Comparison with recent literature

• 0.89% EER makes RawNet3 competitive with recent supervised systems; the best model reports 0.66% EER, while RawNet3 improves over the prior raw-waveform best by 31% relative (1.29% vs 0.89%).The minDCF value of 0.0593 is not comparable because Ptarget differs.Source: §5.4 · p. 4
• RawNet3 trained with DINO is presented as an initial model for iterative clustering and as complementary to existing iterative-clustering approaches.The paper notes that iterative clustering has been shown effective for self-supervised speaker verification.Source: §5.4 · p. 4

• 0.89% EER makes supervised RawNet3 competitive with state-of-the-art handcrafted feature-based counterparts.Source: §6 · p. 4
• 5.40% EER with DINO self-supervision outperforms existing contrastive learning-based works.Source: §6 · p. 4
• DINO pre-training followed by fine-tuning on a smaller dataset yields 25% relative improvement over random initialization.Source: §6 · p. 4