Source-linked AI summary
Utterance-level Aggregation For Speaker Recognition In The Wild
Weidi Xie, Arsha Nagrani, Joon Son Chung, Andrew Zisserman
TL;DR
Speaker recognition in the wild must produce compact utterance representations from variable-length speech containing irrelevant signals. The paper combines a thin-ResNet trunk with NetVLAD or GhostVLAD temporal aggregation and reports state-of-the-art VoxCeleb1 performance with fewer parameters, while finding longer utterances beneficial.
Problem
Speaker recognition in the wild needs fixed-length utterance representations that remain discriminative despite variable duration and irrelevant signal content.
Method
A thin-ResNet extracts frame descriptors, which NetVLAD or GhostVLAD aggregates into a compact fixed-length representation through trainable dictionary-based clustering.
Results
The network achieves state-of-the-art VoxCeleb1 performance with fewer parameters than previous methods, including 3.22% EER using GhostVLAD.
Takeaways & Limitations
For in-the-wild speaker recognition, longer utterances are beneficial, with sequences of 4s or more significantly improving performance over shorter segments.
Takeaways & Limitations
The model design targets three stated requirements: arbitrary input duration, compact descriptors, and discriminative speaker representations.
Abstract
from arXiv · showhide
The objective of this paper is speaker recognition "in the wild"-where utterances may be of variable length and also contain irrelevant signals. Crucial elements in the design of deep networks for this task are the type of trunk (frame level) network, and the method of temporal aggregation. We propose a powerful speaker recognition deep network, using a "thin-ResNet" trunk architecture, and a dictionary-based NetVLAD or GhostVLAD layer to aggregate features across time, that can be trained end-to-end. We show that our network achieves state of the art performance by a significant margin on the VoxCeleb1 test set for speaker recognition, whilst requiring fewer parameters than previous methods. We also investigate the effect of utterance length on performance, and conclude that for "in the wild" data, a longer length is beneficial.
1. INTRODUCTION
Speaker recognition in the wild requires fixed utterance-level representations from variable-length, noisy signals, making effective temporal aggregation central. The paper addresses content-independent pooling by combining CNN frame features with dictionary-based aggregation.
- Motivation: Speaker recognition must condense variable-length speech into a single utterance-level representation while filtering irrelevant signal content.This differs from speech recognition, which seeks frame-level representations.
- Limitations of prior aggregation: Average pooling and fully connected aggregation allow back-propagation but weight all content equally, ignoring which signal segments are relevant.The resulting aggregation is not content dependent.
- Proposed direction: The proposed approach combines CNNs for local pattern capture with dictionary-based NetVLAD aggregation for fixed-sized descriptors from arbitrary-length inputs.The design aims to make the utterance representation unaffected by irrelevant information.
- Contributions: The network uses a thin-ResNet trunk with NetVLAD or GhostVLAD aggregation and is trained end-to-end on VoxCeleb2 with a large margin softmax loss.It is evaluated for verification on VoxCeleb1.
- System pipeline: End-to-end speaker recognition systems commonly combine frame-level feature extraction, temporal aggregation, and classification-loss optimization.Verification may additionally use contrastive or metric-learning losses and similarity scoring.
- Related aggregation methods: Temporal aggregation methods in related systems include average, statistical, attentive, attentive-statistics, and learnable dictionary encoding pooling.These methods differ in whether frames receive equal or learned weighting.
2. METHODS
The method uses a thin, fully convolutional ResNet to extract frame descriptors, then applies NetVLAD or GhostVLAD to aggregate them into compact fixed-length utterance representations.
- Design objectives: The model accepts arbitrary time lengths and produces compact, discriminative fixed-length utterance descriptors.These properties support efficient storage, retrieval, and speaker comparison.
- Architecture: A modified ResNet encodes spectrograms, while NetVLAD or GhostVLAD aggregates frame-level features along the temporal axis.The aggregation produces a fixed-length output descriptor.
- Architecture: The architecture separates shared-CNN spectrogram feature extraction from temporal aggregation of local descriptors into a compact representation.The representation can accommodate arbitrary input length.
- Feature extraction: The thin-ResNet reduces the standard ResNet-34 from 22 million to 3 million parameters for frame-level feature extraction.The table describes convolutional filters and their sizes across the architecture.
- Feature extraction: The modified 34-layer ResNet cuts channels in each residual block to form a thin ResNet-34.It is used as the frame-level feature extractor.
- Temporal aggregation: NetVLAD aggregates frame-level descriptors into a single utterance-level vector, with GhostVLAD providing an additional aggregation variant.NetVLAD is introduced as the second network component.
- Temporal aggregation: The thin ResNet maps a 257×T spectrogram to 512-dimensional descriptors at T/32 time steps, and NetVLAD outputs a K × D matrix.K is the number of clusters and D is the dimensionality of each cluster.
- Temporal aggregation: NetVLAD softly assigns each descriptor to clusters, encodes residuals to cluster centers, then applies L2 normalization, concatenation, and 512-dimensional reduction.The assignment and cluster-center parameters are trainable.
3. EXPERIMENTS
The experiments train the model on disjoint VoxCeleb2 development speakers and evaluate verification on VoxCeleb1, using softmax and additive-margin softmax training losses. Training uses randomized 2.5-second spectrogram crops.
- Evaluation setup: The model is trained on VoxCeleb2 development speech from 5,994 speakers and tested on disjoint VoxCeleb1 verification sets.The development and test datasets contain no speakers in common.
- Training loss: Additive margin softmax is evaluated alongside standard softmax because its angular margin is reported to improve verification performance.The loss introduces a margin in angular space.
- Training loss: The angular-margin loss minimizes the angle between sample features and the correct-class decision hyperplane, using m = 0.4 and s = 30.Both feature and decision vectors are L2-normalized.
- Training procedure: Training uses randomly extracted fixed-size spectrograms corresponding to 2.5-second temporal segments.The STFT uses a 25 ms Hamming window, 10 ms step, and 257 × 250 spectrogram size.
4. RESULTS
The experiments show that dictionary-based VLAD aggregation substantially improves VoxCeleb1 verification while remaining parameter-efficient, and that longer utterances benefit in-the-wild recognition.
- Verification on VoxCeleb1: EER 3.57% vs 4.19% shows NetVLAD with standard softmax outperforms the original ResNet architecture using 10 vs 26 million parameters.AM-Softmax and GhostVLAD further reduce EER to 3.32% and 3.22%, respectively.
- Verification on VoxCeleb1: EER 5.17% vs 7.33% shows the model outperforms the original ResNet architecture on the challenging VoxCeleb1-H test set.The reported margin is larger than on the original VoxCeleb1 test set.
- Verification on VoxCeleb1: EER 3.22% vs 4.48% shows the proposed model outperforms the similar dictionary-based method of Cai et al.
- Additional experiment on GhostVLAD: VLAD aggregation is robust to the number of clusters and to two different loss functions, despite small performance differences.Table 3 varies VLAD cluster counts and loss functions while using the same Thin ResNet-34 architecture.
- Probing verification based on length: Verification performance has a strong correlation with sequence length, with longer test segments increasing the chance of capturing relevant speaker speech.The length experiment evaluates randomly cropped 2s, 3s, 4s, 5s, and 6s segments from VoxCeleb1 verification pairs.
5. CONCLUSION
The paper proposes an end-to-end speaker recognition network combining a thin-ResNet trunk with dictionary-based temporal aggregation. It achieves state-of-the-art VoxCeleb1 performance with fewer parameters and finds longer utterances beneficial for in-the-wild data.
- The proposed network combines a thin-ResNet trunk with NetVLAD or GhostVLAD layers for end-to-end temporal feature aggregation.
- The network achieves state-of-the-art speaker recognition performance on VoxCeleb1 while requiring fewer parameters than previous methods.
- Longer utterances are beneficial for speaker recognition on in-the-wild data.