Source-linked AI summary
VoxCeleb2: Deep Speaker Recognition
Joon Son Chung, Arsha Nagrani, Andrew Zisserman
TL;DR
Speaker recognition remains challenging under noisy, unconstrained conditions because compact speaker representations are difficult to produce. The paper introduces the large-scale VoxCeleb2 dataset and CNN-based VGGVox embeddings with associated training strategies, reporting improved performance on VoxCeleb1 and a broader evaluation protocol.
Problem
Speaker recognition lacks compact representations that work efficiently under noisy, unconstrained conditions and lacks large-scale freely available in-the-wild datasets.
Method
The paper automatically curates VoxCeleb2 from open-source media and trains VGGVox CNN systems to learn speaker-discriminative embeddings from voice spectrograms.
Results
The models achieve state-of-the-art speaker verification performance on VoxCeleb1 and are evaluated with a new test set covering 1,251 speakers.
Takeaways & Limitations
Compact 512D identity embeddings are useful for speaker verification and other tasks such as diarisation and retrieval.
Takeaways & Limitations
The VoxCeleb2 development set is used for speaker recognition while its test set is reserved for other applications, and automated collection uses less strict thresholds because it is training-only.
Abstract
from arXiv · showhide
The objective of this paper is speaker recognition under noisy and unconstrained conditions. We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset. Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.
1. Introduction
The paper addresses speaker recognition under noisy, unconstrained conditions by releasing VoxCeleb2 and developing CNN-based speaker embeddings and training strategies.
- Model: VGGVox maps voice spectrograms into a compact Euclidean space where distances measure speaker similarity.The resulting embeddings can support verification, clustering, and diarisation using standard techniques.
- Dataset: VoxCeleb2 contains over a million utterances from over 6,000 speakers, collected automatically from open-source media with real-world noise.The dataset is several times larger than existing speaker-recognition datasets and includes audio-visual data.
- Model: The system combines a CNN trunk, a pooling method for utterance-level aggregation, and a pairwise loss, using VGG-M and ResNet architectures.These components are varied to learn speaker-discriminative embeddings.
2. Related works
Existing speaker-recognition datasets are limited by size, collection conditions, annotation, or availability, whereas VoxCeleb2 expands scale and diversity through automated collection from online video.
- Prior methods: Traditional speaker-recognition methods rely on hand-crafted features, while deep learning methods use neural networks as feature extractors combined with classifiers.The related-work discussion contrasts i-vector-based systems with deep learning approaches.
- Datasets: Unlike earlier datasets, VoxCeleb2 is collected automatically from open-source media and is designed to avoid limitations involving controlled conditions, manual annotation, and restricted availability.The collection pipeline uses automated face detection, tracking, active-speaker verification, and duplicate removal.
- Datasets: VoxCeleb2 contains over 1 million utterances from over 6,000 celebrities across varied ethnicities, accents, professions, ages, and recording environments.The media includes red-carpet interviews, stadiums, studios, speeches, professional footage, and handheld videos.
- Dataset split: VoxCeleb2’s development set has no identity overlap with VoxCeleb1 or SITW, while its test set is reserved for other audio-visual learning applications.The speaker-recognition experiments use only the development set for training and validation.
- Dataset overview: Figure 1 presents speaker-face examples alongside utterance-length, gender, and nationality distributions, showing the dataset’s audio-visual and demographic structure.Only the top five nationality frequencies are displayed for readability.
4. VGGVox
VGGVox learns utterance-level speaker embeddings from raw-audio spectrograms using CNN feature extraction, pooling, and staged metric-learning strategies, then evaluates them with verification metrics.
- System: VGGVox extracts frame-level features from short-term magnitude spectrograms, pools them into utterance-level embeddings, and trains the model with contrastive loss.The spectrograms are computed directly from raw audio segments without other preprocessing.
- Evaluation: The evaluation reports Equal Error Rate and a cost function with target prior probability Ptar = 0.01 and equal miss and false-alarm weights.Testing uses VoxCeleb1 pairs after training on VoxCeleb2.
- Trunk architectures: The VGG-M trunk modifies the image CNN with frequency-supported fully connected and average-pooling layers, producing temporal-position invariance and fivefold fewer parameters.The architecture remains frequency-sensitive, which is desirable for speech.
- Training: The training strategy first pre-trains speaker identification with softmax cross-entropy, then fine-tunes paired embeddings using contrastive loss.Softmax pre-training provides more stable convergence before pairwise optimisation.
- Training: Hard negative mining selects difficult negative pairs because the number of possible contrastive pairs grows quadratically as the dataset expands.The offline strategy can select harder negatives than online in-batch mining.
- Testing: At test time, the system compares variable average pooling with temporal-crop augmentation using either averaged features or averaged pairwise crop distances.The crop-based methods sample ten 3-second crops, yielding 100 cross-segment crop comparisons in the pairwise variant.
5. Results
The results evaluate VGGVox on the original VoxCeleb1 test set and introduce extended test sets covering broader speaker populations and matched demographic conditions. Performance improves with deeper networks and more training data.
- The original VoxCeleb1 test set evaluates verification performance, with lower values indicating better results.Table 4 reports model performance on this benchmark.
- Performance improves with greater network depth and with more training data from VoxCeleb2 rather than VoxCeleb1.
- Extended test sets: The extended VoxCeleb1-E protocol samples 581,480 random pairs covering 1,251 speakers from the entire VoxCeleb1 dataset.It is designed to reduce dependence on the original test set's limited speaker pool.
- Extended test sets: The extended VoxCeleb1-H protocol contains 552,536 pairs sharing nationality and gender across 18 nationality-gender combinations.
6. Conclusion
The paper presents new speaker-verification architectures and training strategies, a large-scale VoxCeleb2 dataset, and broader VoxCeleb1 evaluation protocols. Its 512D identity embeddings are compact and intended for verification and related tasks.
- The paper introduces new architectures and training strategies and reports state-of-the-art speaker-verification performance on VoxCeleb1.
- VoxCeleb2 is several times larger than any speaker-recognition dataset described by the authors.
- The 512D identity embeddings are compact and useful for tasks such as diarisation and retrieval.
- The re-purposed VoxCeleb1 test set covers 1,251 speakers, enabling broader assessment than the original 40-speaker test set.