Source-linked AI summary
VoxCeleb: a large-scale speaker identification dataset
Arsha Nagrani, Joon Son Chung, Andrew Zisserman
TL;DR
Speaker-identification research lacked large public datasets containing unconstrained speech, limiting evaluation under real-world conditions. The paper introduces an automated visual pipeline to curate VoxCeleb from YouTube and compares CNN-based and traditional methods. The resulting CNN architecture outperforms traditional state-of-the-art baselines for both identification and verification.
Problem
Large-scale public speaker-identification datasets with unconstrained speech are lacking, while existing datasets often require manual annotation.
Method
The paper uses a fully automated, scalable pipeline based on visual active speaker identification and face verification to curate VoxCeleb, then compares CNN architectures with traditional methods.
Results
The CNN architecture outperforms traditional state-of-the-art methods for both speaker identification and verification on VoxCeleb.
Takeaways & Limitations
VoxCeleb provides a large-scale dataset and benchmark baselines for speaker identification and verification under real-world conditions.
Abstract
from arXiv · showhide
Most existing datasets for speaker identification contain samples obtained under quite constrained conditions, and are usually hand-annotated, hence limited in size. The goal of this paper is to generate a large scale text-independent speaker identification dataset collected 'in the wild'. We make two contributions. First, we propose a fully automated pipeline based on computer vision techniques to create the dataset from open-source media. Our pipeline involves obtaining videos from YouTube; performing active speaker verification using a two-stream synchronization Convolutional Neural Network (CNN), and confirming the identity of the speaker using CNN based facial recognition. We use this pipeline to curate VoxCeleb which contains hundreds of thousands of 'real world' utterances for over 1,000 celebrities. Our second contribution is to apply and compare various state of the art speaker identification techniques on our dataset to establish baseline performance. We show that a CNN based architecture obtains the best performance for both identification and verification.
1. Introduction
Speaker identification in real-world conditions is difficult, while large public datasets with unconstrained speech are scarce. The paper addresses this gap with an automated, scalable pipeline and provides VoxCeleb for identification and verification.
- Real-world speaker recognition is challenged by background noise, channel effects, and speaker variations such as age, accent, and emotion.
- Deep CNNs can handle noisy real-world datasets without handcrafted features, but their success depends on large training datasets.
- Large-scale public datasets with unconstrained speech are lacking, and existing alternatives are limited by manual annotation or restricted access.
- The paper proposes a fully automated, scalable pipeline using visual active speaker identification and face verification to create VoxCeleb.
- VoxCeleb supports both speaker identification and verification, with baselines provided for each task.
2. Related Works
Earlier speaker-identification methods relied on low-dimensional handcrafted features, while many datasets were controlled, artificially degraded, or manually processed. Existing datasets therefore provided limited support for large-scale evaluation under real-world conditions.
- Traditional speaker-identification systems used GMMs, JFA, or i-vectors built from low-dimensional audio representations such as MFCCs.
- MFCC performance degrades rapidly in real-world noise and may omit speaker-discriminating information such as pitch.
- Many datasets were collected in controlled, single-speaker settings and excluded audience noise and overlapping speech.
- Other datasets used multi-speaker recordings or artificial degradation to approximate real-world noise.
- Existing datasets generally lacked real-world conditions and required substantial manual collection effort, except for one subtitle- and transcript-mapped broadcast dataset.
3. Dataset Description
VoxCeleb contains over 100,000 utterances from 1,251 celebrities and includes varied demographics and metadata. Its videos capture challenging, noisy multi-speaker environments across diverse recording settings.
- VoxCeleb contains over 100,000 utterances from 1,251 celebrities extracted from YouTube videos.
- The dataset is gender balanced, with 55% male speakers spanning diverse ethnicities, accents, professions, and ages.
- Nationality and gender metadata for each speaker are provided from Wikipedia.
- Videos span red carpets, stadiums, studios, speeches, professionally produced media, and handheld-device recordings.
- The recordings contain background chatter, laughter, overlapping speech, room acoustics, equipment variation, and channel noise.
4. Dataset Collection Pipeline
The pipeline begins with YouTube-based collection and uses automated processing to build a large speaker dataset. Figure 1 and Table 3 summarize the pipeline and verification operating points.
- Pipeline overview: The collection pipeline starts from YouTube videos and processes them through multiple automated stages.The supplied passage introduces a multi-stage approach beginning with YouTube videos; Figure 1 summarizes the pipeline.
- Dataset output: The resulting VoxCeleb collection contains hundreds of utterances for over a thousand persons of interest.The dataset statistics are presented in Table 2, whose entries may report maximum, average, and minimum values.
- Pipeline stages: The first stages generate candidate identities, download interview-focused videos, and detect and track faces.The candidate list contains 2,622 identities, while video searches append “interview” and use HOG-based face detection with facial landmarks.
- Verification evaluation: Precision-recall values are reported at selected operating points for the verification stages.Figure 1 shows operating points for active speaker verification and face verification, while Table 3 reports their precision-recall values.
5. CNN Design and Architecture
The CNN processes standardized spectrogram inputs and supports variable-length audio. Classification provides speaker distributions, while a Siamese embedding is used for verification.
- Input features: Audio is standardized to 16 kHz single-channel 16-bit streams and converted into 512 x 300 spectrograms from 3-second windows.Spectrograms use a 25 ms Hamming window and 10 ms step.
- Identification and verification: The classification network produces a distribution over 1,251 speakers, whereas verification compares learned embeddings with a Siamese contrastive-loss network.The verification embedding is optimized for similarity directly rather than indirectly through classification loss.
- Architecture: The CNN architecture accepts variable-length inputs, although the reported intermediate data sizes up to fc6 use a 3-second input.Average pooling is used for variable-length test data, and Table 4 documents the architecture dimensions.
- Training: Training uses batch normalization, random 3-second temporal crops, and hard-negative mining for verification pairs.Half of the negative examples are random and half are drawn from the hardest 10% of negatives.
6. Experiments
The experiments evaluate identification and verification on VoxCeleb against traditional state-of-the-art baselines. The CNN performs best on both tasks, with especially strong identification accuracy and a narrower verification margin.
- Experimental setup: The experiments compare a CNN baseline with traditional state-of-the-art systems for speaker identification and verification on VoxCeleb.Identification reports top-1 and top-5 accuracy, while verification uses Cmin_det and equal error rate.
- Identification: 80.5% top-1 classification accuracy is achieved across 1,251 identification classes, almost 20% higher than traditional state-of-the-art baselines.The identification CNN uses average pooling for variable-length test data.
- Identification: Variance normalization improves identification performance, as removing it from the CNN-fc-3s variant produces a performance difference.The comparison is between CNN-fc-3s with and without variance-normalization preprocessing.
- Verification: The CNN also outperforms traditional state-of-the-art baselines for verification, although its margin is narrower than for identification.The learned embedding is identified as the crucial step for verification.
7. Conclusions
The paper introduces an automated, scalable pipeline that creates VoxCeleb with 1,251 speakers and over 100,000 utterances. Its CNN architecture outperforms traditional state-of-the-art methods for both identification and verification.
- A fully automated and scalable audio-data pipeline creates VoxCeleb with 1,251 speakers and over 100,000 utterances.
- The developed CNN architecture handles variable-length audio inputs.
- The CNN outperforms traditional state-of-the-art methods for both speaker identification and verification on VoxCeleb.