PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxCeleb: a large-scale speaker identification dataset

Arsha Nagrani, Joon Son Chung, Andrew Zisserman

arXiv:1706.08612v2cs.SD

TL;DR

Existing speaker-identification datasets are often constrained and hand-annotated, limiting their scale. This paper builds VoxCeleb with an automated visual pipeline and finds that CNNs outperform traditional state-of-the-art methods for both identification and verification.

  • Problem

    Existing wild-condition speaker datasets are hand-annotated, and scaling them to thousands of speakers and tens of thousands of utterances would require annotation services.

    Source: §1 · p. 1

  • Method

    The authors use a fully automated, scalable pipeline based on visual active-speaker identification and face verification to curate VoxCeleb and compare CNNs with traditional methods.

    Source: §1 · p. 1

  • Results

    CNNs outperform traditional state-of-the-art baselines for both identification and verification; identification achieves 80.5% top-1 classification accuracy over 1,251 classes.

    Source: §7 · p. 4, §6.3 · p. 4

  • Takeaways & Limitations

    VoxCeleb provides a dataset and baselines for both speaker identification and verification.

    Source: §1 · p. 1

Abstract

from arXiv · show

Most existing datasets for speaker identification contain samples obtained under quite constrained conditions, and are usually hand-annotated, hence limited in size. The goal of this paper is to generate a large scale text-independent speaker identification dataset collected 'in the wild'. We make two contributions. First, we propose a fully automated pipeline based on computer vision techniques to create the dataset from open-source media. Our pipeline involves obtaining videos from YouTube; performing active speaker verification using a two-stream synchronization Convolutional Neural Network (CNN), and confirming the identity of the speaker using CNN based facial recognition. We use this pipeline to curate VoxCeleb which contains hundreds of thousands of 'real world' utterances for over 1,000 celebrities. Our second contribution is to apply and compare various state of the art speaker identification techniques on our dataset to establish baseline performance. We show that a CNN based architecture obtains the best performance for both identification and verification.

1. Introduction

The paper addresses the shortage of scalable unconstrained speaker-identification data by introducing an automated VoxCeleb curation pipeline and evaluating deep CNN methods against traditional state-of-the-art methods.

1. Introduction

  • Real-world speaker recognition is difficult because noise and channel conditions interact with speaker variation in age, accent, emotion, intonation, and speaking manner.Source: §1 · p. 1
  • Deep CNNs have improved across speech recognition and vision, but their success depends importantly on access to large training datasets [4].Source: §1 · p. 1
  • Public unconstrained speaker-identification datasets are scarce; SITW has 299 speakers and was hand-annotated, making larger-scale expansion labor-intensive.The passage notes that scaling to thousands of speakers and tens of thousands of utterances would require a service such as Amazon Mechanical Turk.Source: §1 · p. 1
  • VoxCeleb is curated with a fully automated, scalable pipeline using visual active speaker identification and face verification, avoiding human annotation.The dataset contains hundreds of utterances for over a thousand speakers.Source: §1 · p. 1
  • The paper compares deep CNN architectures trained on raw-audio spectrograms with little preprocessing against traditional state-of-the-art methods and provides baselines for identification and verification.Identification selects a speaker from a closed set, while verification tests whether an utterance matches a target model.Source: §1 · p. 1

2. Related Works

Speaker identification has moved from low-dimensional handcrafted features toward deep CNNs, but CNN-based methods need large datasets collected under real-world conditions. Existing datasets are often controlled or manually curated, leaving that need unmet.

2. Related Works

  • CNNs offer an alternative to GMM, JFA, and i-vector methods that rely on low-dimensional MFCC features, whose noise sensitivity and limited speaker-discriminating information motivate higher-dimensional learned inputs [7, 8, 9, 10, 11, 12, 13, 14, 15].The passage identifies pitch as an example of speaker-discriminating information that MFCCs may lack.Source: §2 · p. 1
  • Many datasets use controlled sources such as forensic, telephone, laboratory, or mobile speech; even more natural speech may be manually cleaned, and these datasets lack audience noise and overlapping speech.Source: §2 · p. 2
  • Meeting recordings and audio broadcasts provide multi-speaker data under less controlled conditions, while NTIMIT and CTIMIT artificially degrade TIMIT recordings to mimic real-world noise.Source: §2 · p. 2
  • Table 1 compares existing datasets, which, aside from broadcast data mapped from subtitles and transcripts, the authors say were mostly collected with substantial manual effort.The passage also states that these datasets lack real-world conditions.Source: §2 · p. 2, Table 1 · p. 2

3. Dataset Description

VoxCeleb contains over 100,000 YouTube-extracted utterances from 1,251 celebrities, spanning varied speaker backgrounds and challenging real-world recording conditions. Both audio and video are released for each speaker.

3. Dataset Description

  • VoxCeleb contains over 100,000 YouTube-extracted utterances from 1,251 celebrities.Speakers are 55% male, and the dataset provides each speaker’s nationality and gender.Source: §3 · p. 2
  • Speakers span a wide range of ethnicities, accents, professions, and ages, with 55% male representation.Nationality and gender, obtained from Wikipedia, are also provided for each speaker.Source: §3 · p. 2
  • Videos cover varied multi-speaker settings and real-world noise, including background chatter, laughter, overlapping speech, room acoustics, and channel noise.Recording conditions range from quiet studio interviews and speeches to outdoor stadiums and handheld-device videos.Source: §3 · p. 2
  • Unlike SITW, VoxCeleb releases both audio and video for each speaker.Source: §3 · p. 2

4. Dataset Collection Pipeline

The paper presents a fully automated, multi-stage pipeline that collects speaker utterances from YouTube and verifies who is speaking. It uses conservative thresholds to reduce false positives, yielding hundreds of utterances for over a thousand people.

Overview

  • The automated pipeline collects hundreds of utterances for over a thousand people and is designed to scale to any number of speakers and available utterances.Figure 1 left summarizes the processing pipeline.Source: §4 · p. 2, Figure 1 · p. 3

Stage 1. Candidate list of POIs

  • The candidate list draws 2,622 identities from the VGG Face dataset, based on frequently searched names in Freebase and IMDB.The list spans actors, sportspeople, entrepreneurs, and other people, with approximately equal numbers of men and women.Source: §4 · p. 2

Stages 1–2. Candidate list and video downloading

  • For each of 2,622 people of interest, the pipeline automatically downloads the top 50 YouTube search results, appending “interview” to queries without other filtering.The candidate list is based on VGG Face identities, drawn from the intersection of frequently searched Freebase names and IMDB.Source: §4 · p. 2

Stage 3. Face tracking

  • Face tracking detects faces and landmarks, finds shot boundaries, then links detections into tracks with a position-based tracker optimized for runtime.Shot boundaries are found by comparing color histograms across consecutive frames.Source: §4 · p. 2

Stage 4. Active speaker verification

  • Sync-Net, a two-stream CNN, checks synchronization between speech and mouth motion to identify visible speakers and reject dubbed or voice-over clips.The method estimates the correlation between the audio track and video mouth motion.Source: §4 · p. 2

Stage 5. Face verification

  • The VGG Face CNN classifies active-speaker tracks as the person of interest or not, using its classification score with a high threshold.The network is based on VGG-16 and trained on the VGG Face dataset.Source: §4 · p. 2

Discussion

  • Conservative thresholds for active-speaker and face verification prioritize confidence and few labeling errors, even though many downloaded videos are discarded.Figure 1 right shows benchmark precision-recall curves and operating points; Table 3 gives precision-recall values at the chosen operating points.Source: §4 · p. 2, Figure 1 · p. 3, Table 3 · p. 3

5. CNN Design and Architecture

The section adapts VGG-M for speaker recognition using minimally preprocessed spectrograms and temporal average pooling. It describes separate identification and verification outputs, variable-length testing, and training procedures.

Architecture

  • Replacing VGG-M’s fc6 with a frequency-domain fully connected layer and temporal average pooling makes the network temporally invariant and reduces parameters from 319M to 67M.The pooling support is 1 × n, with n determined by input duration; the output dimensions remain unchanged.Source: §5 · p. 3

Input features

  • Almost 10% higher classification accuracy follows mean and variance normalization of each spectrogram frequency bin.Audio is converted to single-channel, 16-bit, 16kHz streams; 25ms Hamming windows with 10ms steps produce 512 x 300 spectrograms for 3-second segments, without other speech-specific preprocessing.Source: §5 · p. 3

Identification and verification

  • Identification uses a 1,251-way softmax, while verification learns a 1024-dimensional embedding with a Siamese network and contrastive loss.The section also compares verification using 1024-dimensional fc7 vectors and cosine distance.Source: §5 · p. 3

Testing

  • At test time, average pooling accommodates variable-length utterances in one pass and increases classification accuracy over fixed-length segmentation.The apool6 layer’s support is changed to match the test utterance length.Source: §5 · p. 3, Table 4 · p. 3

Implementation details and training

  • Training uses random 3-second crops; verification fine-tuning freezes all but the modified last layer and mines half of negative pairs from the hardest 10%.The network is first trained for classification, excluding test POIs; it is then trained as a Siamese network with contrastive loss. The other half of negative examples is selected randomly.Source: §5 · pp. 3–4

6. Experiments

The experiments compare the CNN baseline with traditional methods on VoxCeleb for speaker identification and verification. The CNN outperforms the baselines on both tasks.

6.1. Experimental setup

  • Identification testing uses a held-out video per POI with at least five non-overlapping speech segments, while verification tests POIs whose names start with E.Identification reports top-1 and top-5 accuracy; verification uses Cmin and EER, with Cmin evaluated across thresholds using a 0.01 target prior and equal miss and false-alarm weights.Source: §6.1 · p. 4

6.2. Baselines

  • The GMM-UBM baseline uses 13-dimensional MFCCs and a 1,024-component UBM, while the i-vector/PLDA baseline uses 400-dimensional i-vectors reduced to 200 dimensions.Identification uses one-vs-rest SVM classifiers with L2-normalized inputs; verification uses PLDA scoring.Source: §6.2 · p. 4

6.3. Results

  • The average-pooling CNN outperforms the CNN-fc-3s variant, and the comparison with its no-variance-normalization variant shows variance normalization is important.CNN-fc-3s divides test data into 3-second segments and averages scores; the no-variance-normalization variant remains mean-normalized.Source: §6.3 · p. 4
  • 80.5% top-1 accuracy across 1,251 classes—almost 20% above traditional baselines—while CNNs also outperform verification baselines, though by a narrower margin.The identification CNN uses average pooling for variable-length test data; the embedding is crucial to verification performance.Source: Table 7 · p. 4, §6.3 · p. 4, Table 8 · p. 4

7. Conclusions

The paper introduces an automated, scalable pipeline that creates VoxCeleb, a large-scale speaker identification dataset, and a variable-length CNN that outperforms traditional state-of-the-art methods on the dataset.

7. Conclusions

  • The variable-length CNN outperforms traditional state-of-the-art methods for both speaker identification and verification on VoxCeleb.The architecture is designed to handle variable-length audio inputs.Source: §7 · p. 4
  • A fully automated, scalable audio-collection pipeline creates the VoxCeleb speaker identification dataset.Source: §7 · p. 4
  • VoxCeleb contains 1,251 speakers and over 100,000 utterances.Source: §7 · p. 4

Related papers

Loading 1706.08612v2…