Source-linked AI summary
TidyVoice: A Curated Multilingual Dataset for Speaker Verification Derived from Common Voice
Aref Farhadipour, Jan Marquenie, Srikanth Madikeri, Eleanor Chodroff
TL;DR
Speaker recognition lacks large, multilingual, publicly available read-speech datasets, limiting resources for applications such as anti-spoofing. The paper introduces curated TidyVoice partitions from Common Voice and fine-tunes ResNet models on them. Fine-tuning on Tidy-M improves performance across TidyVoice conditions and generalizes to unseen conversational CANDOR data.
Problem
Speaker recognition lacks large, multilingual, publicly available datasets that represent read speech needed for applications such as anti-spoofing.
Method
The authors curate TidyVoice from Common Voice, mitigate speaker heterogeneity, define Tidy-M and Tidy-X conditions, and fine-tune ResNet models on the partitions.
Results
Fine-tuning on Tidy-M improved performance on TidyVoice intra-lingual and mixed-language conditions and outperformed the pretrained baseline on out-of-domain conversational CANDOR data.
Takeaways & Limitations
TidyVoice provides a publicly released resource for multilingual speaker verification, with fine-tuning supporting generalization from read speech to conversational speech.
Takeaways & Limitations
Fine-tuning on the smaller Tidy-X partition produced exceptions, with higher-capacity ResNet-293 more prone to overfitting on the limited speaker set.
Abstract
from arXiv · showhide
The development of robust, multilingual speaker recognition systems is hindered by a lack of large-scale, publicly available and multilingual datasets, particularly for the read-speech style crucial for applications like anti-spoofing. To address this gap, we introduce the TidyVoice dataset derived from the Mozilla Common Voice corpus after mitigating its inherent speaker heterogeneity within the provided client IDs. TidyVoice currently contains training and test data from over 212,000 monolingual speakers (Tidy-M) and around 4,500 multilingual speakers (Tidy-X) from which we derive two distinct conditions. The Tidy-M condition contains target and non-target trials from monolingual speakers across 81 languages. The Tidy-X condition contains target and non-target trials from multilingual speakers in both same- and cross-language trials. We employ two architectures of ResNet models, achieving a 0.35% EER by fine-tuning on our comprehensive Tidy-M partition. Moreover, we show that this fine-tuning enhances the model's generalization, improving performance on unseen conversational interview data from the CANDOR corpus. The complete dataset, evaluation trials, and our models are publicly released to provide a new resource for the community.
1. INTRODUCTION
Speaker recognition needs large, multilingual, publicly accessible read-speech data, but existing resources are often monolingual, English-biased, domain-specific, or licensed. TidyVoice addresses these gaps with curated Common Voice data, distinct monolingual and multilingual conditions, and released models and trials.
- Current speaker-recognition datasets are often predominantly monolingual and heavily biased toward English.
- Read-speech data is particularly important for applications such as anti-spoofing, which evaluate text-prompted utterances.
- Existing multilingual resources include large in-the-wild benchmarks and licensed NIST SRE corpora, but they differ in domain and accessibility.
- TidyVoice contains over 212,000 monolingual speakers across 81 languages and around 4,500 multilingual speakers across 40 unique languages.
- Tidy-M provides matched-language target and non-target trials across 81 languages, while Tidy-X provides intra- and cross-lingual trials across 40 languages.
- The authors evaluate ResNet-34 and ResNet-293 models and publicly release datasets, splits, trial pairs, and trained models.
2. DATASETS AND CONDITIONS
TidyVoice is built from Common Voice read speech after addressing unreliable client IDs that can combine recordings from multiple speakers. It defines monolingual and multilingual partitions with separate language-matched and cross-language verification conditions.
- Common Voice is a publicly available, massively multilingual crowdsourced collection of read speech, but its client ID is not always a reliable speaker identifier.
- Speaker heterogeneity occurs when multiple individuals contribute recordings under a single Common Voice client ID.
- A verification pipeline uses cosine similarity within each language and excludes files scoring below 0.4 against a designated enrollment file.
- Cross-lingual screening removed IDs with many cosine-similarity scores below 0.2, identifying 433 of 4,907 multilingual speakers as likely identity switches.
- Tidy-M: Tidy-M covers 81 languages, assigns speakers with more than four utterances to training, and requires at least five test speakers per language.
- Tidy-X: Tidy-X covers multilingual speakers from 40 languages, with target trials pairing the same speaker within and across languages.
3. EXPERIMENTAL SETUP
The experiments train and fine-tune ResNet speaker-embedding models with standardized acoustic features and additive angular margin loss. Evaluation covers established VoxCeleb benchmarks, proposed TidyVoice trials, and conversational out-of-domain CANDOR data.
- Experiments use the WeSpeaker open-source toolkit for acoustic features, model architectures, training, and evaluation.
- Inputs are 80-dimensional log Mel-filterbank energies extracted with 25 ms frames, 10 ms hops, and 200-frame training segments.
- ResNet-34 and ResNet-293 extract utterance-level speaker embeddings using residual blocks and attentive statistics pooling.
- Models include ResNet-34 trained from scratch on Tidy-M and VoxCeleb2/VoxBlink2-pretrained models fine-tuned on Tidy-M or Tidy-X.
- Fine-tuning uses additive angular margin loss and adapts pretrained models to TidyVoice's speaker diversity and linguistic variation.
- Evaluation reports EER and minDCF from cosine scores between 256-dimensional utterance embeddings.
4. RESULTS AND DISCUSSION
Fine-tuning on TidyVoice generally improved speaker-verification performance, including on unseen conversational CANDOR data, though gains varied by training condition and model size. Tidy-M fine-tuning showed especially strong language-level improvements, while Tidy-X exposed cross-language difficulty and possible overfitting with ResNet-293.
- Generalization: Fine-tuning on Tidy-M improved performance on unseen conversational CANDOR data, indicating generalization beyond the read-speech training domain.The improvement was observed for conversational and spontaneous speech in CANDOR.
- Model comparison: ResNet-293 consistently outperformed ResNet-34 in almost all evaluated cases.
- Tidy-M results: Odia EER fell from 29.51% to 0.87%, a relative improvement of 97%, after fine-tuning.
- Tidy-M results: Czech EER fell from 26.67% to 1.57%, while Kabyle EER fell from 15.43% to 0.37%.
- Tidy-X results: Tidy-X was hardest for target-speaker different-language versus non-target same-language trials, and easiest for target-speaker same-language versus non-target different-language trials.
- Limitations: Fine-tuning on the smaller Tidy-X partition sometimes led ResNet-293 to overfit, possibly shifting learning toward language-effect cancellation.
5. CONCLUSION
Fine-tuning on Tidy-M improves performance on both TidyVoice evaluation conditions and enhances generalization to conversational CANDOR data. The dataset, evaluation trials, and models are publicly released, with scope for future expansion.
- Fine-tuning on Tidy-M improves performance over the baseline on both Tidy-M intra-lingual and Tidy-X mixed-language conditions.
- Fine-tuning on read speech improves performance on out-of-domain conversational CANDOR data relative to the pre-trained baseline.
- The full TidyVoice dataset, evaluation trials, and developed models are publicly released.
- The dataset can be dynamically expanded to additional languages and speakers as Mozilla Common Voice grows.