Source-linked AI summary
CN-CELEB: a challenging Chinese speaker recognition dataset
Yue Fan, Jiawen Kang, Lantian Li, Kaicheng Li, Haolin Chen, Sitong Cheng, Pengyuan Zhang, Ziya Zhou, Yunqi Cai, Dong Wang
TL;DR
Speaker recognition systems remain insufficiently reliable in unconstrained conditions, while common datasets are too constrained to evaluate that difficulty. The paper introduces CN-Celeb, a large Chinese, multi-genre dataset collected with automated processing and human checks. Experiments show substantially worse performance on CN-Celeb than on VoxCeleb, indicating that real-life performance may be much worse than previously thought.
Problem
Existing speaker-recognition datasets are mostly collected under constrained conditions, limiting research on arbitrary real-world variation and tending to produce overoptimistic performance.
Method
The paper constructs CN-Celeb by re-implementing VoxCeleb’s automated pipeline, adding a face-speaker double check and human verification.
Results
CN-Celeb performance was significantly worse than VoxCeleb performance, including 14.24% on CN-Celeb(E) versus 11.34% after VoxCeleb(L) backend retraining on SITW.
Takeaways & Limitations
CN-Celeb provides a free resource for evaluating speaker-recognition systems under unconstrained real-world conditions.
Abstract
from arXiv · showhide
Recently, researchers set an ambitious goal of conducting speaker recognition in unconstrained conditions where the variations on ambient, channel and emotion could be arbitrary. However, most publicly available datasets are collected under constrained environments, i.e., with little noise and limited channel variation. These datasets tend to deliver over optimistic performance and do not meet the request of research on speaker recognition in unconstrained conditions. In this paper, we present CN-Celeb, a large-scale speaker recognition dataset collected `in the wild'. This dataset contains more than 130,000 utterances from 1,000 Chinese celebrities, and covers 11 different genres in real world. Experiments conducted with two state-of-the-art speaker recognition approaches (i-vector and x-vector) show that the performance on CN-Celeb is far inferior to the one obtained on VoxCeleb, a widely used speaker recognition dataset. This result demonstrates that in real-life conditions, the performance of existing techniques might be much worse than it was thought. Our database is free for researchers and can be downloaded from http://project.cslt.org.
1. INTRODUCTION
Speaker recognition remains unreliable in unconstrained conditions because recordings can vary arbitrarily in text, channels, noise, speaking style, and physiology. CN-Celeb addresses the shortage of challenging data with a Chinese, multi-genre dataset collected using automated pre-selection and human checking.
- Unconstrained speaker recognition remains unreliable because recordings may contain arbitrary variation in text, channels, noise, speaking styles, and physiological status.
- Most existing datasets use constrained acoustic environments, channels, and speaking styles, producing overoptimistic performance and insufficient evidence for unconstrained research.
- CN-Celeb contains more than 130,000 utterances from 1,000 Chinese celebrities and targets speaker recognition in real-world conditions.
- CN-Celeb spans 11 speech genres, whereas most VoxCeleb utterances came from interviews; this diversity improves real-world representativeness while increasing difficulty.
- The collection combines automated pipeline pre-selection with human checking because complex genres increase automated errors and may contain valuable boundary samples.
2. THE CN-CELEB DATASET
CN-Celeb is a large Chinese speaker-recognition dataset designed to complement VoxCeleb with diverse, noisy, short, and cross-channel real-world speech. Its collection pipeline combines automated processing with human verification to improve efficiency and precision.
- 2.1. Data description: CN-Celeb contains over 130,000 utterances from 1,000 Chinese celebrities across 11 genres and 274 hours of speech.
- 2.2. Challenges with CN-Celeb: CN-Celeb includes ambient noise, background babbling, music, cheers, laughter, and overlapped background speakers, especially in drama and movie genres.
- 2.2. Challenges with CN-Celeb: Different genres for the same speaker create substantial speaking-style variation, while different recording times and devices create cross-time and cross-channel problems.
- 2.2. Challenges with CN-Celeb: Most utterances are short, matching common applications but leading to unreliable decisions.
- 2.3. Collection pipeline: The pipeline uses automated extraction followed by human checking, making segmentation faster than purely manual processing and reducing errors from purely automated processing.
- 2.3. Collection pipeline: A face-speaker double check fuses image and speech information to increase recall while maintaining precision.
- 2.3. Collection pipeline: SyncNet failures in complex genres motivated an additional speaker-recognition confidence check for retaining or discarding segments.
3. EXPERIMENTS ON SPEAKER RECOGNITION
Experiments compare i-vector and x-vector speaker-recognition systems across VoxCeleb and CN-Celeb evaluation settings. CN-Celeb produces substantially poorer performance, indicating greater challenge and limited cross-dataset generalization.
- Experimental setup: The experiments compare VoxCeleb and CN-Celeb using i-vector and x-vector speaker-recognition systems with LDA-PLDA back ends.VoxCeleb and CN-Celeb were separately configured for training and evaluation, with comparable VoxCeleb(L) and CN-Celeb(T) data settings used for back-end training.
- Data: CN-Celeb(T) contains 111,260 training utterances from 800 speakers, while CN-Celeb(E) contains 18,849 evaluation utterances from 200 speakers.
- Basic results: For VoxCeleb-trained systems, EER performance on CN-Celeb(E) is much worse than on SITW and SITW(S) for both i-vector and x-vector systems.SITW(S) was resegmented to make average utterance lengths similar to CN-Celeb(E).
- Further comparison: 14.24% is the reported performance for a pure CN-Celeb system on CN-Celeb(E), despite both training and evaluation data coming from CN-Celeb.With VoxCeleb(L) back-end retraining, SITW performance reaches 11.34%, lower than the same-source CN-Celeb(E) result.
- Further comparison: CN-Celeb differs significantly from VoxCeleb and is more challenging for speaker-recognition research.The comparative experiments show the same trend for i-vector and x-vector systems.
4. CONCLUSIONS
The paper introduces CN-Celeb as a free dataset for speaker-recognition research, with broad real-world coverage. Experiments show that it is significantly different from and more challenging than VoxCeleb, suggesting current techniques may perform much worse in unconstrained conditions than expected.
- CN-Celeb is a free speaker-recognition dataset containing more than 130k utterances from 1,000 Chinese celebrities across 11 real-world genres.
- Experiments using two state-of-the-art speaker-recognition models show that CN-Celeb is significantly different from VoxCeleb and more challenging for research.
- The reported EER performance suggests current speaker-recognition techniques might perform much worse in unconstrained conditions than previously thought.