Source-linked AI summary
CN-Celeb: multi-genre speaker recognition
Lantian Li, Ruiqi Liu, Jiawen Kang, Yue Fan, Hao Cui, Yunqi Cai, Ravichander Vipperla, Thomas Fang Zheng, Dong Wang
TL;DR
Speaker recognition lacks sufficient evidence and data for the difficult multi-genre setting, where intrinsic and extrinsic variations combine across genres and conditions. The paper publishes larger multi-genre data and studies baseline difficulty, multi-genre training, and generalization; it finds genre complexity and mismatch are major challenges and that multi-genre training improves performance. The study remains preliminary, with more data and stronger models still needed.
Problem
Multi-genre speaker recognition combines intrinsic and extrinsic variations, while existing systems and datasets provide inadequate support for conclusive study.
Method
The paper publishes CN-Celeb2, combines it with CN-Celeb1, evaluates speaker-recognition systems, and studies multi-genre training and meta-learning for novel genres.
Results
The study identifies genre complexity and genre mismatch as major challenges and obtains significant improvement from multi-genre training.
Takeaways & Limitations
CN-Celeb provides data sufficient to train and test speaker-recognition systems under multi-genre conditions, supporting research on robustness and generalization.
Takeaways & Limitations
Only 33% of CN-Celeb2 speakers are multi-genre, and the study is described as an initial, preliminary effort requiring more data and stronger models.
Abstract
from arXiv · showhide
Research on speaker recognition is extending to address the vulnerability in the wild conditions, among which genre mismatch is perhaps the most challenging, for instance, enrollment with reading speech while testing with conversational or singing audio. This mismatch leads to complex and composite inter-session variations, both intrinsic (i.e., speaking style, physiological status) and extrinsic (i.e., recording device, background noise). Unfortunately, the few existing multi-genre corpora are not only limited in size but are also recorded under controlled conditions, which cannot support conclusive research on the multi-genre problem. In this work, we firstly publish CN-Celeb, a large-scale multi-genre corpus that includes in-the-wild speech utterances of 3,000 speakers in 11 different genres. Secondly, using this dataset, we conduct a comprehensive study on the multi-genre phenomenon, in particular the impact of the multi-genre challenge on speaker recognition and the performance gain when the new dataset is used to conduct multi-genre training.
1. Introduction
Speaker recognition must handle diverse intrinsic and extrinsic speech variations, with multi-genre conditions combining especially complex changes across speaking style, recording conditions, and sessions. Existing systems perform substantially worse under multi-genre mismatch, while available data remains too limited for comprehensive study; CN-Celeb2 expands the corpus and enables broader investigation.
- Speaker-independent variations include recording devices, acoustics, noise, channels, pronunciation, speaking style, rate, emotion, and physical status.
- Multi-genre recognition combines complex variations, such as enrollment and testing speech differing in age, style, genre, environment, and microphone.
- 15.52% EER on CN-Celeb.E represented a 300% increase over the 3.75% SITW result, demonstrating severe multi-genre degradation.
- CN-Celeb1 had only 1,000 speakers and 270 hours, limiting its use as a standalone training set for deep learning models.
- CN-Celeb2 contains over 520,000 utterances from 2,000 Chinese celebrities across the same 11 genres as CN-Celeb1.
2. Speaker recognition: challenge, technique and data
Speaker recognition evolved from controlled pronunciation and phonetic variation toward session variation and unconstrained in-the-wild speech. Deep learning and larger corpora improved robustness, but existing datasets still constrain speaking styles and recording conditions, leaving multi-genre recognition inadequately supported.
- Early systems addressed pronunciation randomness in text-dependent recognition using DTW and HMM methods.
- GMM and GMM-UBM became effective approaches for phonetic variation in text-independent speaker recognition.
- i-vector and PLDA methods addressed session variation involving channels, speaking styles, and other long-term changes.
- In-the-wild recognition introduces unconstrained recording conditions and more complex variation because speakers may be unaware of recording.
- SITW and VoxCeleb enlarged unconstrained data resources, but their predominantly interview-based data constrained speaking-style and recording-condition variation.
- Existing datasets were not truly multi-genre, while CN-Celeb1 enabled only preliminary studies because multi-genre training data remained insufficient.
3. CN-Celeb2: features and collection pipeline
CN-Celeb2 extends CN-Celeb1 into a substantially larger multi-genre corpus collected from broader media sources, while retaining 11 genres and real-world recording diversity. Its main scope limitation is that only 33% of speakers are multi-genre, although multi-session data remains useful for studying the challenge.
- Corpus scale: CN-Celeb2 contains 529,485 utterances from 2,000 Chinese celebrities, totaling 1,090 hours—around four times CN-Celeb1's volume.
- Data sources: CN-Celeb2 expands media coverage beyond CN-Celeb1 by adding singing, recitation, and vlog data from multiple platforms.CN-Celeb1 data came from Bilibili, whereas CN-Celeb2 also used NetEase Cloud, Changba, Himalaya, and TikTok.
- Data profile: Short utterances form a larger proportion in both corpora, reflecting real-life applications while increasing speaker-recognition difficulty.
- Genre coverage: CN-Celeb2 adds coverage for underrepresented genres such as vlog and live broadcast.
- Multi-genre coverage: Only 33% of CN-Celeb2 speakers are multi-genre, compared with 75% in CN-Celeb1, because few celebrities are active across multiple domains.Despite fewer multi-genre speakers, CN-Celeb2 includes diverse multi-session data that can support methods for limited multi-genre conditions.
- Collection pipeline: The collection pipeline combines automatic segment extraction with human checking, which is especially necessary for complex genres with high pruning rates.Human checking is costly but removes incorrect automatically produced frames; pruning rates are relatively small for speech, recitation, and interview and high for play and movie.
4. Experiment I: Multi-genre challenge
The experiment studies how state-of-the-art speaker-recognition systems behave when training, enrollment, and test genres differ. It focuses on both genre mismatch between training and evaluation and mismatch between enrollment and test genres.
- The experiment evaluates systems when the enrollment or test genre differs from the training genre and when enrollment and test use different genres.
4.1. Basic results
The baseline study evaluates i-vector and x-vector systems on matched and multi-genre conditions using SITW and CN-Celeb.E. Performance is substantially worse on CN-Celeb.E, and more advanced x-vector techniques improve results on SITW but remain challenged by multi-genre data.
- Datasets and systems: The study builds i-vector and x-vector baselines and evaluates them on SITW and CN-Celeb.E.SITW represents single-genre testing, while CN-Celeb.E contains multi-genre speakers and tests multi-genre robustness.
- Datasets and systems: CN-Celeb.E contains 18,224 utterances from 200 multi-genre speakers, with 28-second enrollment and 8-second test utterances on average.
- Datasets and systems: The Kaldi SITW recipe defines the baseline systems, including MFCC-based i-vectors with PLDA scoring and an x-vector system.The recipe is used for reproducibility and is described as competitive and generalizable.
- Baseline results: More than 300%: the x-vector EER increases on CN-Celeb.E compared with SITW(S), showing severe degradation under multi-genre conditions.The overall baseline EER results are reported in Table 7.
- Baseline results: Advanced x-vector architectures improve performance on SITW but do not eliminate the difficulty of CN-Celeb.E.The tested techniques include ResNet, self-attentive pooling, and additive angular margin loss.
4.2. Within-genre results
Within-genre performance varies substantially across genres even when enrollment and test genres match. Speech-like genres perform relatively well, while singing, recitation, drama, and movie are substantially more difficult, and channel and language differences also affect interview results.
- Evaluation setup: Within-genre testing uses all 3,000 CN-Celeb speakers, filters utterances shorter than 5 seconds, and randomly enrolls with five utterances per speaker.The remaining utterances are used for testing, with DET curves reported alongside EER results.
- Genre differences: EER is below 8% for speech, live broadcast, vlog, and interview, but exceeds 12% for singing, recitation, drama, and movie.Both i-vector and x-vector systems show this genre-dependent performance pattern.
- Genre differences: Genre difficulty reflects both the amount of speaker-independent variation and the training-data bias toward interview speech.Speech and interview have more controlled channels, backgrounds, and speaking styles than singing and drama.
- Domain effects: 6.98% vs. 3.48%: x-vector EER is worse on CN-Celeb interview data than on SITW despite matched interview-like genres.The passage attributes this difference to channel and language discrepancies between CN-Celeb and SITW sources.
- Acceptance analysis: At the 7.43% overall-EER acceptance threshold, only speech, vlog, live broadcast, interview, and entertainment achieve reasonable performance.The thresholded comparison is illustrated for the x-vector system.
4.3. Cross-genre results
Cross-genre results are organized as enrollment-to-test performance matrices and show that most mismatched conditions are difficult. Performance depends on genre complexity and genre match, while calibration metrics reveal important score-calibration problems.
- Evaluation structure: The cross-genre matrices report EER for each enrollment genre by test genre, with diagonal cells representing within-genre results and a final column aggregating all test genres.The matrices are provided for both i-vector and x-vector systems.
- Enrollment effects: Speech enrollment gives the best average performance, whereas singing enrollment gives the worst; simpler enrollment conditions generally produce better results.The paper notes that enrollment commonly occurs in quiet environments with careful pronunciation.
- Mismatch effects: Matched directional pairs such as singing-speech and speech-singing have roughly similar results, suggesting comparable enrollment- and test-genre variation.
- Mismatch effects: Cross-genre performance depends on genre complexity and enrollment-test match: movie enrollment has 14.17% EER for movie tests but 11.67% for speech tests.The speech test is easier because its condition is simpler, despite the genre mismatch.
- Overall cross-genre performance: 7.43%: only several cross-genre conditions meet the x-vector system’s overall acceptance threshold, indicating that most conditions remain unacceptable.The paper characterizes the cross-genre phenomenon as highly complex and challenging.
- Score calibration: C_llr and C_min llr show trends similar to EER, so EER remains the main metric for relative-performance discussion.C_min llr reflects expected error rate, whereas EER is measured at equal false-acceptance and false-rejection rates.
- Score calibration: A large C_llr–C_min llr gap appears especially under stronger enrollment-test mismatch, indicating that score calibration matters when thresholds are used for decisions.Calibration does not change individual cross-genre EERs, but test-dependent calibration can make scores comparable across tests.
4.4. Statistical analysis
The statistical analysis interprets multi-genre degradation as changes in speaker-vector distributions relative to training. Matched genres resemble VoxCeleb more closely, whereas cross-genre enrollment and testing combine different statistical properties in one PLDA model.
- Analysis framework: The analysis computes inter-speaker variance, intra-speaker variance, and global mean shifts to explain performance degradation under within- and cross-genre testing.The interpretation assumes PLDA is optimal when vector distributions match the training condition and the model is well trained.
- Distribution shifts: VoxCeleb statistics are more similar to matched CN-Celeb genres such as speech and interview than to unmatched genres such as singing and recitation.Mean shifts are also less significant for matched genres.
- Distribution shifts: Because PLDA is trained on VoxCeleb, significant statistical changes and mean shifts can impair performance on genres unlike the training data.
- Cross-genre mismatch: Cross-genre conditions are especially problematic because enrollment and test data have different statistical properties but must be represented by one PLDA model.This links the observed mismatch to the model’s shared scoring representation.
4.5. Qualitative analysis
t-SNE visualizations show that x-vectors separate speakers better than i-vectors, but genre variation still produces overlapping and complex speaker distributions.
- Visualization setup: The visualizations project i-vector and x-vector representations from 10 speakers across 11 genres into two-dimensional t-SNE spaces.Figure 12 shows i-vectors and Figure 13 shows x-vectors, with colors separately encoding speakers and genres.
- i-vector analysis: i-vectors leave speakers largely intermingled because they reflect both speaker traits and acoustic conditions.The analysis attributes this difficulty to the i-vector model’s unsupervised nature.
- x-vector analysis: x-vectors produce larger inter-speaker distances and smaller intra-speaker distances than i-vectors.This indicates an advantage in handling acoustic complexity across genres.
- x-vector analysis: Genre complexity still creates complicated intra-speaker distributions and overlap among different speakers in x-vector representations.The remaining overlap demonstrates that multi-genre speaker recognition remains challenging.
5. Experiment II: Multi-genre training
The experiments compare single-genre, multi-genre, and partial multi-genre training for front-end and PLDA components. Multi-genre training improves multi-genre test performance overall, while partial multi-genre data provides lower-cost but smaller gains.
- Training schemes: Multi-genre training uses CN-Celeb.T, whereas single-genre training uses VoxCeleb; CN-Celeb.T contains 2,800 speakers.The experiments distinguish true multi-genre data from partial multi-genre data, where cross-genre speaker identities are removed.
- Evaluation: The study evaluates multi-genre training separately for i-vector/x-vector front ends and the PLDA back end using overall EER on CN-Celeb.E.The results are reported in Table 12.
- Front-end training: 20.35% vs. 20.13% shows no clear x-vector front-end advantage for multi-genre over single-genre training under cosine scoring.The authors attribute this result partly to the speaker-number imbalance between CN-Celeb.T and VoxCeleb.
- Front-end training: 20.83% vs. 20.35% shows partial multi-genre training is inferior to true multi-genre training for x-vectors under cosine scoring.For i-vectors, the two schemes obtain the same EER because their training is unsupervised; partial labels also lose cross-genre information for x-vectors.
- Back-end training: PLDA scoring improves with multi-genre training for both i-vector and x-vector systems.The reported comparison isolates the back-end effect by comparing single-genre and multi-genre training schemes.
- Partial multi-genre training: 14.76% vs. 16.59% shows partial multi-genre training outperforms single-genre training for the x-vector system, despite 14.76% vs. 13.44% being worse than true multi-genre training.The authors conclude that multi-genre data remains useful without cross-genre speakers and is cheaper to collect.
- Overall findings: Best performance is obtained when both front-end and PLDA models are multi-genre trained, while partial training provides reasonable lower-cost gains.The summary identifies true multi-genre training as more effective than partial multi-genre training.
6. Conclusion
The paper introduces CN-Celeb2 and combines it with CN-Celeb1 to enable broader multi-genre speaker-recognition experiments. It finds that genre complexity and mismatch are central challenges, while multi-genre training improves performance and multi-genre speakers are important.
- Dataset and study: CN-Celeb2 is combined with CN-Celeb1 to provide sufficient data for training and testing speaker-recognition systems in multi-genre conditions.The paper presents CN-Celeb2 as the dataset foundation for its comprehensive study.
- Main findings: The evaluation identifies difficult genres and shows that genre complexity and genre mismatch are major challenges in multi-genre speaker recognition.These findings come from evaluating state-of-the-art systems on the new dataset.
- Main findings: Multi-genre training yields significant performance improvement, and the experiments identify multi-genre speakers as important.The conclusion reports these as central outcomes of the second experiment.
- Scope and future work: The study is preliminary, with further work needed on larger datasets, genre-independent front ends, genre-robust back ends, and physiological models of pronunciation changes.The authors explicitly frame these as remaining research directions.
- Scope and future work: The authors anticipate that multi-genre variation will remain a major obstacle before speaker-recognition techniques achieve ubiquitous practical applicability.This conclusion preserves the paper’s forward-looking scope.