Source-linked AI summary
Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
Tzu-Chieh Wei, Yi-Cheng Lin, Huang-Cheng Chou, Kuan-Yu Chen, Hsin-Yen Sung, Shrikanth Narayanan, Hung-yi Lee
TL;DR
Speaker verification systems struggle to assess identity consistently in non-verbal vocalizations (NVVs) and can lose speech performance when adapted to them. The paper evaluates 10 NVV types and proposes domain-aware Mixture-of-Experts routing with conditional distillation and contrastive domain bridging, reducing speech-NVV EER from 38.93% to 22.66% while improving speech EER from 13.17% to 9.24%.
Problem
Existing speaker verification systems generalize poorly to NVVs, while adapting them to NVV data can substantially degrade speech-to-speech verification.
Method
The framework combines frozen Data2Vec and ECAPA-TDNN features with domain-aware MoE routing, conditional distillation, and supervised contrastive domain bridging.
Results
38.93% to 22.66%: the framework reduces speech-NVV EER while improving speech EER from 13.17% to 9.24%.
Takeaways & Limitations
The study provides a systematic 10-type NVV evaluation and a speaker verification approach that bridges speech-NVV mismatch while mitigating speech-performance degradation.
Takeaways & Limitations
Fine-tuned configurations, including this approach, still have higher SvS EER than the zero-shot wavlm-base-plus-sv baseline, reflecting a training-data scale discrepancy.
Abstract
from arXiv · showhide
As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.
1. Introduction
Modern TTS and VC systems approach human naturalness for modal speech, but reliable speaker verification across speech and diverse non-verbal vocalizations remains unresolved. The paper frames NVV verification as a heterogeneous multi-domain problem and proposes MoE routing with conditional knowledge distillation to preserve speech performance.
- Motivation: TTS and VC systems achieve near-human naturalness for modal speech, while realistic laughter, coughing, and breathing remain challenging to generate.Generated NVVs must also maintain speaker identity, requiring automated and scalable speaker verification across verbal and non-verbal segments.
- Challenge: Different NVV categories have distinct acoustic signatures, requiring speaker verification systems to accommodate heterogeneous domains that diverge from phonemically structured speech.Whether modern speaker-ververification representations generalize across this acoustic diversity remains unexplored.
- Challenge: Existing speaker verification systems show greater overlap between same-speaker and different-speaker score distributions on NVV pairs, while NVV fine-tuning severely degrades speech-to-speech verification.The paper identifies this speech-performance degradation as a critical side effect overlooked in much prior NVV work.
- Proposed approach: The proposed framework combines Mixture of Experts routing with conditional knowledge distillation within an SSL and ECAPA-TDNN pipeline.Dedicated expert subnetworks are motivated by the acoustic heterogeneity between speech and diverse non-phonemic vocalizations, while distillation targets preservation of baseline speech performance.
- Contributions: The study systematically evaluates modern speaker verification across a taxonomy of 10 NVV types and introduces conditional distillation to learn non-verbal representations while mitigating speech-capability loss.The contributions also include an MoE-based system that separates nonverbal and speech processing paths.
2. Methodology
The methodology combines a frozen Data2Vec–ECAPA-TDNN speaker-verification backbone with domain-aware Mixture-of-Experts routing. Multi-objective training uses routing constraints, conditional distillation, and supervised contrastive bridging to specialize NVV representations while preserving speech structure.
- Architecture: The framework uses a frozen Data2Vec feature extractor and ECAPA-TDNN speaker-embedding backend, avoiding phonemic discretization in the front end.Data2Vec predicts continuous latent representations through self-distillation, and its NVV verification performance empirically motivates the choice.
- Mixture-of-Experts Routing: An MoE gating network routes frame representations to specialized experts through TopK(softmax(W_gh), k), with k = 2 in all experiments.The routed output is a weighted sum of the selected experts’ outputs.
- MoE Integration Strategies: The study compares Post-Fusion MoE, which adapts weighted frozen hidden states, with IR-MoE, which inserts trainable adapters after each frozen transformer block.IR-MoE propagates adapted representations forward through the transformer stack.
- Multi-Objective Training: Training jointly optimizes the standard AAM-softmax speaker loss, event-guided routing constraints, conditional knowledge distillation, and supervised contrastive domain bridging.Routing penalties promote load balancing, intra-event consistency, and inter-event separation without hard-coding expert assignments.
- Domain Preservation and Bridging: Conditional distillation applies only to speech using a frozen WavLM-based teacher, while contrastive positives include same-speaker speech-NVV pairs and negatives come from different speakers.At inference, utterance-type labels are unnecessary because the event-guided MoE router autonomously routes speech and NVV inputs.
3. Experiment
The experiment evaluates speaker verification across separated speech and non-verbal vocalization data using the NonverbalTTS dataset, fixed ECAPA-TDNN backends, and multiple SSL baselines. Performance is assessed with cosine-scored EER and mDCF, while expert-count analysis identifies a four-expert configuration as best for NvS EER.
- Data preparation: MFA word-level alignments separate utterances into isolated NVV clips and speech segments for MoE routing and conditional distillation.This preprocessing creates distinct speech and NVV data streams for specialized nonverbal speaker verification training.
- Baselines: The proposed model is compared with seven ECAPA-TDNN baselines using Fbank, single-SSL, or dual-SSL front-ends, including Data2Vec, WavLM, and Voc2Vec configurations.The dual-SSL systems are WavLM + Voc2Vec, Data2Vec+Voc2Vec, and WavLM + Data2Vec.
- Domain-gap analysis: Target speech–vocal burst trials overlap substantially with non-target distributions, indicating a domain gap and increased false-rejection risk in ECAPA-TDNN.The overlap is shown through cosine similarity distributions for speech–vocal burst trials.
- Ablation: The four-expert IR-MoE configuration achieves the best NvS EER, while SvS EER continues decreasing as the number of experts increases.This experiment isolates the effect of IR-MoE expert count on verification performance.
- Metrics and configuration: Verification uses cosine similarity scoring and reports Equal Error Rate (EER) and minimum normalized detection cost function (mDCF) with Ptarget = 0.05.The downstream ECAPA-TDNN architecture remains fixed across experiments, using 1024 convolutional frame-level channels and 192-dimensional speaker embeddings.
4. Results and Analyses
Results show that speech-trained speaker verification generalizes poorly to non-verbal vocalizations, while conditional distillation and a four-expert MoE improve cross-domain verification and preserve speech performance. Fine-tuning still trails the zero-shot speech baseline on SvS, reflecting the training-data scale gap.
- Domain mismatch: 5.60% SvS EER for wavlm-base-plus-sv rises to 38.93% on NvS and 39.13% on NvN, exposing severe zero-shot domain mismatch.The results indicate that modal-speech representations fail to generalize to the irregular acoustic structures of NVVs.
- Conditional distillation: 13.17% SvS EER with AAM-Softmax and supervised contrastive loss falls to 9.24% with conditional distillation, while NvS EER reaches 22.66%.Distillation constrains speech embeddings toward pretrained-teacher representations and bridges the speech-NVV domain gap.
- Limitations: 5.60% SvS EER remains better than every fine-tuned configuration, including the proposed model, because VoxCeleb2 and NonverbalTTS differ greatly in training-data scale.The paper identifies large-scale joint training as future work for addressing this discrepancy.
- Mixture of Experts: 22.66% NvS EER is achieved by the four-expert IR-MoE configuration, which provides the best overall balance through separate verbal and non-verbal processing paths.The strongest standalone Data2Vec+ECAPA-TDNN baseline has 23.33% NvS EER but lacks dynamic feature routing.
5. Conclusion and Future Work
This paper presents the first systematic study of speaker identity verification across 10 NVV types and identifies acoustic mismatch and catastrophic forgetting as key limitations of adapting standard SV models to NVVs. It proposes an Inter-Layer Residual MoE architecture with conditional distillation to address these challenges.
- Contributions: The study evaluates speaker identity verification across 10 distinct NVV types.It is presented as the first systematic study of this scope.
- Limitations: Standard speaker verification models suffer severe acoustic mismatch when adapted to NVVs.The conclusion identifies acoustic mismatch as a critical limitation in current SV systems.
- Limitations: Adapting standard speaker verification models to NVVs also causes catastrophic forgetting.This is described as a second major failure mode alongside acoustic mismatch.
- Proposed approach: The proposed solution combines an Inter-Layer Residual MoE architecture with a novel conditional distillation strategy.The framework is introduced to overcome the identified adaptation challenges.
7. Generative AI Use Disclosure
Generative AI tools were used only to polish the manuscript’s language, not to contribute to its substantive scientific content. The authors remain solely responsible for the research design, experiments, analysis, and reported results.
- 7. Generative AI Use Disclosure: Generative AI tools assisted with language polishing, while the authors retained sole responsibility for the research design, experiments, analysis, and reported results.The tools did not contribute to the substantive scientific content.