Source-linked AI summary
Your Voice Cloning System is Secretly a Voice Anonymizer
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak, Jan Deriu
TL;DR
Speaker anonymization must hide speaker identity while preserving linguistic content and quality. This paper repurposes XTTSv2 without retraining, conditioning voice conversion on original prosody and a pseudo-speaker identity, and evaluates iterative privacy-utility refinement across seven languages. The system achieves near-optimal privacy and substantially better speech quality than dedicated baselines, with competitive intelligibility.
Problem
Speaker anonymization research has been predominantly limited to English, while complex systems trained from scratch create barriers for low-resource languages.
Method
The paper repurposes XTTSv2 without retraining by combining original prosodic structure with a synthesized pseudo-speaker identity and iterative refinement.
Results
The system achieves near-optimal EER across seven languages and substantially improves speech quality relative to dedicated anonymization baselines.
Takeaways & Limitations
Existing multilingual TTS models can be leveraged for more accessible speaker anonymization across languages already supported by XTTSv2.
Abstract
from arXiv · showhide
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER $\approx$ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
1. Introduction
Speaker anonymization research faces multilingual coverage and complexity barriers. This work repurposes XTTSv2 to anonymize speakers without retraining while preserving prosody and targeting pseudo-speaker identities.
- Speaker anonymization suppresses identifying attributes while preserving linguistic content, prosody, and speech quality.
- Prior research has developed signal-processing and neural anonymization systems, including GAN- and self-supervised-learning approaches.
- Research has predominantly focused on English, limiting applicability to many languages and speakers.
- Complex pipelines trained from scratch create barriers for low-resource languages with scarce component-specific data.
- XTTSv2 is a multilingual voice-cloning model trained on 27,000 hours of speech and designed to preserve prosody and naturalness.
- The paper repurposes XTTSv2 without retraining, using pseudo-speaker conditioning and an iterative harmonic-mean objective to balance privacy and utility.
2. Model
The model repurposes XTTSv2’s voice-conversion components to combine original prosody and linguistic content with a composite pseudo-speaker identity. Iterative processing selects outputs balancing speaker dissimilarity and intelligibility.
- XTTSv2 generation is conditioned on a target speaker identity and the original speech’s prosodic structure.
- VQ-VAE extracts discrete codebook representations that preserve pitch contour, rhythm, and energy independently of speaker identity.
- The pipeline transcribes speech, extracts codebook and speaker representations, and selects gender-matched reference material for anonymization.
- A composite pseudo-speaker is created by selecting low-similarity utterances from 10 pool speakers and processing their concatenation through the Perceiver conditioner.
- 2.3. Iterative Refinement: The system iteratively reapplies anonymization to its output and selects the iteration maximizing the privacy-utility criterion H.
- Reference pools contain language- and gender-specific speakers selected for the best trade-off between speaker dissimilarity and intelligibility preservation.
3. Experimental Setup
The evaluation covers seven European languages in CommonVoice and Multilingual LibriSpeech, comparing XTTSv2-based anonymization with established baselines. Privacy, intelligibility, and speech quality are measured using shared evaluation models and human listening tests.
- Experiments use CommonVoice 23.0 and Multilingual LibriSpeech across English, German, French, Spanish, Italian, Portuguese, and Dutch.
- MLS provides high-quality audiobook recordings from 272 speakers, whereas CV includes 8,784 speakers and noisier consumer-recorded speech.
- Pseudo-speaker pools are built from clean MLS training speech using 10 speakers per language-gender combination.
- The comparison includes SALT and MultiLingual anonymization systems alongside the XTTSv2-based approach.
- The pipeline extracts transcription, prosody, speaker identity, and gender before constructing pseudo-speakers and converting voices.
- Iterative refinement applies the pipeline to its own output while computing a harmonic mean balancing privacy and utility.
- Privacy uses ECAPA2 EER, intelligibility uses Whisper-Large-V3 WER, and quality uses ∆UTMOS plus English CMOS listening tests.
4. Results
Across seven languages and two datasets, the XTTSv2-based system achieves near-optimal privacy and generally competitive intelligibility. It substantially preserves speech quality relative to dedicated anonymization baselines, with dataset-dependent quality differences.
- 0.49 average EER on CV and 0.46 on MLS approach the 0.50 theoretical maximum for privacy.
- 0.16 average WER on CV compares with 0.26 for SALT and 0.27 for MultiLingual, while MLS WER is 0.16.
- 38% relative WER reduction on CV is reported relative to the anonymization baselines.
- +0.17 ∆UTMOS on CV contrasts with -0.74 for SALT and -0.62 for MultiLingual.
- -0.35 ∆UTMOS on MLS is less degraded than -1.29 for SALT and -1.23 for MultiLingual.
- CMOS = -0.90 on English MLS is the least degradation, compared with -1.00 for SALT and -1.85 for MultiLingual.
- Iterative Refinement: Iteration 4 is selected in 25.0% of cases, while iteration 0 is selected in 19.6% and iterations 1–3 each approximately 18%.
5. Discussion & Conclusion
The approach uses XTTSv2 for speaker anonymization without retraining, achieving near-optimal EER across seven languages and substantially improving speech quality over dedicated baselines. Future work targets informed attackers, non-European languages, and other voice cloning architectures.
- The approach leverages XTTSv2 for speaker anonymization without retraining.
- The system achieves near-optimal EER across seven languages while substantially improving speech quality relative to dedicated baselines.
- Future work should address informed attackers, extend to non-European languages, and explore other voice cloning architectures.
6. AI Declaration
The authors used Claude Opus 4.5 to edit and rewrite text passages for better readability.
- Claude Opus 4.5 was used to edit passages of text.
- Claude Opus 4.5 was used to rewrite passages of text.
- The editing and rewriting aimed to improve readability.