Source-linked AI summary
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone
Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, Moacir Antonelli Ponti
TL;DR
Zero-shot TTS still struggles to generalize across unseen speakers, languages, and differing voice or recording characteristics. YourTTS extends VITS with multilingual and speaker-conditioning modifications, achieving strong zero-shot TTS and voice-conversion results while supporting single-speaker target-language training and short speaker adaptation. The paper also documents instability, mispronunciations, gender effects, and longer speech requirements for higher-quality adaptation.
Problem
Zero-shot TTS faces a similarity gap for unseen speakers and requires substantial, diverse training speakers, limiting quality in low-resource languages and mismatched voice conditions.
Method
YourTTS builds on VITS with novel modifications for zero-shot multi-speaker and multilingual training, including multilingual conditioning and speaker-conditioned generation.
Results
YourTTS achieves state-of-the-art zero-shot multi-speaker TTS and comparable zero-shot voice-conversion results on VCTK, with promising single-speaker target-language results and less-than-one-minute adaptation.
Takeaways & Limitations
The approach supports zero-shot synthesis in a target language with one training speaker and adaptation to substantially different voices or recording conditions using under 1 minute of speech.
Takeaways & Limitations
The model can produce unnatural durations and mispronunciations, Portuguese voice-conversion performance is gender-sensitive, and more than 45 seconds is more adequate for higher-quality adaptation.
Abstract
from arXiv · showhide
YourTTS brings the power of a multilingual approach to the task of zero-shot multi-speaker TTS. Our method builds upon the VITS model and adds several novel modifications for zero-shot multi-speaker and multilingual training. We achieved state-of-the-art (SOTA) results in zero-shot multi-speaker TTS and results comparable to SOTA in zero-shot voice conversion on the VCTK dataset. Additionally, our approach achieves promising results in a target language with a single-speaker dataset, opening possibilities for zero-shot multi-speaker TTS and zero-shot voice conversion systems in low-resource languages. Finally, it is possible to fine-tune the YourTTS model with less than 1 minute of speech and achieve state-of-the-art results in voice similarity and with reasonable quality. This is important to allow synthesis for speakers with a very different voice or recording characteristics from those seen during training.
1. Introduction
YourTTS addresses persistent generalization challenges in zero-shot multi-speaker TTS through a multilingual approach, reporting strong results across speech synthesis and voice conversion tasks.
- ZS-TTS aims to synthesize voices for speakers unseen during training using only a few seconds of speech.
- Similarity between observed and unobserved speakers remains an open problem, especially with limited speaker diversity, low-resource languages, or mismatched voice characteristics.
- YourTTS introduces novel modifications focused on zero-shot multi-speaker and multilingual training.
- The work reports state-of-the-art zero-shot multi-speaker TTS results and results comparable to SOTA for zero-shot voice conversion on VCTK.
- YourTTS is presented as the first multilingual approach for zero-shot multi-speaker TTS, with state-of-the-art English results.
- The model supports promising target-language zero-shot synthesis with one target-language training speaker and can be fine-tuned with less than 1 minute of speech.
2. YourTTS Model
YourTTS builds on VITS with multilingual conditioning, speaker conditioning, and training objectives designed for zero-shot multi-speaker synthesis and voice consistency.
- YourTTS builds upon VITS and uses raw text instead of phonemes to support languages lacking strong open-source grapheme-to-phoneme converters.
- Trainable language embeddings are concatenated with character embeddings, while the text encoder is expanded to 10 transformer blocks and 196 hidden channels.
- External speaker embeddings condition the flow decoder, posterior encoder, vocoder, duration predictor, and vocoder-related outputs.
- Speaker Consistency Loss maximizes cosine similarity between speaker embeddings extracted from generated and ground-truth audio.
- During training, the posterior encoder predicts latent variable z from spectrograms and speaker embeddings, while MAS aligns the prior with text-encoder outputs.
- The stochastic duration predictor uses speaker and language embeddings with MAS-derived durations to model human-like speech rhythms.
3. Experiments
Experiments evaluate YourTTS across multilingual datasets, unseen speakers, training configurations, and implementation settings for zero-shot synthesis and adaptation.
- Audio datasets: The study uses English VCTK, Portuguese TTS-Portuguese, and French M-AILABS datasets, with one dataset per language for training.
- Evaluation data: Speaker adaptation experiments use four Common Voice speakers to test adaptation under more realistic conditions.
- Training experiments: Four experiments compare monolingual, bilingual, trilingual, and expanded English-speaker training configurations.
- Training setup: Transfer learning accelerates training, while later experiments continue from earlier models and apply Speaker Consistency Loss fine-tuning.
- Training setup: Multilingual experiments use weighted random sampling to produce language-balanced batches during training.
4. Results and Discussion
YourTTS is evaluated with MOS, SECS, and Sim-MOS on unseen speakers across English and Portuguese datasets. Results show strong VCTK performance, dataset- and training-dependent similarity, and promising Portuguese synthesis despite limited speaker coverage.
- VCTK dataset: SCL improved SECS similarity in 2 of 3 VCTK experiments, but overlapping Sim-MOS confidence intervals make its effect on that metric inconclusive.For experiment 2, both metrics agreed on a positive effect of SCL.
- VCTK dataset: VCTK experiments achieved quality and similarity similar to ground truth and superior results to prior studies, establishing state-of-the-art zero-shot multi-speaker TTS performance.The best experiments maintained good quality and similarity for speakers unseen during training.
- LibriTTS dataset: Experiment 4 achieved the best LibriTTS similarity, while the monolingual experiment achieved the best MOS, reflecting differences in speaker diversity and training-data quality.Experiment 4 used approximately 1.2k speakers, whereas experiment 1 used only VCTK data.
- Portuguese MLS dataset: 4.11±0.07 MOS was highest for Portuguese experiment 3+SCL, although confidence intervals overlapped with the other experiments.Experiment 3 had the best Sim-MOS at 3.19±0.10, while SECS favored experiment 4+SCL.
- Portuguese MLS dataset: Adding French appeared to improve Portuguese quality and SECS similarity, partly because of better French data quality and reduced dominance of the single Portuguese training speaker.The explanation follows language-balanced batching across the datasets.
5. Zero-Shot Voice Conversion
YourTTS performs zero-shot voice conversion by combining speaker-independent encoder behavior with external speaker embeddings. It achieves strong English intra-lingual conversion, while Portuguese and some cross-lingual transfers expose limitations associated with missing female voices during training.
- Method: External speaker embeddings enable YourTTS to mimic unseen speakers during zero-shot voice conversion using its posterior encoder, decoder, and HiFi-GAN generator.The encoder receives no speaker-identity information, encouraging a speaker-independent predicted distribution.
- Intra-lingual results: 4.20±0.05 MOS and 4.07±0.06 Sim-MOS were achieved for English-to-English conversion, compared with lower reported AutoVC and NoiseVC results.The evaluation selected eight balanced VCTK test speakers, while the cited comparison used ten unseen speakers.
- Intra-lingual results: 3.64±0.09 MOS and 3.43±0.09 Sim-MOS were achieved for Portuguese-to-Portuguese conversion, with lower similarity for female-to-female than male-to-male transfers.Female-to-female similarity was 3.35±0.19 versus 3.80±0.15 for male-to-male transfers.
- Cross-lingual results: Cross-language transfer between English and Portuguese appeared comparable to Portuguese intra-lingual transfer, but Portuguese-to-English quality dropped, especially for Portuguese male to English female conversion.The authors associate the difficulty with converting across language and gender under limited female-speaker training data.
- Cross-lingual results: The absence of Portuguese female voices during training hindered generalization for transfers involving Portuguese, while English conversion performance was not significantly influenced by gender.This limitation is specific to the reported gender-generalization pattern.
6. Speaker Adaptation
Speaker adaptation tests whether brief target-speaker recordings can improve YourTTS for voices and recording conditions absent from training. Fine-tuning with under one minute substantially improves similarity, but shorter adaptation can reduce naturalness.
- Evaluation: The adaptation evaluation reports SECS, MOS, and Sim-MOS for ground truth, zero-shot synthesis, and fine-tuned synthesis.Table 3 also records speaker gender, speech duration, and sample counts.
- Speaker adaptation: Fine-tuning with less than 1 minute of speech from speakers with unseen recording characteristics significantly improved similarity in all experiments.The adaptation used four Common Voice speakers and fine-tuned each individually from an experiment 4 checkpoint with SCL.
- English adaptation: In English, fine-tuning produced Sim-MOS comparable to ground truth for both male and female speakers.Fine-tuned SECS exceeded ground truth, which the authors relate to copying recording characteristics and reference-sample distortions.
- Portuguese adaptation: Portuguese male similarity increased from 3.35±0.12 to 4.19±0.07 after 31 seconds of fine-tuning, while female similarity rose from 2.77±0.15 to 4.43±0.06 after 20 seconds.These comparisons are between zero-shot and fine-tuned modes.
- Quality trade-off: Approximately 1 minute of adaptation can preserve or increase naturalness, whereas 44 seconds or less reduces generated-speech quality compared with zero-shot or ground-truth speech.The paper describes a direct relationship between speech amount and MOS in these experiments.
7. Conclusions, limitations and future work
YourTTS achieves state-of-the-art results on VCTK zero-shot multi-speaker TTS and voice conversion, with promising results for single-speaker target-language training. It can also adapt to substantially different voices and recording conditions using less than one minute of speech, although limitations remain.
- Conclusions: Single-speaker target-language training produces promising results for zero-shot multi-speaker TTS and voice conversion.
- Limitations: TTS instability can produce unnatural durations, and mispronunciations occur for some words, especially in Portuguese.The model does not use phonetic transcriptions, which makes it more prone to mispronunciation problems.
- Limitations: Portuguese voice-conversion performance is significantly influenced by speaker gender because female voices are absent from training.
- Future work: Future work targets duration-predictor improvements, training in more languages, and data augmentation for low-resource automatic speech recognition.
A. Erratum
An implementation mistake prevented Speaker Consistency Loss gradients from propagating during four fine-tuning experiments. The bug was subsequently fixed in Coqui TTS version v0.12.0 or higher.
- Erratum: An implementation mistake stopped Speaker Consistency Loss gradients from propagating during four fine-tuning experiments.Those experiments were therefore equivalent to training for more steps without Speaker Consistency Loss.
- Erratum: The bug was fixed in Coqui TTS version v0.12.0 or higher.