Source-linked AI summary
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, Julian Weber
TL;DR
ZS-TTS systems have generally supported one language or only a few higher-resource languages, leaving a need for broader multilingual coverage. XTTS extends Tortoise with architectural changes for multilingual training, voice cloning, and faster inference. Trained in 16 languages, it achieved state-of-the-art results in multilingual ZS-TTS and was publicly released.
Problem
Existing ZS-TTS systems mostly support one language, while multilingual systems cover few languages and remain limited to several medium- and high-resource languages.
Method
XTTS builds on Tortoise with modifications for multilingual training, voice cloning, and faster training and inference, including a VQ-VAE and conditioned GPT-2 encoder.
Results
XTTS achieved state-of-the-art multilingual ZS-TTS results in 16 languages and can perform cross-language ZS-TTS without parallel training data.
Takeaways & Limitations
XTTS provides a publicly available massively multilingual ZS-TTS model covering low- and medium-resource languages.
Abstract
from arXiv · showhide
Most Zero-shot Multi-speaker TTS (ZS-TTS) systems support only a single language. Although models like YourTTS, VALL-E X, Mega-TTS 2, and Voicebox explored Multilingual ZS-TTS they are limited to just a few high/medium resource languages, limiting the applications of these models in most of the low/medium resource languages. In this paper, we aim to alleviate this issue by proposing and making publicly available the XTTS system. Our method builds upon the Tortoise model and adds several novel modifications to enable multilingual training, improve voice cloning, and enable faster training and inference. XTTS was trained in 16 languages and achieved state-of-the-art (SOTA) results in most of them.
1. Introduction
Existing ZS-TTS systems largely support one language, while multilingual efforts cover few languages and mainly higher-resource settings. XTTS addresses this gap with a publicly available model trained across 16 languages.
- Motivation: Most ZS-TTS models support only a single language, motivating multilingual approaches for synthesizing unseen speakers from limited speech.Multilingual training can reduce the speech hours and speakers needed for a target language.
- Motivation: Prior multilingual systems supported limited language sets: YourTTS used three languages, VALL-E X and Mega-TTS 2 used two, and Voicebox used six.These systems therefore remained concentrated on a few medium- and high-resource languages.
- XTTS contribution: XTTS is a massively multilingual ZS-TTS model supporting 16 specified languages, including English, Spanish, French, German, Chinese, Korean, and Japanese.The supported set also includes Italian, Portuguese, Polish, Turkish, Russian, Dutch, Czech, Arabic, and Hungarian.
- XTTS contribution: XTTS builds on Tortoise with modifications for multilingual training, improved voice cloning, and faster training and inference.The system was trained in 16 languages and reports state-of-the-art results in most of them.
- Reported contribution: The authors report that XTTS achieves state-of-the-art multilingual ZS-TTS results in 16 languages and supports cross-language synthesis without parallel training data.XTTS checkpoints are publicly available through Coqui TTS and Hugging Face.
2. XTTS model
XTTS combines a VQ-VAE, GPT-2-based encoder, and HiFi-GAN-based decoder, with conditioning and codebook changes designed for multilingual voice cloning and efficient synthesis.
- Architecture: XTTS builds upon Tortoise and comprises three components: VQ-VAE, encoder, and decoder.Figure 1 provides an overview of the XTTS architecture.
- VQ-VAE: The VQ-VAE encodes mel-spectrogram frames with one 8192-code codebook at 21.53 Hz, then retains the 1024 most frequent codes.The authors report that filtering less frequent codes improved expressiveness in preliminary experiments.
- Encoder: The 443M-parameter GPT-2 encoder maps 6681-token BPE text and conditioning embeddings to predicted VQ-VAE audio codes.Its Conditioning Encoder produces 32 1024-dimensional embeddings from mel-spectrograms using attention layers and a Perceiver Resampler.
- Encoder: Using 32 conditioning embeddings instead of one improved speaker-cloning capability during massive multilingual training in preliminary experiments.The conditioning embeddings are produced independently of input audio length.
- Decoder: The 26M-parameter HiFi-GAN-based decoder consumes GPT-2 latent vectors and speaker embeddings rather than VQ-VAE codes.This design avoids pronunciation issues and artifacts associated with reconstructing audio directly from highly compressed VQ-VAE codes.
3. Experiments
The experiments assemble multilingual and monolingual training comparisons using controlled datasets, revised YourTTS baselines, and three training runs. XTTS and YourTTS were trained under different compute and optimization configurations.
- Dataset: The XTTS dataset combines public and internal data, with most internal data in English and Common Voice supplying much of the other-language data.English includes 541.7 hours from LibriTTS-R and 1812.7 hours from LibriLight.
- Baseline design: The authors argue that prior comparisons with the original multilingual YourTTS checkpoint were unfair because speech hours and speaker counts strongly affect ZS-TTS training.That checkpoint used more than 1k English speakers but only 5 French and 1 Portuguese speaker, with 66% of batches drawn from 6 speakers.
- Baseline design: YourTTS was retrained on LibriTTS and the XTTS dataset to enable comparisons with English systems, the original multilingual checkpoint, and XTTS.A language batch balancer was used for both XTTS and YourTTS trained on the XTTS dataset.
- Training experiments: The study conducted three runs: English YourTTS for 405k steps, 16-language YourTTS for 1.96M steps, and XTTS for approximately 2.5M steps.The first two runs fixed the SCL bug.
- Training setup: YourTTS experiments used one NVIDIA A100 GPU, whereas XTTS used four A100 GPUs; YourTTS also used transfer learning for speed.The papers report separate optimizer and learning-rate configurations for YourTTS and XTTS.
4. Results and Discussion
XTTS was evaluated against strong zero-shot TTS baselines using objective, subjective, and multilingual tests. It achieved the best CER in English and better CER and speaker similarity in almost all evaluated languages, while user studies favored it for several quality dimensions.
- Evaluation setup: The evaluation compared XTTS with StyleTTS 2, Tortoise, YourTTS, HierSpeech++, Mega-TTS 2, and a YourTTS model trained on the XTTS dataset.The evaluation used 240 FLORES+ sentences per language, 20 DAPS speakers, and 3–8-second reference segments for cross-lingual testing.
- Evaluation metrics: The English objective evaluation reported CER, UTMOS, and SECS, while the subjective evaluation measured CMOS and SMOS preferences.The objective quality measure used UTMOS as an easily comparable, though non-absolute, metric.
- English evaluation: XTTS achieved the best CER in English and competitive UTMOS and SECS results among the compared models.The comparison also indicates that multilingual and monolingual YourTTS results differ substantially, making direct comparisons between them unfair.
- English evaluation: XTTS received significantly better CMOS scores for naturalness, acoustic quality, and human likeness than previous works.Its SMOS score was slightly worse than previous models, which the authors associate with the complexity of massive multilingual training.
- Multilingual evaluation: XTTS achieved better CER and speaker similarity than Mega-TTS 2 in almost all supported languages.The multilingual comparison used XTTS and YourTTS trained on the XTTS dataset alongside the original Mega-TTS 2 model.
5. Speaker Adaptation
The speaker-adaptation experiment tested XTTS on distinctive voices and recording conditions using approximately 10 minutes of speech per selected speaker. Fine-tuning improved cross-lingual speaker similarity substantially.
- Speaker selection: The adaptation study selected well-known or unique-style voices, including whispering voices, across multiple languages to test robustness to new speakers and recording conditions.The authors identify differing recording conditions and unusual speaker voices as challenges for ZS-TTS generalization.
- Fine-tuning results: Fine-tuning improved SECS from 0.5852 to 0.7166 when cloning the selected voices cross-lingually.The evaluation replaced the DAPS speakers with the selected speakers while retaining the cross-lingual evaluation approach.
- Fine-tuning results: The experiment indicates that XTTS fine-tuning substantially improved speaker similarity for cross-lingual speaker transfer.The reported result concerns adaptation to the selected voices and recording conditions.
6. Conclusions and future work
XTTS achieves multilingual ZS-TTS state-of-the-art results across 16 languages and supports style adaptation from limited speech. The authors identify decoder replacement and speaker–prosody disentanglement as future directions.
- Conclusions: XTTS achieved state-of-the-art multilingual zero-shot multi-speaker TTS results in 16 languages.The conclusion presents this as the paper’s main result.
- Conclusions: With 10 minutes of whispering English speech, XTTS could mimic whispering voice style in all 16 languages.The authors also report strong prosody and style mimicking after fine-tuning with a small portion of speech.
- Conclusions: XTTS used a 21.53 Hz encoder frame rate compared with 75 Hz for VALL-E, making it faster than VALL-E.The conclusion attributes the speed difference to the encoder token frame rates.
- Future work: Future work targets generating speech with the VQ-VAE decoder and disentangling speaker and prosody information for cross-speaker prosody transfer.These are stated as intended improvements to the current system.