Source-linked AI summary
Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning
Yu Zhang, Ron J. Weiss, Heiga Zen, Yonghui Wu, Zhifeng Chen, RJ Skerry-Ryan, Ye Jia, Andrew Rosenberg, Bhuvana Ramabhadran
TL;DR
Multilingual TTS must separate speaker identity from language when training data contains monolingual speakers and speaker identity is therefore correlated with language. The paper develops a Tacotron-based model using shared phonemic representations, adversarial disentanglement, speaker and language embeddings, and residual encoding. It synthesizes high-quality speech across three languages and transfers voices across languages, with naturalness above 3.85 MOS across language pairs in the full model, though Mandarin naturalness remains below ground truth and sparse-speaker cloning can fail.
Problem
Multilingual TTS lacks shared representations when languages differ and speaker identity is perfectly correlated with language in monolingual training data.
Method
The model combines phonemic inputs, speaker and language embeddings, residual autoencoding, and token-level adversarial training within a Tacotron-based multilingual TTS system.
Results
The full model achieves naturalness MOS above 3.85 across all language pairs and transfers voices across languages, while Mandarin naturalness remains below ground truth.
Takeaways & Limitations
The model can synthesize training speakers in multiple languages and control foreign accents moderately without bilingual or parallel training examples.
Takeaways & Limitations
In the single-speaker-per-language setting, cross-language cloning without adversarial loss failed for phoneme inputs and for English-to-Mandarin transfer.
Abstract
from arXiv · showhide
We present a multispeaker, multilingual text-to-speech (TTS) synthesis model based on Tacotron that is able to produce high quality speech in multiple languages. Moreover, the model is able to transfer voices across languages, e.g. synthesize fluent Spanish speech using an English speaker's voice, without training on any bilingual or parallel examples. Such transfer works across distantly related languages, e.g. English and Mandarin. Critical to achieving this result are: 1. using a phonemic input representation to encourage sharing of model capacity across languages, and 2. incorporating an adversarial loss term to encourage the model to disentangle its representation of speaker identity (which is perfectly correlated with language in the training data) from the speech content. Further scaling up the model by training on multiple speakers of each language, and incorporating an autoencoding input to help stabilize attention during training, results in a model which can be used to consistently synthesize intelligible speech for training speakers in all languages seen during training, and in native or foreign accents.
1. Introduction
Multilingual TTS is difficult when languages use incompatible representations and speaker identity is perfectly correlated with language. The proposed Tacotron-based model addresses this with shared representations and adversarial disentanglement, enabling cross-language voice transfer.
- Motivation: Multilingual TTS is challenging because language-dependent representations and imbalanced data limit sharing across unrelated languages.Mandarin and English have no overlap in text representation, while bilingual recordings are expensive to collect.
- Motivation: Most training speakers speak only one language, making speaker identity perfectly correlated with language.This correlation makes it difficult for the model to separate speaker characteristics from language-dependent speech content.
- Approach: The model uses a single-stage Tacotron-based architecture with no language-specific components and achieves naturalness comparable to monolingual baselines.Its design evaluates alternative input representations and supports cross-lingual voice cloning.
- Approach: A per-input-token speaker-adversarial loss enables cross-lingual voice transfer when only one training speaker is available per language.The loss discourages text representations from encoding speaker identity, addressing the speaker-language correlation.
- Results: The proposed model disentangles speakers from languages and consistently synthesizes high-quality speech for all speakers despite their original-language correlation.This result is reported as a component evaluation of the multilingual model.
2. Model Structure
The model extends Tacotron 2 with speaker and language conditioning, residual latent encoding, and adversarial training. It evaluates multilingual text representations while using phonemes and token-level adversarial loss to promote language- and speaker-independent text representations.
- Base architecture: Tacotron 2 generates log-mel spectrogram frames from text with attention-based sequence-to-sequence modeling.The multilingual architecture adds conditioning and latent components around this base model.
- Base architecture: Speaker and optional language embeddings condition synthesis, while an adversarial speaker classifier and residual encoder provide additional inputs.A separately trained WaveRNN neural vocoder converts the generated spectrograms into waveform audio.
- Input representations: The study compares character, phoneme, and UTF-8 byte representations for multilingual TTS.The comparison targets how input representation affects multilingual parameter sharing and pronunciation learning.
- Input representations: The UTF-8 representation uses 256 possible token values, but multilingual models must learn byte sequences for multi-byte characters such as Mandarin.This representation can promote sharing while introducing sequence-attention demands for languages with multi-byte characters.
- Input representations: Phoneme inputs use 88 shared symbols, simplifying pronunciation modeling and sharing equivalent phonemes across languages.Mandarin additionally uses tone embeddings, while English and Spanish use stress embeddings.
- Residual encoding: A variational autoencoder-like residual encoder represents latent factors in training audio that are not explained by text, speaker, or language inputs.The model uses a 16-dimensional latent space and feeds the prior mean during inference, which improves cross-lingual transfer stability and naturalness.
- Adversarial training: Domain-adversarial training discourages the encoded text sequence from capturing speaker information when speaker identity and language are correlated.A speaker classifier receives the text encoding through a gradient reversal layer.
- Adversarial training: The adversarial loss is applied to each encoded text element, with reversal-layer gradient clipping to limit unstable gradients from language-dependent tokens.An additional adversarial layer on the variational autoencoder had no effect after reducing the latent dimension.
3. Experiments
The experiments evaluate multilingual TTS using varied input representations, speaker coverage, adversarial disentanglement, and residual encoding. Results show that phoneme inputs, speaker-adversarial training, broader speaker coverage, and a residual encoder support natural, cross-language voice cloning, though accent and language distance remain important factors.
- Model and training setup: The dataset contains 385 hours of English from 84 speakers, 97 hours of Spanish from 3 speakers, and 68 hours of Mandarin from 5 speakers.The model uses high-quality speech from three languages, with substantially greater English speaker coverage.
- Evaluation: Speech naturalness is evaluated with crowdsourced MOS ratings on a 1–5 Absolute Category Rating scale, while cross-language cloning also measures speaker similarity.Similarity tests pair synthesized utterances with reference utterances from the same speaker, despite differing languages.
- Comparing input representations: Phoneme-based multilingual models achieve MOS scores above 4.0 in all languages, with Mandarin benefiting from phoneme inputs over character- and byte-based variants.Multilingual phoneme models match monolingual performance for Spanish and Mandarin but are slightly worse for English; multispeaker performance is similar to single-speaker-per-language performance.
- Cross-language voice cloning: Without speaker-adversarial loss, English-to-Mandarin cloning fails with phoneme inputs, while byte inputs clone English to Spanish with high similarity but significantly reduced naturalness.The unsuccessful Mandarin case was not formally tested because the speaker embedding did not affect the model output; character and byte inputs otherwise produced similar results.
- Cross-language voice cloning: Adding speaker-adversarial training enables English-to-Mandarin cloning with very high similarity MOS, but naturalness remains lower because of accent and pronunciation mismatches.The full model uses phoneme inputs in subsequent experiments because they guarantee correct pronunciations and more fluent speech.
- Cross-language voice cloning: With 84 English, 3 Spanish, and 5 Mandarin speakers, full-model naturalness exceeds 3.85 across language pairs, while Mandarin remains below ground-truth naturalness.English and Spanish results generally approach ground truth except for English-to-Spanish; English voice transfer to Spanish and Mandarin is reported as almost accent-free.
4. Conclusions
The model extends Tacotron 2 to multilingual TTS trained only on monolingual speakers, producing high-quality speech in three languages and transferring training voices across languages.
- The model synthesizes high-quality speech in three languages while transferring training voices across languages.It is trained only on monolingual speakers.
- The model supports moderate accent control when speaking foreign languages.
- The system has rudimentary support for code switching, demonstrated on the companion webpage.