Source-linked AI summary
Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers
Zhaoyu Liu, Brian Mak
TL;DR
The paper addresses cross-lingual multi-speaker TTS for native and accented English and Mandarin speech from seen and unseen speakers. It combines x-vector speaker encoding, shared phoneme and language conditioning, stress-tone embeddings, Tacotron synthesis, and WaveNet vocoding. The system produces high-quality native speech across speaker settings, while normalization improves many outputs and Cantonese-trained WaveNet synthesizes English and Mandarin well.
Problem
The paper investigates cross-lingual multi-speaker TTS for native or accented English and Mandarin speech from native or foreign seen and unseen speakers.
Source: §1 · p. 1Method
The system combines x-vector speaker embeddings, shared phoneme and language embeddings, stress-tone embeddings, a Tacotron-based synthesizer, and a WaveNet vocoder.
Source: §1 · p. 2, §2.4 · p. 2Results
The method generates high-quality natural and intelligible native speech for native and foreign seen and unseen speakers, while accented speech is less intelligible and natural.
Source: §3.2 · p. 4, §4 · p. 4Takeaways & Limitations
L2-norm or whitening normalization improves output quality in many cases, and a WaveNet trained on Cantonese can synthesize Mandarin and English well.
Source: §2.4 · p. 2, §3.2 · p. 4, §4 · p. 4
Abstract
from arXiv · showhide
We investigate a novel cross-lingual multi-speaker text-to-speech synthesis approach for generating high-quality native or accented speech for native/foreign seen/unseen speakers in English and Mandarin. The system consists of three separately trained components: an x-vector speaker encoder, a Tacotron-based synthesizer and a WaveNet vocoder. It is conditioned on 3 kinds of embeddings: (1) speaker embedding so that the system can be trained with speech from many speakers will little data from each speaker; (2) language embedding with shared phoneme inputs; (3) stress and tone embedding which improves naturalness of synthesized speech, especially for a tonal language like Mandarin. By adjusting the various embeddings, MOS results show that our method can generate high-quality natural and intelligible native speech for native/foreign seen/unseen speakers. Intelligibility and naturalness of accented speech is low as expected. Speaker similarity is good for native speech from native speakers. Interestingly, speaker similarity is also good for accented speech from foreign speakers. We also find that normalizing speaker embedding x-vectors by L2-norm normalization or whitening improves output quality a lot in many cases, and the WaveNet performance seems to be language-independent: our WaveNet is trained with Cantonese speech and can be used to generate Mandarin and English speech very well.
1. INTRODUCTION
The paper targets cross-lingual multi-speaker TTS that can produce native or accented English and Mandarin for seen and unseen speakers. It uses shared phonemes, speaker/language/stress-tone conditioning, and investigates speaker-vector normalization and cross-language WaveNet training.
1. INTRODUCTION
- The approach aims to synthesize high-quality native or accented speech for native or foreign speakers who may be seen or unseen during training.Source: §1 · p. 1
- The paper uses publicly available English and Mandarin corpora, making its results reproducible.Source: §1 · p. 1
- Training uses hundreds of speakers with less than 25 minutes each and evaluates unseen speakers, unlike [17], which uses few speakers with tens of hours each.Source: §1 · p. 2
- The system uses shared phonemes with speaker, language, stress, and tone embeddings, employing x-vectors instead of [17]'s d-vectors.Source: §1 · p. 2
- The model is simpler than alternatives because it omits residual encoding and adversarial training while testing normalization of speaker embeddings to improve intelligibility, naturalness, and similarity.Source: §1 · p. 2
- The study also tests whether a WaveNet trained with Cantonese can synthesize the intended English and Mandarin speech.Source: §1 · p. 2
2. MODEL STRUCTURE
The model combines shared phoneme inputs with explicit tone, stress, language, and speaker representations, followed by a Tacotron-based spectrogram synthesizer and WaveNet vocoder.
2. MODEL STRUCTURE
- The architecture uses a multi-lingual, multi-speaker TTS system conditioned on speaker, language, and tone/stress embeddings.Source: §2 · p. 2
2.1. Input Representation: Phoneme, Tone, Stress Embeddings
- A shared phoneme set maps pinyin to ARPABET, while tone and stress use separate 7-D one-hot embeddings concatenated with phoneme embeddings.Source: §2.1 · p. 2
2.2. Speaker Encoder
- The speaker encoder extracts fixed-dimensional x-vectors from the sixth hidden layer of a TDNN-based speaker-discriminative model.Source: §2.2 · p. 2
2.3. Mel-spectrogram Synthesizer
- The mel-spectrogram synthesizer is based on multi-speaker Tacotron 2 and concatenates speaker and language embeddings with encoder context.Source: §2.3 · p. 2
2.4. WaveNet
- The WaveNet vocoder uses 30 dilated causal-convolution layers and is trained on Cantonese while synthesizing English and Mandarin speech.Source: §2.4 · p. 2
3. EXPERIMENTS & RESULTS
Experiments evaluate speaker embeddings and multilingual TTS quality across English and Mandarin, including seen and unseen speakers. Normalized x-vectors improve unseen-speaker intelligibility and naturalness, while accented speech remains weaker and English scores exceed Mandarin.
3. EXPERIMENTS & RESULTS
- Training uses open-source English, Mandarin, and Cantonese corpora, with Librispeech and SurfingTech for the synthesizer and CUSENT for WaveNet.The x-vector encoder uses all four datasets, while the data are distributed across model components.Source: §3 · pp. 2–3
3.1. Speaker Verification Evaluation
- Speaker verification compares 400-D i-vectors and 64/128/512-D x-vectors, but better SV EER does not guarantee better synthesized audio; 128-D x-vectors are selected.Adding speakers from other datasets does not further reduce SV EER, and normalization is then applied to x-vectors.Source: §3.1 · p. 3
3.2. Subjective Evaluation
- The evaluation trains an English baseline and three multilingual systems using unnormalized, L2-normalized, or whitened x-vectors, with MOS ratings for intelligibility, naturalness, and speaker similarity.The multilingual evaluation includes native and accented English and Mandarin from seen and unseen speakers.Source: §3.2 · pp. 3–4
- The baseline achieves high naturalness for seen and unseen speakers, with speaker similarity ranging from moderately similar to very similar.Ground-truth ratings vary with dataset accents, familiarity, emotion, and speaker style.Source: §3.2 · p. 4
- Both L2-norm and whitening normalization produce unseen-speaker speech rated between good and extremely intelligible, unlike unnormalized foreign-native speech.Unnormalized FNE and FNM speech is rated bad/poor, while normalization improves clarity and stopping behavior.Source: §3.2 · p. 4
- Whitening improves naturalness for native English and foreign-native Mandarin, whereas foreign-accented speech remains poor despite improvement from L2-norm normalization.These comparisons come from the naturalness MOS evaluation.Source: §3.2 · p. 4
- Across intelligibility, naturalness, and speaker similarity, synthesized English receives better MOS results than synthesized Mandarin.The authors suggest differences in corpus quality and rater linguistic background as plausible explanations.Source: §3.2 · p. 4
4. CONCLUSION
The proposed multilingual multi-speaker TTS system matches monolingual baselines and produces natural native English and Mandarin with whitened x-vectors. It also supports intelligible cross-language voice cloning and indicates that WaveNet performance may be language-independent.
4. CONCLUSION
- The multilingual multi-speaker TTS approach performs no worse than monolingual baselines.Source: §4 · p. 4
- With x-vector whitening, the system produces very natural native English and Mandarin, with moderate speaker similarity for unseen speakers.Source: §4 · p. 4
- The model clones foreign speakers into another language with intelligible and natural native-like speech, while also generating accented speech for foreign speakers.The conclusion specifically identifies FNE and FNM results for cross-language native-like speech.Source: §4 · p. 4
- A WaveNet trained on Cantonese generates high-quality Mandarin and English speech, suggesting language-independent vocoder performance.Source: §4 · p. 4