End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning

• End-to-end TTS has been little studied for low-resource languages, where only very limited paired text-audio data are available.Source: §1 · p. 1
• Prior cross-lingual TTS research focused on MLMS statistical parametric systems, using language-independent representations, IPA, or shared phoneme inventories.The paper notes that cross-lingual information had not been widely studied for end-to-end TTS.Source: §1 · p. 1
• The proposed approach transfers from a high-resource source language using a learned mapping between linguistic symbols, with pronunciation information preserved during transfer.Objective and subjective tests report intelligible speech with few target-language pairs; the approach is competitive with handcrafted IPA mapping when both inputs are phonemes, and analytic studies find the mapping correlates with phonetic expertise.Source: §1 · p. 1

2. Proposed approach

• Source-model parameters can initialize target-model training, but differing source and target symbol sets prevent direct transfer of their symbol embeddings.The proposed approach addresses this input-space mismatch with two baselines and a learned mapping.Source: §2 · p. 2

2.1. Separate symbol space

• The Separate baseline finetunes the source model on the target language but learns target symbol embeddings from scratch.Source: §2.1 · p. 2

2.2. Unified symbol space

• The Unified baseline maps both languages to a shared symbol set, allowing source embeddings to initialize target embeddings, but requires expert-designed mappings that may be unavailable for phoneme-to-character pairs.Source: §2.2 · p. 2

2.3. Learned symbol space

• The learned-symbol approach uses a Phonetic Transformation Network (PTN) to automatically map source symbols to target symbols according to their sounds without linguistic expertise.Source: §2.3 · p. 2

2.3.1. Phonetic transformation network

• The PTN is trained after a source-language ASR system is pretrained and fixed, using target-language data and CTC loss to predict target symbol labels from source-symbol posteriors.Source: §2.3.1 · p. 2

2.3.2. Symbol mapping discovery

• A source symbol is mapped to a target symbol when its PTN output exceeds the transformation threshold; the corresponding source embedding is transferred, with highest-probability selection for collisions.Target symbols with no mapped source symbol have embeddings learned from scratch.Source: §2.3.2 · p. 2

3.1. TTS model

• The end-to-end TTS system uses the original Tacotron architecture and Griffin-Lim waveform synthesis, leaving other architectures for future work.Its spectral-analysis setting follows Tacotron.Source: §3.1 · pp. 2–3

3.2. ASR and PTN model

• The ASR system uses a modified pure-CNN model; a pyramidal RNN was tested but performed below expectations in preliminary studies.The PTN has three fully connected ReLU layers, each with dropout rate 0.4.Source: §3.2 · p. 3

• Speech clarity is evaluated with character error rate (CER) from Google’s speech-to-text API and mel-cepstral distortion (MCD), where smaller MCD is better.Naturalness is assessed with mean opinion score (MOS) tests.Source: §4 · p. 3
• The experiments distinguish phn2phn, with phonemes as input in both languages, from phn2char, with source phonemes and target characters; Separate, Unified, and Learned name the symbol-space approaches.Source: §4 · p. 3

4.1.1. Source language

• English is the high-resource source; LJ Speech provides around 24 hours of paired text and speech for TTS pretraining, while 100 hours of clean LibriSpeech training data support ASR training.The clean LibriSpeech development set is used for early stopping.Source: §4.1.1 · p. 3

4.1.2. Target language

• Mandarin, German, and French are the target languages, using an internal Mandarin corpus, German LibriVox data, and GlobalPhone data for French.The French data comprise approximately 18 minutes of paired speech and text.Source: §4.1.2 · p. 3
• The target-language data are split into training and testing sets, as illustrated in Table 14.Source: §4.1.2 · p. 3

• 10k source-language parameter updates produce the initial TTS model, which is used to initialize continued training on target-language paired data for every transfer method.Source: §4.2 · p. 3
• Separate randomly initializes target embeddings, while Unified maps symbols to IPA and reuses source embeddings for matching IPA representations.Unified randomly initializes target symbols without matching source IPA representations.Source: §4.2 · p. 3
• Learned uses a source-pretrained ASR model and automatically learned symbol mapping to initialize target embeddings as in Unified.The ASR model is pretrained for 300k updates; the transformation threshold ξ is 0.4 for all target languages.Source: §4.2 · p. 3

4.3.1. Objective tests

• Unified and Learned consistently outperform Separate on CER across languages and target-data amounts in phn2phn tests, while MCD results align with this pattern.As target data shrink, Separate degrades most; Learned remains competitive with Unified, and scratch training yields CER above 80% for every language even with all training data.Source: §4.3.1 · p. 3, Figure 2 · p. 4, Figure 3 · p. 4
• Learned shows a large CER gap over Separate for German and French in phn2char tests, where target inputs are characters and Unified is not applicable.Source: §4.3.1 · pp. 3–4, Figure 4 · p. 4
• The French MCD evaluation randomly selects training data and leaves the remainder for testing because the corpus is very small and MCD requires ground-truth audio.The procedure is repeated three times and the average score is reported.Source: §4.3.1 · p. 3

4.3.2. Subjective tests

• With 25 minutes of Mandarin paired data, Separate, Unified, and Learned perform almost the same and all outperform Scratch; with 15 minutes, Separate degrades while Learned remains comparable to Unified.The subjective tests use 40 raters and 80 unseen utterances, with at least five ratings per utterance.Source: §4.3.2 · p. 4

4.4. Symbol mapping studies

• Under phn2phn, the study assesses learned symbol mappings against IPA; Table 3 reports precision and recall on 15-minute target data.Source: §4.4 · p. 4, Table 3 · p. 4
• For Mandarin, phn2char is not considered reasonable because characters correspond to syllables rather than phonemes, so its performance is not reported.Source: §4.4 · p. 4
• The learned mappings are far better than random guessing, with stronger performance on German and French than on Mandarin.The authors suggest the difference may result from German and French being more similar to English, the source language.Source: §4.4 · p. 4, Table 3 · p. 4
• Some Mandarin mappings pair similar-sounding phonemes despite different IPA representations, such as <ù> and <S>.The paper notes that their pronunciations are quite alike and similar to English “sh.”Source: §4.4 · p. 4

5. Conclusion

• The proposed approach discovers cross-lingual symbol mappings to help transfer knowledge learned from abundant source-language data.Source: §5 · p. 4
• Experiments show that the method produces far more natural-sounding speech than training only on target-language data.Source: §5 · p. 4
• The method achieves promising results compared with an approach using strong linguistic background expertise.Source: §5 · p. 4