Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement

• Low-resource TTS training can be unstable, while conventional pre-training and fine-tuning complicate convergence and speaker adaptation.Source: §1 · p. 1
• Monolingual speaker-adaptation methods typically use language-specific text encoders, limiting transfer to new languages and sometimes requiring pre-trained ASR models.Source: §1 · p. 1
• The paper develops an IPA-input Tacotron framework that accepts multiple languages through a trainable lookup table without separate language encoders or switching techniques.Source: §1 · p. 1
• The approach targets low-resource adaptation to new speakers from the same or a different language.Source: §1 · p. 1
• It also evaluates whether source speakers can speak the target language intelligibly and whether target speakers can speak the source language.Source: §1 · p. 1

2.1. Model Architecture

• The model is an attention-based Tacotron sequence-to-sequence system with text encoding, attention, speaker embeddings, and phoneme embeddings.Source: §2.1 · p. 2
• A residual connection from the input layer to the encoder outputs improves training and inference robustness and speeds monolingual transfer convergence.Source: §2.1 · p. 2
• Text is normalized into IPA, mapped through a fixed universal embedding table, encoded by convolutional and bidirectional-LSTM layers, and decoded into Mel-spectrogram frames.Source: §2.1 · p. 2
• The universal IPA table lets speakers use language-specific subsets while focusing adaptation on characters absent from the source language.Source: §2.1 · p. 2

2.1.1. Training and Inference

• Training maximizes conditional speech likelihood over transcript, speaker identity, encoder, decoder, phoneme-lookup, and speaker-lookup parameters, using the Tacotron loss in both stages.Source: §2.1.1 · p. 2
• Expert Alignment performs two-step cross-lingual inference by computing alignment with a target-language expert speaker before using it for a different-language speaker.Source: §2.1.1 · p. 2

3. TRANSFER LEARNING SCENARIOS

• The transfer setup preserves the source speaker while adding target speakers, avoiding source-speaker forgetting during fine-tuning.The experiments use 20 minutes of data per speaker.Source: §3 · p. 2

3.1. Mono-Lingual Speaker Adaptation

• Monolingual adaptation assumes the text encoder already produces suitable representations and keeps it frozen for stability with noisy data.Freezing and unfreezing the encoder and phoneme table produced almost the same performance, but the encoder was retained frozen.Source: §3.1 · pp. 2–3
• The monolingual objective retains old and new speaker identities while updating the non-frozen transfer parameters.Source: §3.1 · p. 3

3.2. Cross-Lingual Speaker Adaptation

• Cross-lingual adaptation avoids freezing the text encoder because target-language phoneme combinations can differ from the source language.Source: §3.2 · p. 3
• Old-speaker embedding vectors remain frozen in cross-lingual transfer, while new-speaker embeddings adapt to the TTS model.Source: §3.2 · p. 3

4. EXPERIMENT SETUP

• The study evaluates transferability across English, German, Spanish, French, and Korean datasets, using both internal and publicly available corpora.Source: §4 · p. 3, §4.1 · p. 3

4.1. Datasets

• Transfer data include two random VCTK speakers and 20-minute English and German speech samples, including a noisy imperfect-alignment English dataset.Source: §4.1 · p. 3

4.2. Baseline Model

• The baseline extends NVIDIA’s Tacotron 2 implementation with a residual connection and uses eSpeak NG for multilingual text-to-phoneme conversion.Source: §4.2 · p. 3

4.3. Evaluation Metrics

• Objective evaluation measures pronunciation using ASR-derived WER and MER and evaluates acoustic similarity with MCD during transfer learning.Source: §4.3 · p. 3
• Subjective evaluation estimates MOS from 20 sentences per speaker in English and German using Amazon Mechanical Turk.117 workers participated, with 20 retained for German and 49 for English after gold-standard screening.Source: §4.3 · p. 3

5.1. Mono-Lingual Single Speaker Adaptation

• Monolingual transfer achieved reasonably good results after almost 3-4K iterations.Source: §5.1 · p. 4

5.2. Cross-Lingual Speaker Adaptation

• Cross-lingual experiments evaluated adaptation from English, Korean, and Spanish to four speakers in different target languages.Scores were computed for the target speaker in the target language.Source: §5.2 · p. 4
• Cross-lingual adaptation also tested source-speaker pronunciation on target-language sentences with low phonemization overlap.Figure 4(b) illustrates failed Spanish-speaker attention on an English sentence before adaptation and the attention after ES2EN adaptation.Source: §5.2 · p. 4, Figure 4 · p. 4

5.3. Cumulative Cross-Lingual Speaker Adaptation

• Cumulative language learning tested whether a model adapted from English to Spanish could subsequently learn third and fourth languages.Source: §5.3 · p. 4
• Cumulative models generally converged faster than the bilingual case, likely because they learned additional phoneme combinations and speech variations.Figure 3 visualizes expanded German IPA-character embeddings after adapting an ES2KO model to a German speaker.Source: §5.3 · p. 4, Figure 3 · p. 4

• The proposed Tacotron 2 modifications support low-resource cross-lingual speaker adaptation and cumulative generalization to more languages.Source: §6 · p. 4