PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features

Florian Lux, Ngoc Thang Vu

arXiv:2203.03191v1cs.CLcs.LGeess.AS

TL;DR

Modern TTS is constrained by limited training data across many languages, and transferring across languages remains challenging. The paper combines articulatory phoneme representations with language-agnostic meta-learning; it reports low-resource fine-tuning with 30 minutes of paired data and adaptation to unseen languages. Its findings are bounded by evaluation across languages with similar phoneme inventories.

  • Problem

    Most modern TTS methods are limited to resource-rich languages, while cross-lingual use of data remains a challenge.

    Source: §1 · p. 1

  • Method

    The paper combines articulatory phoneme representations with a modified MAML procedure that updates shared parameters using batches from different language tasks.

    Source: §1 · p. 1, §3.3 · p. 4

  • Results

    The approach supports fine-tuning a TTS model with 30 minutes of paired data and enables zero-shot adaptation to unseen languages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Articulatory inputs support multilingual knowledge sharing without language-specific projection spaces and also benefit single-language training through phoneme knowledge sharing.

    Source: §6 · p. 8

    The findings investigate transfer only between languages with similar phoneme inventories; languages using lexical tone may require pretraining on more closely related high-resource languages.

    Source: §5 · p. 8

Abstract

from arXiv · show

While neural text-to-speech systems perform remarkably well in high-resource scenarios, they cannot be applied to the majority of the over 6,000 spoken languages in the world due to a lack of appropriate training data. In this work, we use embeddings derived from articulatory vectors rather than embeddings derived from phoneme identities to learn phoneme representations that hold across languages. In conjunction with language agnostic meta learning, this enables us to fine-tune a high-quality text-to-speech model on just 30 minutes of data in a previously unseen language spoken by a previously unseen speaker.

1 Introduction

The paper targets low-resource TTS, where most languages lack the data used by modern systems. It combines articulatory input features with meta-learning to transfer across languages and fine-tune with little data.

1 Introduction

  • Modern TTS performs well with enough data, but cross-lingual data use remains challenging and most methods are limited to resource-rich languages.The passage notes that over 6,000 languages are not resource-rich.Source: §1 · p. 1
  • Earlier articulatory-feature work addressed code-switching, while this paper extends articulatory inputs with meta-learning for low-resource TTS.The cited code-switching work is limited to languages with similar phoneme types; another low-resource approach used supplementary linguistic features and proprietary data and models.Source: §1 · pp. 1–2
  • The proposed approach combines articulatory phoneme features and a modified MAML procedure to support low-resource TTS, including fine-tuning with 30 minutes of paired data.The authors also report that it enables zero-shot adaptation to unseen languages.Source: §1 · pp. 1–2
  • The authors report benefits from articulatory inputs over phoneme identities, meta-parameters for low-resource fine-tuning, and generalization to unseen phonemes.They also describe insights into data and training-time needs across languages and speakers.Source: §1 · p. 2

2 Background and Related Work

Input representations differ in how well they transfer across languages and handle unseen units. MAML seeks parameters that adapt across tasks, including unseen ones, but standard training incurs costly second-order derivatives.

2.1 Input Representations

  • Phoneme inputs align more closely with audio than raw graphemes, but language-specific phoneme sets can leave unseen units at inference or fine-tuning.IPA is one standard phonetic representation discussed; the passage also notes data-driven grapheme-to-phoneme conversion across over 500 languages.Source: §2.1 · p. 2
  • Latent multilingual representations require training data in every language where they will be applied, ruling out zero-shot settings.Source: §2.1 · p. 2
  • Articulatory descriptions specify phonemes through features such as tongue position and voicing, supporting production of unseen phonemes from their descriptions.The paper extends this representation to cross-language transfer with meta-learning.Source: §2.1 · p. 2

2.2 Model Agnostic Meta Learning (MAML)

  • MAML trains task-specific copies of a shared Meta Model, then backpropagates summed task losses through the inner loop to update that model.Unrolling the inner loop entails costly second-order derivatives.Source: §2.2 · p. 3
  • First-order MAML reduces computational cost by applying the task-specific model’s final inner-loop gradient directly to the Meta Model.The passage also notes other variants intended to address second-order derivative costs.Source: §2.2 · p. 3

3 Approach

The system uses a speaker- and language-unconditioned TTS model with articulatory inputs, and describes components intended to remain exchangeable across architectures. Its language-agnostic meta-learning procedure updates shared parameters from language-specific batches to stabilize training.

3.1 System Description

  • The system uses an unconditioned single-speaker model for adaptation to a new speaker and language, and its pipeline blocks can be exchanged for methods with matching interfaces.The authors found speaker- and language-conditioned models more sensitive to hyperparameter choice.Source: §3.1 · p. 3, Figure 1 · p. 3
  • FastSpeech 2 requires phoneme-duration annotations for fine-tuning data, generated by applying Viterbi decoding to Tacotron 2 attention maps for improved alignments.The model uses duration-averaged pitch and energy labels, making alignment quality important to avoid downstream prosody errors.Source: §3.1 · p. 3

3.2 Articulatory Vectors

  • PanPhon vectors encode one feature per dimension with values of -1, 0, or 1, including phonological features such as syllabicity.Source: §3.2 · p. 4
  • Papercup features use one-hot vectors for articulatory descriptions, leaving the network to learn dependencies among features such as openness and frontness.Source: §3.2 · p. 4

3.3 Language Agnostic Meta Learning

  • Standard MAML is unstable for TTS because task-specific adaptation may require hundreds of updates, while language-and-speaker tasks make adaptation toward one speaker’s characteristics difficult to balance.The passage attributes the challenge partly to TTS being a one-to-many mapping with spectrogram distance as an imperfect objective.Source: §3.3 · p. 4
  • LAML sums losses from one batch per language and updates the Meta Model directly with Adam, stabilizing learning while retaining updates toward a more universal configuration.The authors describe this as a simplification of MAML for languages treated as tasks.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate articulatory-feature embeddings in single-language TTS and combine them with LAML for cross-lingual fine-tuning, using automatic measures and human evaluation. The datasets include English, German, and six other languages.

4.1.1 Embedding Function Design

  • Concatenating PanPhon and Papercup features and adding a nonlinearity improved embedding-space matching; the chosen function projects through 100 dimensions with tanh to 512 dimensions.Source: §4.1.1 · p. 5, Table 1 · p. 5

4.1.2 Convergence Time

  • Articulatory-feature models reached diagonal attention by 500 steps, versus 2,000 for lookup embeddings, but subjective quality was indistinguishable after 80,000 steps.The earlier alignment may help low-resource training by allowing it to stop before overfitting becomes problematic.Source: §4.1.2 · p. 5, Figure 2 · p. 5

4.2 Cross-Lingual Experiments

  • A model fine-tuned on 30 minutes of Karlsson data matched or outperformed the 29-hour baseline on multiple metrics after pretraining on speech from eight languages.The pretraining used one speaker per language and included Tacotron 2 and FastSpeech 2 models.Source: §4.2 · p. 5
  • The lookup-table meta-checkpoint did not converge, while English single-speaker models fine-tuned on 30 minutes of German overfit and collapsed; the authors conclude both LAML and articulatory features are needed.Source: §4.2 · pp. 5–6

4.2.1 Intelligibility

  • WER was slightly lower for both low-resource systems than their baselines on 100 German radio-news texts; low-resource gains on Russian-city names were attributed to retained Russian pretraining.WER was measured using the IMS-Speech ASR.Source: §4.2.1 · p. 6, Table 2 · p. 6

4.2.2 Naturalness

  • In the 34-speaker preference study, low-resource Tacotron 2 was preferred in over half of cases; FastSpeech 2 was perceived as good or better in 56%.Source: §4.2.2 · p. 6, Figure 3 · p. 7

5 Further Analysis and Future Work

The analysis examines adaptation speed, zero-shot phoneme handling, and language–speaker interactions, while identifying data and language-inventory boundaries for the findings.

What is the ideal amount of training steps for fine-tuning?

  • Speaker-embedding similarity tracks adaptation toward the target speaker; adding it as an objective only marginally reduces required steps while greatly increasing training time.The embedding follows the ECAPA-TDNN architecture and is trained on VoxCeleb 1 and 2, which the authors say do not overlap with their other data.Source: §5 · p. 7, Figure 4 · p. 7

Can this setup handle zero-shot phonemes?

  • Unseen phonemes have meaningful positions in the learned representations, supporting generalization beyond the phonemes seen during training.The authors report that these pronunciations can be understood in longer sequences before adaptation improves them.Source: §5 · p. 7, Figure 5 · p. 8
  • Under 100 adaptation steps, the pronunciation of unseen phonemes rapidly matches the correct pronunciation, although initial production is imperfect.Their positions in the learned representation are meaningful rather than collapsed onto the most similar seen phoneme.Source: §5 · p. 7

Does this setup learn the difference between language and speaker?

  • An unseen Hungarian text synthesized by the fine-tuned model resembles the Hungarian female speaker, prompting a hypothesis that input phonemes prime speaker-dependent parameters.The authors suggest that investigating parameter-group interactions could reduce the number of parameters trained and, in turn, the data needed.Source: §5 · pp. 7–8

How can we bring down FastSpeech 2’s data need further?

  • FastSpeech 2 benefits more from additional data and training time; its nearly twice-as-high parameter count may explain this, and fixing some parameters could help.Source: §5 · p. 8

Does this work across language families?

  • The experiments cover transfers between languages with similar phoneme inventories; the authors suggest lexical-tone targets may require pretraining on more closely related high-resource languages.They nevertheless believe the technique and analysis may hold across language families and types.Source: §5 · p. 8

6 Conclusion

The paper presents articulatory features and a modified meta-learning procedure for low-resource TTS. Its conclusions include faster single-language training and zero-shot inference on unseen phonemes.

6 Conclusion

  • The approach uses articulatory features and language-agnostic meta-learning to train a model for a language with only 30 minutes of data.Source: §6 · p. 8

Articulatory Features for TTS

  • Articulatory features support multilingual input without language-specific projection spaces and help single-language TTS converge earlier through phoneme knowledge sharing.Source: §6 · p. 8

MAML on TTS

  • Removing MAML’s inner loop yields a simpler procedure for pretraining a TTS checkpoint capable of low-resource fine-tuning.Source: §6 · p. 8

Zero-shot capabilities

  • Articulatory features enable zero-shot inference on unseen phonemes, further enhanced by LAML; the authors suggest the combination may reduce codeswitching to token-wise language identification.Source: §6 · p. 8

Related papers

Loading 2203.03191v1…