Source-linked AI summary
Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation
Dongchan Min, Dong Bok Lee, Eunho Yang, Sung Ju Hwang
TL;DR
Personalized TTS needs to reproduce an unseen speaker’s voice from a few short audio samples, but existing approaches require fine-tuning or provide limited adaptation. StyleSpeech uses SALN for reference-conditioned synthesis, and Meta-StyleSpeech adds discriminators, style prototypes, and episodic training. The models are reported to generate high-quality, speaker-consistent speech and achieve strong performance across multi-speaker generation and one-shot adaptation tasks.
Problem
Personalized TTS must capture an unseen speaker’s voice from few short samples, while existing methods require fine-tuning or are not straightforwardly applicable to TTS.
Method
StyleSpeech uses SALN to condition text features on a reference-speech style vector, while Meta-StyleSpeech adds style and phoneme discriminators with style prototypes and episodic training.
Results
The proposed models achieve state-of-the-art TTS performance across multi-speaker generation and one-shot short-length speaker adaptation tasks.
Takeaways & Limitations
StyleSpeech and Meta-StyleSpeech can synthesize high-quality, expressive speech from a single short-duration target-speaker audio sample.
Abstract
from arXiv · showhide
With rapid progress in neural text-to-speech (TTS) models, personalized speech generation is now in high demand for many applications. For practical applicability, a TTS model should generate high-quality speech with only a few audio samples from the given speaker, that are also short in length. However, existing methods either require to fine-tune the model or achieve low adaptation quality without fine-tuning. In this work, we propose StyleSpeech, a new TTS model which not only synthesizes high-quality speech but also effectively adapts to new speakers. Specifically, we propose Style-Adaptive Layer Normalization (SALN) which aligns gain and bias of the text input according to the style extracted from a reference speech audio. With SALN, our model effectively synthesizes speech in the style of the target speaker even from single speech audio. Furthermore, to enhance StyleSpeech's adaptation to speech from new speakers, we extend it to Meta-StyleSpeech by introducing two discriminators trained with style prototypes, and performing episodic training. The experimental results show that our models generate high-quality speech which accurately follows the speaker's voice with single short-duration (1-3 sec) speech audio, significantly outperforming baselines.
1. Introduction
Personalized TTS must capture an unseen speaker’s voice from very little audio, but existing approaches require fine-tuning or do not adapt reliably. StyleSpeech uses SALN for reference-conditioned synthesis, while Meta-StyleSpeech adds discriminators and episodic training for unseen-speaker adaptation.
- Motivation: Personalized TTS seeks high-quality speech that captures a given speaker’s voice from only a few, potentially single, speech samples.The task is challenging because speech contains speaker identity, prosody, and other expressive factors.
- Existing approaches: Fine-tuning approaches require target-speaker transcripts, audio samples, and hundreds of optimization steps, limiting practical applicability.Reference-conditioned methods avoid fine-tuning by extracting a latent vector from speech, but the passage does not state their adaptation quality here.
- Research gap: Few-shot meta-learning has mainly targeted image domains, and existing methods are not straightforwardly applicable to TTS.This motivates adapting meta-learning ideas specifically to short-length speaker adaptation.
- Proposed models: StyleSpeech introduces SALN, which aligns text features’ gain and bias with a style vector extracted from a reference speech audio.The model is designed for high-quality, expressive, multi-speaker adaptive TTS.
- Proposed models: Meta-StyleSpeech extends StyleSpeech with phoneme and style discriminators trained using style prototypes and episodic meta-learning.Episodic training simulates one-shot adaptation, while the style discriminator encourages speaker-consistent style embeddings.
- Results: The proposed models achieve state-of-the-art TTS performance across multi-speaker generation and one-shot short-length speaker adaptation tasks.This is the paper’s stated overall result across multiple tasks.
2. Related Work
Neural TTS research progressed from autoregressive systems toward faster, more robust non-autoregressive models, while speaker adaptation commonly relied on fine-tuning or reference-derived style representations. Few-shot meta-learning largely focused on images, leaving short-length TTS adaptation relatively overlooked.
- Neural TTS: Autoregressive neural TTS models can suffer from slow inference and robustness problems such as word omission and skipping.WaveNet, Deep Voice, Char2Wav, and Tacotron are identified as examples of this generation approach.
- Neural TTS: Non-autoregressive models such as ParaNet and FastSpeech address these issues with faster inference and improved robustness over autoregressive models.The passage positions these systems as responses to limitations of earlier autoregressive generation.
- Speaker adaptation: Speaker adaptation methods commonly train on multi-speaker data and then fine-tune the whole model or selected parts for new speakers.The related-work passage describes fine-tuning as a major adaptation strategy.
- Speaker adaptation: Other speaker-adaptation methods model style directly from speech audio rather than relying only on model fine-tuning.The supplied passage introduces this alternative but is truncated before describing its methods in detail.
- Meta-learning: Metric-based meta-learning learns embeddings that bring same-class instances closer and separate instances from different classes.This approach is presented as part of the broader few-shot classification literature.
- Meta-learning: Existing few-shot generation methods largely focus on images, whereas this work addresses few-shot, short-length adaptation of TTS models.The passage characterizes short-length TTS adaptation as relatively overlooked.
3. StyleSpeech
StyleSpeech combines a mel-style encoder with a FastSpeech2-based generator, using SALN to condition synthesis on speaker style extracted from reference speech. It is trained with reconstruction losses to generate multi-speaker speech from phoneme input and short reference audio.
- Mel-style encoder: The mel-style encoder extracts a style vector w containing speaker identity and prosody from reference speech X.It processes mel-spectrogram frames with spectral layers, gated CNNs, frame-level self-attention, and temporal averaging.
- Mel-style encoder: Frame-level multi-head self-attention helps the mel-style encoder extract style information from short speech samples.The attention output is temporally averaged into a one-dimensional style vector.
- Generator: The generator combines phoneme input t with style vector w to synthesize a mel-spectrogram, using phoneme encoding, variance adaptation, and decoding.The variance adaptor predicts phoneme-level speech properties such as pitch and energy.
- Style-Adaptive Layer Norm: SALN predicts style-dependent gain and bias, replacing fixed LayerNorm parameters to adaptively scale and shift normalized features.SALN is applied in FFT blocks of the phoneme encoder and mel-spectrogram decoder.
- Training: Training minimizes reconstruction error between synthesized and ground-truth mel-spectrograms using an L1 loss.The reconstruction loss also includes pitch, energy, and duration losses as in FastSpeech2.
4. Meta-StyleSpeech
Meta-StyleSpeech extends StyleSpeech with style and phoneme discriminators and meta-learning to improve adaptation to unseen speakers from a single available speech sample. Its training alternates generator, encoder, and discriminator updates using reconstruction, adversarial, and classification losses.
- Motivation: Meta-StyleSpeech targets adaptation to unseen speakers when only a single speech audio sample is available.The method addresses shifted speaker distributions and short-reference adaptation challenges.
- Style discriminator: The style discriminator learns speaker-specific style prototypes and encourages generated speech to cluster around the target speaker’s prototype.The style vector and prototypes are compared through dot-product logits and a cross-entropy classification loss.
- Phoneme discriminator: The phoneme discriminator distinguishes real from generated speech while conditioning on the input phoneme sequence.It operates at frame level, pairing mel-spectrogram frames with corresponding phonemes before averaging frame scores.
- Objectives: The generator’s query-speech objective combines reconstruction and adversarial losses, while support-speech reconstruction empirically improves generated mel-spectrogram quality.The discriminator loss follows LS-GAN, replacing binary cross-entropy terms with least-squares losses.
- Episodic meta-learning: Meta-training alternates updates of the generator and mel-style encoder against updates of the style and phoneme discriminators.The respective updates minimize Lrecon and Ladv versus LDs, LDt, and Lcls losses.
5. Experiment
The experiments evaluate StyleSpeech and Meta-StyleSpeech on seen and unseen speakers using subjective, objective, adaptation, visualization, and ablation analyses. Meta-StyleSpeech generally provides the strongest speech quality, similarity, and unseen-speaker adaptation results.
- Evaluation on Trained Speakers: StyleSpeech and Meta-StyleSpeech outperform the baseline in MOS, MCD, and WER for seen speakers, with Meta-StyleSpeech achieving the best performance in the evaluated experiments.The models also achieve higher similarity to reference speech than other TTS baselines.
- Unseen Speaker Adaptation: Meta-StyleSpeech achieves the best MOS, MCD, and WER generation quality for unseen speakers, largely outperforming the baselines.The unseen-speaker evaluation uses one randomly selected reference audio per speaker.
- Unseen Speaker Adaptation: Meta-StyleSpeech significantly outperforms the baseline and StyleSpeech across reference-audio lengths, including speech shorter than 1 second.Generated speech from the shortest references has sharp harmonics and resolved formants comparable to the ground-truth mel-spectrogram.
- Ablation Study: Removing the text discriminator reduces both generated-speech quality and adaptation ability, while removing the style discriminator or style prototypes reduces unseen-speaker adaptation.The results support distinct roles for the text discriminator, style discriminator, and style prototypes during meta-training.
6. Conclusion
StyleSpeech generates expressive multi-speaker speech from a single short reference audio, while Meta-StyleSpeech improves adaptation to unseen speakers through discriminators, style prototypes, and meta-learning.
- StyleSpeech generates high-quality, expressive speech from a single short-duration audio sample of the target speaker.
- Meta-StyleSpeech extends StyleSpeech with additional discriminators, style prototypes, and episodic meta-learning to improve adaptation to unseen speakers.
- Meta-StyleSpeech achieves improved adaptation to unseen speakers even with a single reference speech shorter than one second.
A. Detailed model architectures
The detailed architecture comprises a mel-style encoder, generator, variance adaptor, and phoneme and style discriminators, with each component assigned specialized processing modules.
- StyleSpeech comprises a mel-style encoder and generator, while Figure 7 depicts the phoneme and style discriminators.
- The mel-style encoder uses spectral processing, temporal processing with residual gated 1D CNNs, and multi-head self-attention.
- The variance adaptor contains pitch, energy, and duration predictors based on FastSpeech2 architectures.
- The phoneme discriminator processes mel-spectrograms with phoneme embeddings, while the style discriminator has a similar architecture.
B. Meta-training Algorithm
Meta-training uses episodic speaker sampling with support samples and query texts, alternating updates to the encoder-generator pair and the discriminators with style prototypes.
- Each episode samples N speakers and one support sample plus one query text for each speaker.
- Step I updates the mel-style encoder and generator using adversarial and reconstruction objectives.
- Step II updates the phoneme discriminator, style discriminator, and style prototypes.
- The generator objective combines reconstruction loss weighted by 10 with adversarial loss.
- The discriminator objective combines phoneme-discriminator loss, style-discriminator loss, and classification loss.
C. MOS evaluation interface
Figure 8 presents the interface used to evaluate speech naturalness with a mean opinion score evaluation.
- Figure 8 shows the interface used for mean opinion score evaluation.
- The interface is designed for evaluating naturalness.
- The figure documents an evaluation interface rather than a model architecture.