Source-linked AI summary
Sample Efficient Adaptive Text-to-Speech
Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie, Caglar Gulcehre, Aäron van den Oord, Oriol Vinyals, Nando de Freitas
TL;DR
The paper addresses the need to adapt TTS systems to new speakers without hours of speaker-specific recordings. It meta-trains a conditional multi-speaker WaveNet and benchmarks embedding optimization, full-model fine-tuning, and encoder-based embedding prediction. These methods achieve strong naturalness and voice similarity with only seconds or minutes of adaptation audio, while the study remains limited to clean, controlled data.
Problem
Conventional WaveNet-based TTS requires hours of training data for each speaker, motivating adaptation to new voices from much less audio.
Method
The paper trains a shared conditional WaveNet with speaker embeddings, then adapts new speakers by optimizing embeddings, fine-tuning the full model, or predicting embeddings with an auxiliary encoder.
Results
The proposed approaches achieve high-fidelity, voice-similar synthesis from seconds or minutes of audio; with a few minutes, the model matches state-of-the-art sample naturalness and outperforms recent work in voice matching.
Takeaways & Limitations
A meta-trained TTS prior can rapidly acquire new speaker voices, with full-model adaptation after speaker-embedding optimization producing the strongest reported performance.
Takeaways & Limitations
The study considers clean, high-quality training data collected in controlled environments and does not address few-shot adaptation from noisy data.
Abstract
from arXiv · showhide
We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.
1 INTRODUCTION
The paper reframes adaptive TTS as few-shot meta-learning: training learns a prior that rapidly adapts to new speakers rather than a fixed deployed model. It introduces WaveNet-based adaptation strategies that achieve high-fidelity, voice-similar speech from limited recordings.
- Motivation and framing: Few-shot meta-learning trains a neural-network prior that rapidly adapts to new speakers instead of producing a fixed final TTS model.The prior is combined with few deployment-time data to learn new speaker behaviors.
- Motivation and framing: The approach uses task-independent parameters for general behavior and task-dependent parameters learned rapidly for each new task.Task-dependent parameters play a role analogous to latent variables, providing flexibility across diverse speakers.
- WaveNet and the adaptation problem: WaveNet provides the shared model core, but conventional speaker adaptation requires hours of training data for each speaker.The paper develops a training procedure intended to synthesize new voices from no more than 10 minutes of data with high sample quality.
- Proposed strategies: The paper benchmarks embedding-only fine-tuning, end-to-end fine-tuning, and an auxiliary encoder that predicts new speaker embeddings.The first two approaches are non-parametric, while the encoder-based approach is parametric and text-independent.
- Results: All proposed approaches generate high-fidelity utterances resembling demonstration speakers, with strongest results when the entire model is fine-tuned end-to-end.The reported state-of-the-art results cover sample naturalness and voice similarity, including robustness across differently recorded speech datasets.
- Significance: Reducing the required recording duration is especially valuable where long, high-quality recordings are scarce, such as voice restoration for patients with voice-impairing conditions.The paper identifies this application as a motivating example rather than an evaluated deployment setting.
2 WAVENET ARCHITECTURE
The architecture models waveform generation autoregressively with a conditional WaveNet. It conditions generation on speaker embeddings, linguistic features, and pitch, while learning speaker embeddings alongside shared WaveNet parameters.
- Autoregressive waveform model: WaveNet factorizes the joint waveform distribution into timestep-wise conditional distributions using the probabilistic chain rule.Each waveform sample is modeled conditioned on preceding samples and the model inputs.
- Conditional inputs: The multi-speaker WaveNet conditions on speaker embedding e_s, phoneme-derived linguistic features l, and logarithmic fundamental frequency f0.The linguistic features encode phoneme sequences, while f0 controls pitch dynamics in the generated utterance.
- Meta-learning workflow: The system follows separate training, adaptation, and inference stages, with training producing a prior that is combined with few data during adaptation.Adaptation fine-tunes the components marked for updating before the adapted model is deployed.
- Speaker representation: Training learns a table of speaker embeddings together with the shared WaveNet parameters.These embeddings capture salient voice characteristics and support few-shot generalization to unseen speakers.
3 FEW-SHOT ADAPTATION WITH WAVENET
The paper develops few-shot WaveNet adaptation through non-parametric fine-tuning of speaker embeddings or all model parameters, plus a parametric encoder that predicts embeddings from demonstration data. These approaches are designed to adapt new voices rapidly while addressing overfitting and adaptation-time computation.
- 3 FEW-SHOT ADAPTATION WITH WAVENET: Few-shot adaptation uses either fine-tuning or an auxiliary encoder to obtain speaker embeddings for new voices.The non-parametric methods adapt embeddings or all model parameters, while SEA-ENC predicts an embedding from demonstration data.
- 3 FEW-SHOT ADAPTATION WITH WAVENET: The WaveNet is first pre-trained on multiple speakers, then fine-tuned using held-aside adaptation data for a new speaker.This procedure combines a multi-speaker conditional model with speaker-specific adaptation.
- 3.1 NON-PARAMETRIC FEW-SHOT ADAPTATION VIA FINE-TUNING: SEA-EMB1 optimizes only a new speaker embedding while keeping the shared WaveNet core fixed.The embedding is initialized randomly and optimized so demonstration waveforms are likely under the fixed-core model.
- 3.1 NON-PARAMETRIC FEW-SHOT ADAPTATION VIA FINE-TUNING: SEA-ALL additionally fine-tunes the full model, but its larger parameter set can overfit adaptation data.The method therefore uses 10% of demonstration data for early termination, whereas SEA-EMB can converge with seconds of adaptation data.
- 3.2 PARAMETRIC FEW-SHOT ADAPTATION USING AN EMBEDDING ENCODER: SEA-ENC predicts a new speaker embedding from demonstration data using an auxiliary encoder trained jointly with the WaveNet.It can operate transcript-independently from the input waveform and requires negligible computation during adaptation, but may introduce embedding bias.
- 3 FEW-SHOT ADAPTATION WITH WAVENET: Speaker-dependent linguistic and fundamental-frequency information is normalized so speaker identity is modeled primarily through the global speaker embedding.The stated goal is to make input features as speaker-independent as possible.
4 RELATED WORK
Related work frames few-shot learning as rapid adaptation from limited data, spanning meta-learning, generative modeling, and neural TTS. The paper focuses on adapting autoregressive WaveNet to unseen speakers, where few-seconds adaptation remains challenging.
- FEW-SHOT LEARNING AND META-LEARNING: Few-shot learning seeks models that learn rapidly from small amounts of data, with prior applications in vision, language, and speech recognition.Meta-learning provides a framework in which an expensive training process produces systems able to learn quickly from few data.
- FEW-SHOT LEARNING AND META-LEARNING: Optimization-learning methods and MAML adapt learned base parameters to new task losses using a fixed optimizer and a few gradient steps.The cited work applies these ideas across few-shot learning settings, including robotics.
- GENERATIVE MODELING: Generative few-shot learning has used matching networks, variable inference, sequential generative models, and attention-augmented PixelCNN.These approaches address few-shot generation and density modeling from different architectural perspectives.
- RELATED ADAPTATION APPROACHES: The authors report that attention models and MAML failed to learn informative speaker embeddings in preliminary experiments.This motivates the paper’s focus on a different adaptation strategy for speaker voice modeling.
- NEURAL TEXT-TO-SPEECH: The study extends autoregressive WaveNet to adapt speakers absent from training, amid broader efforts toward end-to-end neural TTS.Other systems such as Tacotron 2 use WaveNet as a vocoder, while the present work targets few-shot adaptation of the WaveNet model itself.
- NEURAL TEXT-TO-SPEECH: Producing high-quality speech for unseen speakers from only a few seconds of audio remains a challenge for neural TTS systems.Concurrent work includes memory-based and auxiliary fitting approaches for few-shot voice-style adaptation.
5 EVALUATION
The evaluation tests sample naturalness, voice similarity, speaker verification, and adaptation efficiency across LibriSpeech and VCTK. SEA-ALL generally performs best, achieving strong naturalness and similarity with limited adaptation data, while verification results indicate realistic speaker characteristics.
- Experimental setup: The evaluation combines naturalness MOS, subjective voice-similarity MOS, speaker-verification analysis, and adaptation-time measurements.Experiments use LibriSpeech and VCTK held-out datasets, with varying adaptation-data sizes.
- Naturalness of the generated samples (MOS): 4.13 MOS is achieved by the best LibriSpeech few-shot model using only 5 minutes of adaptation data.This approaches the 4.21 MOS reported for WaveNet trained on 24 hours of production-quality speech.
- Naturalness of the generated samples (MOS): SEA-ALL outperforms SEA-EMB and SEA-ENC for naturalness, with additional adaptation data helping LibriSpeech but not VCTK.The gap from real utterances is wider on VCTK, possibly because of different recording conditions.
- Voice similarity (MOS): SEA-ALL achieves the strongest subjective voice similarity, reaching an average score of 3.97 on VCTK.Its advantage over SEA-EMB grows with more adaptation data, while generated voices remain distinguishable from real utterances to human listeners.
- Voice similarity (speaker verification): SEA-ALL produces clear speaker clusters with large inter-cluster distance and low intra-cluster separation in d-vector projections.The generated and real samples overlap, suggesting both speaker identity preservation and difficulty distinguishing synthetic from real speech.
- Voice similarity (speaker verification): SEA-ALL outperforms the other approaches in verification, with error rates decreasing as demonstration data increases.On LibriSpeech with sufficient adaptation data, SEA-ALL’s EER is even lower than that of real utterances; in an adversarial test on VCTK, its AUC is 0.56.
6 CONCLUSION
The paper studies three meta-learning variants for sample-efficient adaptive TTS. Fine-tuning the entire model after optimizing the speaker embedding performs strongly with very little audio, while the study remains scoped to clean, controlled data and raises misuse concerns.
- Three meta-learning variants are studied for sample-efficient adaptive TTS.
- 10 seconds of audio can yield impressive performance when the entire model is fine-tuned after optimizing the speaker embedding.
- A few minutes of adaptation data match state-of-the-art sample naturalness and outperform recent work in matching new speakers’ voices.
- The study considers clean, high-quality recordings collected in controlled environments, leaving noisy-data few-shot learning open.
- Reducing adaptation data increases both beneficial and harmful applications of TTS, including synthesized media creation.
A LINGUISTIC FEATURES AND FUNDAMENTAL FREQUENCY INPUTS
The linguistic-feature pipeline derives phoneme and prosodic inputs from text and waveform data for WaveNet conditioning. At inference, duration and fundamental-frequency inputs are predicted rather than extracted from natural speech.
- The pipeline generates linguistic features and fundamental frequency from input text and waveform data.
- A text-analysis front end predicts phoneme sequences using tokenization, tagging, parsing, normalization, a lexicon, and grapheme-to-phoneme rules.
- Training and adaptation extract fundamental frequency with YIN and phone durations through forced alignment of text-audio pairs.
- At inference, an LSTM-based predictor generates phone durations and fundamental frequency from the input.
- The linguistic features differ from the referenced formulation by omitting ToBI end-tone marking and adding dependency-parser-derived categorical features.
B EMBEDDING ENCODER
The speaker-embedding encoder combines a pretrained speaker-verification pathway with a convolutional pathway. Their outputs are projected into the speaker-embedding space and summed.
- The encoder sums outputs from a pretrained TI-SV pathway and a 16-layer one-dimensional convolutional pathway.
- TI-SV maps variable-length waveforms to 256-dimensional d-vectors, while the convolutional pathway extracts residual speaker information.
- A shallow MLP and temporal averaging project the two pathway outputs into the speaker-embedding space.
C DET CURVES VARYING TRAINING DATA SIZES
The appendix provides DET curves for speaker verification across models trained with different data sizes. The curves use generated utterances from LibriSpeech and VCTK, with equal error rates marked.
- DET curves compare speaker-verification models across different training-data sizes.
- The plotted utterances use 1 minute or 10 seconds of data from LibriSpeech and VCTK, and EER is marked with a dot.
D ROC CURVES VARYING TRAINING DATA SIZES
The section presents ROC-curve analyses for speaker verification and real-versus-generated utterance detection across adaptation settings with different training-data sizes.
- ROC curves compare speaker verification performance for adversarial examples from adaptation models trained with different data sizes.
- Figure 10 evaluates real-versus-generated utterance detection using models adapted with either 1 minute or 10 seconds of LibriSpeech and VCTK audio.
- A lower ROC curve indicates that real and generated samples are harder to distinguish.