Source-linked AI summary
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu
TL;DR
The paper addresses the data burden of multispeaker TTS and the challenge of synthesizing voices unseen during training. It combines an independently pretrained speaker-verification encoder with a Tacotron 2-style synthesizer and WaveNet vocoder, conditioning speech generation on seconds of reference audio. The system transfers speaker characteristics to unseen voices, while larger encoder training sets improve adaptation and sampled embeddings produce fictitious speakers.
Problem
Multispeaker TTS requires extensive high-quality speech-transcript data, making data-efficient synthesis and zero-shot generation for unseen speakers difficult.
Method
The system independently trains a speaker-verification encoder on noisy untranscribed speech, a speaker-conditioned sequence-to-sequence synthesizer, and a WaveNet vocoder.
Results
The system synthesizes reasonably similar natural speech for seen and unseen speakers, with larger speaker-encoder training sets improving adaptation and sampled embeddings generating fictitious voices.
Takeaways & Limitations
Transfer learning separates speaker modeling from synthesis, reducing multispeaker TTS data requirements while enabling zero-shot voice generation from brief reference audio.
Takeaways & Limitations
The model does not attain human-level naturalness, cannot transfer accents, and cannot completely isolate speaker voice from reference prosody.
Abstract
from arXiv · showhide
We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of many different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech from thousands of speakers without transcripts, to generate a fixed-dimensional embedding vector from seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2, which generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a sequence of time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task, and is able to synthesize natural speech from speakers that were not seen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.
1 Introduction
The paper targets data-efficient zero-shot multispeaker TTS by separating speaker modeling from speech synthesis. It transfers speaker representations learned from large-scale speaker verification to synthesize voices, including unseen speakers, from brief untranscribed audio.
- Motivation: Few-shot zero-shot TTS uses seconds of untranscribed target-speaker audio to synthesize new speech without updating model parameters.The goal is natural speech for many speakers while avoiding extensive speaker-specific recordings and transcripts.
- Motivation: Large-scale multispeaker TTS is data-intensive because natural speech requires many high-quality speech-transcript pairs and typically tens of minutes per speaker.The proposed strategy decouples speaker modeling from synthesis so the components can use independent datasets.
- Results: The synthesis and encoder networks generalize across unbalanced and disjoint speaker sets, with 1.2K synthesis speakers and 18K encoder speakers improving adaptation quality.The larger encoder speaker set also enables synthesis of novel speakers by sampling from the embedding prior.
- Related work: Prior multispeaker systems commonly learn fixed embeddings for training speakers, limiting synthesis to voices seen during training.Other adaptation methods required substantial enrollment speech or model fine-tuning, whereas speaker encoders support more data-efficient adaptation.
- Approach: The speaker encoder is independently trained for speaker verification on noisy, untranscribed speech from tens of thousands of speakers, then transferred to TTS.Unlike jointly trained approaches, this setup explores transfer learning from a pretrained speaker-verification model.
2 Multispeaker speech synthesis model
The system independently combines a speaker encoder, Tacotron 2-style synthesizer, and WaveNet vocoder. At inference, a few seconds of arbitrary untranscribed reference speech condition synthesis for potentially unseen speakers, with speaker and prosodic characteristics reflected in outputs.
- Architecture: The model has three independently trained components: a recurrent speaker encoder, a sequence-to-sequence mel-spectrogram synthesizer, and an autoregressive WaveNet vocoder.The encoder produces a speaker vector, the synthesizer predicts mel spectrograms from text, and the vocoder produces time-domain waveforms.
- Speaker encoder: The speaker encoder maps speech spectrograms to a fixed-dimensional embedding trained with speaker verification so same-speaker utterances are similar and different-speaker utterances are separated.Training uses speaker identity labels but no transcripts; the embedding is intended to capture speaker characteristics from short, noisy, text-independent audio.
- Synthesizer: The synthesizer conditions its attention mechanism on the target speaker embedding while using frozen encoder parameters and no explicit speaker labels during TTS training.It is trained on transcript-audio pairs and predicts mel spectrograms from phoneme sequences.
- Neural vocoder: The WaveNet vocoder converts synthesized mel spectrograms into waveforms and is not directly conditioned on the speaker encoder.The synthesizer’s mel spectrogram is treated as containing the relevant voice detail for multispeaker waveform generation.
- Inference and zero-shot adaptation: A few seconds of arbitrary untranscribed reference audio can condition new text, even when the reference speaker is outside the training set.Figure 2 illustrates held-out male and female speakers, with output spectrograms reflecting pitch, formants, alignment, and some speaking-rate characteristics.
3 Experiments
Experiments evaluate naturalness, speaker similarity, verification-based voice matching, embedding geometry, and the effects of speaker diversity and dataset choice. The system produces natural speech for unseen speakers, but similarity is reduced by prosodic variation, accent mismatch, and limited synthesizer diversity.
- 3.1 Speech naturalness: About 4.0 MOS was achieved across datasets, with VCTK roughly 0.2 points higher than LibriSpeech for seen speakers.The LibriSpeech gap is associated with missing punctuation and greater background noise.
- 3.1 Speech naturalness: Unseen-speaker speech was rated at least as natural as seen-speaker speech, with LibriSpeech unseen-speaker MOS up to 0.2 points higher.The authors associate this difference with randomly selected references that sometimes contain uneven or non-neutral prosody.
- 3.2 Speaker similarity: 3.28 similarity MOS on unseen VCTK speakers indicates broad transfer of gender, pitch, and formant ranges, while characteristic prosody is partly lost.Similarity scores were lower for unseen speakers than for ground-truth speech comparisons.
- 3.2 Speaker similarity: The speaker encoder’s North American-only training data constrained VCTK similarity because accent mismatch could make matching voices appear different.Rater comments indicated similar tone and inflection despite some accent differences.
- 3.2 Speaker similarity: LibriSpeech-trained models generalized better across datasets, while training the synthesizer on only 100 speakers was insufficient for high-quality speaker transfer.Cross-dataset evaluation found LibriSpeech models synthesized VCTK speakers with higher similarity than VCTK models achieved for LibriSpeech speakers.
- 3.3 Speaker verification: 5-6% EERs were obtained for the LibriSpeech synthesizer using reference speakers from either dataset, whereas the VCTK synthesizer performed substantially worse, especially out of domain.An expanded 20-voice discrimination task produced a 2.86% EER, although synthetic utterances remained closer to same-speaker synthetic speech than to real speech.
- 3.4 Speaker embedding space: Embedding visualizations place synthetic utterances near real utterances from the same speaker, while keeping synthetic and real speech in distinct clusters; speakers also separate by gender.The geometry supports speaker separation and target-voice proximity, but also reveals a systematic real-versus-synthetic distinction.
- 3.4 Speaker embedding space: Speaker diversity improved adaptation: 18K encoder-training speakers improved quality over 1.2K synthesis speakers and enabled novel-speaker synthesis from sampled embeddings.Random points in the embedding space generated fictitious speakers absent from both component-network training and test sets.
4 Conclusion
The system uses independently trained speaker and synthesis components to generate multispeaker speech, including voices unseen during training. Transfer learning lowers multispeaker data requirements, but naturalness, accent transfer, and prosody separation remain limited.
- The system combines an independently trained speaker encoder, sequence-to-sequence synthesizer, and neural vocoder to produce speech for seen and unseen speakers.Evaluations used speaker verification and subjective listening tests to assess similarity to target speakers.
- Increasing speaker encoder training data improves speaker transfer when the synthesizer has sufficient speaker diversity.The encoder and synthesizer can use unbalanced and disjoint speaker sets; 18K encoder speakers improved adaptation over 1.2K synthesizer speakers.
- Separating encoder and synthesizer training reduces requirements for multispeaker TTS data and avoids speaker labels, clean speech, and transcripts across the respective datasets.Independent training also removes the need for additional triplet or contrastive losses in the synthesizer configuration.
- The model can generate realistic speech from fictitious speakers dissimilar from the training set by sampling from the embedding prior.This result indicates that the learned representation captures speaker variation beyond the training identities.
- The model does not attain human-level naturalness, cannot transfer accents, and cannot completely isolate speaker voice from reference prosody.The authors attribute the naturalness gap partly to generating many speakers with less data per speaker and lower-quality datasets.
Appendix A Additional joint training baselines
The appendix evaluates jointly trained speaker encoder and synthesizer baselines on LibriSpeech and compares their naturalness and similarity with separately trained systems.
- The evaluation compares jointly trained baselines with separately trained, embedding-lookup, and proposed-model results using naturalness and speaker-similarity MOS.The comparison includes models trained entirely on LibriSpeech except the proposed model row.
- The jointly trained systems use 1.2K LibriSpeech speakers and 64-dimensional speaker embeddings.One baseline has no output constraints, while the other adds a speaker discrimination loss through a softmax speaker classifier.
- Both jointly trained models achieve similar naturalness MOS on Seen speakers, while the discriminative speaker-loss variant performs better on Unseen speakers.For Unseen speakers, its naturalness and similarity are nearly the same as the separately trained baseline.
Appendix B Speaker variation
Speaker-level variation affects both ground-truth consistency and synthesized naturalness, with VCTK showing more consistent evaluations than LibriSpeech. On VCTK, synthesized naturalness also varies substantially across speakers.
- LibriSpeech utterances vary in tone and style even within speakers, producing variable self-similarity and naturalness scores.Some speakers mimic voices in a different gender, and recording noise further lowers naturalness for some speakers.
- VCTK is more consistent than LibriSpeech in both naturalness and self-similarity evaluations.
- On VCTK, synthesized naturalness varies across speakers: p240 scores 4.48 versus 4.57 for ground truth, while p260 is 0.5 points below its ground truth.Table 4 compares speaker-specific MOS for ground-truth and synthesized speech.
- The appendix reports ground-truth MOS breakdowns for unseen speakers, with similarity evaluations comparing two utterances from the same speaker.
- Figure 4 presents per-speaker naturalness MOS for ground-truth and synthesized speech on unseen VCTK speakers.
Appendix C Impact of reference speech duration
Reference speech duration affects speaker similarity and naturalness in the proposed system. Longer references improve similarity until quality saturates at about 5 seconds on VCTK.
- Longer reference speech significantly improves speaker similarity by enabling a more precise speaker embedding, with quality saturating at about 5 seconds on VCTK.
- Shorter reference utterances provide slightly better naturalness because they better match the synthesizer’s training-reference durations.The median training-reference duration is 1.8 seconds.
Appendix D Evaluation speaker sets
Table 10 presents the speaker sets used for evaluation.
- Table 10 lists the speaker sets used for evaluation.
Appendix E Fictitious speakers
The appendix illustrates synthesis from random speaker embeddings using a fixed sentence. These samples preserve phonetic content while varying in fundamental frequency and speaking rate.
- The example sentence is “this is a big red apple.”
- Several random speaker embeddings are used to condition synthesis of the example sentence.
- All samples retain consistent phonetic content while varying in fundamental frequency and speaking rate.
Appendix F Speaker similarity MOS evaluation interface
Figure 6 shows the interface used for evaluating speaker similarity with mean opinion scores.
- Figure 6 presents the interface for mean opinion score evaluation of speaker similarity.