Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

• The system targets zero-shot voice synthesis from seconds of untranscribed reference speech without parameter updates, with accessibility potential and a misuse concern addressed by testing distinguishability from real voices.Source: §1 · p. 1
• Because high-quality multispeaker speech-transcript data is costly, the approach separates speaker modeling from TTS, using independently trained speaker-verification embeddings to condition a smaller synthesis dataset.The speaker encoder is trained on untranscribed speech from many speakers, including reverberation and background noise.Source: §1 · pp. 1–2
• 1.2K-speaker TTS training paired with an 18K-speaker encoder improves adaptation quality and enables novel-speaker synthesis by sampling the embedding prior.Source: §1 · p. 2
• Prior TTS systems used fixed training-speaker embeddings or required extensive enrollment and transcripts for unseen voices [18].Tacotron 2 was single-speaker; few-shot encoders [2] [10] enabled adaptation from short, transcript-free speech, while [10] trained jointly with contrastive and cycle-consistency losses.Source: §1 · p. 2
• Unlike jointly trained encoding systems [10], this work transfers an independently pretrained speaker-verification encoder trained on untranscribed audio from tens of thousands of speakers [22].The encoder uses a generalized end-to-end loss [22].Source: §1 · p. 2
• Compared with the classifier-based transfer setup, this work uses end-to-end synthesis and an embedding network not limited to a closed speaker set, finding zero-shot transfer requires thousands of training speakers.Source: §1 · p. 2

2 Multispeaker speech synthesis model

• Three independently trained components map speech to a speaker embedding, text to a mel spectrogram, and the spectrogram to waveform samples using an autoregressive WaveNet [19].The recurrent speaker encoder is based on [22], and the sequence-to-sequence synthesizer on.Source: §2 · pp. 2–3, Figure 1 · p. 3

2.1 Speaker encoder

• A text-independent speaker-verification encoder maps variable-length speech to fixed-dimensional d-vectors, training same-speaker utterances for high cosine similarity and different speakers to be far apart.Although not optimized directly for synthesis-relevant speaker characteristics, the learned embedding is suitable for conditioning synthesis on speaker identity.Source: §2.1 · p. 3

2.2 Synthesizer

• Tacotron 2 conditions on speaker embeddings concatenated with encoder outputs; training uses frozen pretrained encoder embeddings from target audio, phoneme inputs, and no explicit speaker labels.The authors compare encoder-derived embeddings with learned per-speaker lookup embeddings and report that adding L1 to spectrogram L2 loss improves robustness on noisy training data.Source: §2.2 · pp. 3–4

2.3 Neural vocoder

• A 30-layer dilated-convolution autoregressive WaveNet vocoder [19] converts predicted mel spectrograms to waveforms without direct speaker-encoder conditioning.The multispeaker vocoder is trained on data from many speakers because the mel spectrogram is described as capturing the detail needed for high-quality voices.Source: §2.3 · p. 4

2.4 Inference and zero-shot speaker adaptation

• A few seconds of arbitrary, untranscribed reference audio unrelated to the target text enables zero-shot synthesis for unseen speakers, with pitch, formants, sibilants, and speaking rate reflected in outputs.Figure 2 illustrates synthesis for three speakers held out of training, using five-second reference utterances.Source: §2.4 · pp. 4–5, Figure 2 · p. 4

3 Experiments

• Experiments trained separate synthesis and vocoder networks on VCTK and LibriSpeech, using an independently trained encoder and 1–5 MOS ratings for naturalness and target-speaker similarity.LibriSpeech synthesis targets were denoised, but its vocoder used synthesizer-predicted spectrograms; VCTK's vocoder used ground-truth mels.Source: §3 · p. 5

3.2 Speaker similarity

• On VCTK seen speakers, the proposed model matched the lookup-table baseline's similarity MOS, while on LibriSpeech it scored lower; unseen VCTK speakers received a 3.28 similarity score.The authors note that encoder accent mismatch may reduce VCTK similarity ratings; cross-dataset tests found LibriSpeech-trained synthesis preserved naturalness and transferred similarity to VCTK better than VCTK-trained synthesis did to LibriSpeech.Source: Table 2 · p. 6, §3.2 · pp. 6–7, Table 3 · p. 7

3.1 Speech naturalness

• The proposed model achieved about 4.0 MOS across datasets, and unseen-speaker speech was at least as natural as seen-speaker speech.On LibriSpeech, unseen-speaker MOS was up to 0.2 points higher, which the authors attribute to reference utterances with uneven prosody; they suggest disentangling speaker identity from prosody.Source: §3.1 · p. 6

3.3 Speaker verification

• The LibriSpeech-trained synthesizer achieved similar 5–6% SV-EERs on speakers from both datasets, while a 20-voice real-versus-synthetic task yielded 2.86% EER.Synthetic speech tended to resemble the target speaker but remained closer in embedding space to other synthetic utterances from that speaker, rather than being confusable with real speech.Source: Table 4 · p. 7, §3.3 · p. 7

3.4 Speaker embedding space

• In embedding visualizations, synthetic utterances lay near real utterances from the same speaker, yet formed distinct clusters; speakers also separated by gender.The visualizations indicate that the encoder represents speaker space in a way that distinguishes speakers and captures gender.Source: §3.4 · pp. 7–8, Figure 3 · p. 8

3.5 Number of speaker encoder training speakers

• As the speaker encoder's training-speaker count increased, naturalness, similarity, and objective EER improved; its training data need not be transcribed or as high-quality as TTS data.The matched LibriSpeech Clean encoder-and-synthesizer condition gave slightly better naturalness and similar similarity to the comparison condition; the authors conclude that large untranscribed encoder datasets can complement smaller high-quality TTS datasets.Source: §3.5 · pp. 8–9

3.6 Fictitious speakers

• Random points on the unit hypersphere produced fictitious speakers with speech as natural as that of seen or unseen real speakers, while low nearest-neighbor cosine similarity and high EER indicated distinct identities.The evaluation compared 10 generated speakers with their nearest neighbors in component-network training sets.Source: Table 6 · p. 9, §3.6 · p. 9

4 Conclusion

• Speaker-verification evaluations and listening tests show reasonably similar synthesized speech for target speakers, including speakers unseen during training.The system combines an independently trained speaker encoder, a Tacotron 2-based sequence-to-sequence synthesizer, and a neural vocoder.Source: §4 · p. 9
• With sufficient speaker diversity in synthesizer training, increasing speaker-encoder training data significantly improves speaker-transfer quality.Source: §4 · p. 9
• Separating encoder and synthesizer training lowers multispeaker TTS data requirements and simplifies synthesizer training without triplet or contrastive losses [10].Synthesizer training needs no speaker identity labels, and encoder training needs neither clean speech nor transcripts. A low-dimensional speaker vector limits benefits from more than a few seconds of reference speech; adaptation approaches are needed [2], [5].Source: §4 · p. 9
• The model generates realistic speech from fictitious speakers dissimilar to the training set, suggesting it represents speaker variation realistically.Source: §4 · p. 9
• The model falls short of human-level naturalness, cannot transfer accents, and does not fully separate speaker voice from reference prosody.Despite the WaveNet vocoder’s high inference cost, naturalness is limited by generating many speakers with less data per speaker and lower-quality datasets; independent speaker and accent embeddings could address accent transfer, while reference-prosody leakage resembles [16] and contrasts with single-speaker results.Source: §4 · p. 9

Appendix A Additional joint training baselines

• The proposed model with a separately trained 18K-speaker encoder significantly outperforms all baselines on LibriSpeech speech naturalness and speaker-similarity MOS.The table reports MOS with 95% confidence intervals; all evaluations are on LibriSpeech.Source: Table 7 · p. 12, Appendix A · p. 12
• Joint training is evaluated as a baseline, although separate encoder training is necessary when using a larger corpus of untranscribed speech.The joint-training evaluation is described as similar to [10].Source: Appendix A · p. 12
• The joint-training baselines use 1.2K Clean LibriSpeech speakers and 64-dimensional embeddings, comparing unconstrained encoder outputs with embeddings trained using a softmax speaker-classification loss.The classification variant projects the embedding through a linear layer to produce speaker-classifier logits and optimizes cross-entropy.Source: Appendix A · p. 12
• Both jointly trained systems have similar Seen-speaker naturalness MOS, while adding speaker-discrimination loss improves Unseen-speaker performance and nearly matches the Table 5 baseline on Unseen-speaker naturalness and similarity.The Table 5 baseline uses a separately trained speaker encoder optimized to discriminate between speakers.Source: Appendix A · p. 12, Table 7 · p. 12

Appendix B Speaker variation

• LibriSpeech utterances vary in tone and style even for the same speaker, lowering and diversifying self-similarity; recording noise also yields speaker-dependent naturalness scores, unlike more consistent VCTK.Some LibriSpeech speakers even mimic voices of a different gender.Source: Appendix B · p. 12, Table 8 · p. 13
• Speaker p240 scored 4.48 MOS against 4.57 for ground truth, while p260 scored 0.5 points below its ground truth.These examples show that synthesized naturalness on VCTK is speaker-dependent.Source: Appendix B · p. 12, Figure 4 · p. 13

Appendix C Impact of reference speech duration

• On VCTK, longer reference speech significantly improves similarity, with quality saturating at about 5 seconds; 2 seconds achieves close to the best performance.The authors attribute the similarity gain to more precise speaker embeddings from longer reference speech.Source: Table 9 · p. 13, Appendix C · p. 13
• Shorter reference utterances give slightly better naturalness because they better match training utterance durations, whose median is 1.8 seconds.The synthesizer is trained on utterances with a median duration of 1.8 seconds.Source: Appendix C · p. 13
• Saturation at 5 seconds reflects the speaker embedding’s limited capacity; related work [2] found that more data required full-model fine-tuning, a pattern confirmed in [5].Work [2] found adapting only a speaker embedding effective with limited adaptation data.Source: Appendix C · p. 13

Appendix D Evaluation speaker sets

• Table 10 presents the speaker sets used for evaluation.Source: Table 10 · p. 14

Appendix E Fictitious speakers

• Random unit-hypersphere speaker embeddings produce utterances with consistent phonetic content but varying fundamental frequency and speaking rate.The figure shows several synthesized versions of the same sentence.Source: Figure 5 · p. 14

Appendix F Speaker similarity MOS evaluation interface

• Figure 6 shows the interface used for the MOS evaluation of speaker similarity.Source: Figure 6 · p. 15