Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings

1. INTRODUCTION

• Speaker adaptation remains challenging despite realistic end-to-end TTS with MOS approaching natural speech and strong multi-speaker systems.Fine-tuning methods require transcribed adaptation data and additional computation, while jointly trained speaker encoders are limited to TTS training data.Source: §1 · p. 1
• Transfer learning uses a separately trained speaker embedding network, allowing speaker models to exploit larger, lower-quality datasets and support zero-shot adaptation without necessarily transcribing data or fine-tuning the TTS model.The embedding can provide robust representations less dependent on channel and recording conditions, using relatively little target-speaker data.Source: §1 · p. 1
• In zero-shot TTS, observed that synthetic speech for unseen speakers, compared with seen speakers, had lower target-speaker similarity, often mismatched accents, and lost characteristic prosodic nuances.Source: §1 · p. 1
• X-vector systems outperformed i-vector systems in NIST SRE18 summaries, while learnable dictionary encoding and angular softmax were reported to boost speaker recognition on VoxCelebs [26] [27].The study tests whether these speaker-recognition advances transfer to speaker adaptation in TTS.Source: §1 · p. 1
• The study extends an improved Tacotron system to multi-speaker TTS, compares neural speaker embeddings for unseen-speaker modeling, and analyzes how voice quality and similarity relate to ASV accuracy.It also asks whether embeddings that perform best for ASV are best for TTS, a comparison the authors describe as the first investigation of its kind.Source: §1 · p. 1

2. NEURAL SPEAKER EMBEDDINGS

• A typical end-to-end speaker-recognition system uses an encoder, statistical pooling, and a classifier; embeddings are obtained after pooling and before classification.Source: §2 · pp. 1–2

2.1. Encoder Network

• The original x-vector encoder was a TDNN, while later work explored ResNet34 variants [25] [27]; this study uses TDNN for x-vectors and ResNet34 for LDE.Source: §2.1 · p. 2

2.2. Pooling methods

• Learnable Dictionary Encoding soft-assigns frame representations to learned clusters and aggregates cluster means and standard deviations into an utterance-level embedding.It computes L2 distances from frames to clusters, derives soft cluster weights, then aggregates over time.Source: §2.2 · p. 2

2.3. Classifier

• The study examines normal softmax and angular softmax; angular-margin criteria are intended to produce more separable embeddings [24].The standard speaker-ID classifier uses softmax with cross-entropy training.Source: §2.3 · p. 2

3. MULTI-SPEAKER TTS MODEL ARCHITECTURE

• A phoneme-input Tacotron encoder uses LSTM and self-attention outputs with forward and additive attention, concatenated before decoding to an 80-dimensional mel-spectrogram.The LSTM path uses forward attention to speed alignment, while the self-attention path uses additive attention to capture longer-range information.Source: §3 · p. 2
• Speaker embeddings are averaged per training speaker, projected to 64 dimensions, and considered at the encoder outputs, decoder prenet, or postnet.The embeddings are extracted from each training utterance using the speaker encoder and used to condition speaker characteristics.Source: §3 · p. 2
• Zero-shot adaptation averages embeddings from untranscribed target-speaker utterances without fine-tuning, then uses WaveNet to convert predicted mel-spectrograms into 16-bit 16kHz waveforms.The averaged speaker embedding conditions generation of the target speaker’s mel-spectrograms.Source: §3 · pp. 2–3

4.1. Speaker Verification

• LDE embeddings performed on par with x-vectors on VoxCeleb1 verification, with smaller embeddings, larger angular margin, and pooling both mean and standard deviation improving performance.The verification results were reported on the original VoxCeleb1 test set.Source: Table 1 · p. 3, §4.1 · p. 3

4.2. Preliminary Experiments for Speaker Similarity in TTS

• Gender-dependent training with x-vectors at both the prenet and attention mechanism produced the best unseen-speaker similarity and was selected for later experiments.Configurations were evaluated using cosine similarity between synthesized-speech x-vectors and target-speaker x-vectors; postnet input alone was excluded because it produced poor-quality speech.Source: Table 2 · p. 4, §4.2 · p. 4

4.3. Comparing Different Embeddings for Speaker Similarity

• The crowdsourced comparison evaluated 15 TTS systems, each using 600 test utterances synthesized from 50 sentences for each of four seen and eight unseen speakers, rating naturalness MOS and speaker-similarity DMOS.A total of 463 subjects participated.Source: §4.3 · p. 4
• LDE-3 was best for completely unseen test speakers in naturalness and similarity, significantly outperforming x-vectors (p=5.9e-11; p=0.02).It was also the best embedding type in EER; advanced neural embeddings improved unseen-speaker similarity and naturalness, while angular softmax and post-processing had small effects. No meaningful correlations between ASV and TTS scores were found.Source: Table 3 · p. 4, §4.3 · p. 4

• LDE-based neural speaker embeddings improve speaker similarity and naturalness for unseen speakers and support zero-shot speaker adaptation.Source: §5 · p. 4
• A remaining gap in speaker similarity between seen and unseen speakers suggests TTS overfitting to seen speakers and room for improvement.Source: §5 · p. 4
• Future work will explore speaker-space augmentation to mitigate overfitting and evaluate similarity in nuanced aspects such as dialect and speaking style.Source: §5 · p. 4