Source-linked AI summary
Deep Voice 2: Multi-Speaker Neural Text-to-Speech
Sercan Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, Yanqi Zhou
TL;DR
The paper addresses the data and development burden of supporting multiple voices in neural TTS. It introduces shared neural TTS models with low-dimensional trainable speaker embeddings, alongside improved Deep Voice 2 and a WaveNet vocoder for Tacotron. The resulting systems generate high-quality, distinguishable voices and learn hundreds of voices from small amounts of per-speaker data.
Problem
Supporting multiple TTS voices traditionally requires separate speech databases or model parameters, increasing data and development effort.
Method
The paper adds low-dimensional trainable speaker embeddings to shared Deep Voice 2 and Tacotron models, and uses a WaveNet vocoder with Tacotron.
Results
The systems achieve high-quality synthesis while preserving distinguishable speaker identities, and learn effectively from small amounts of data spread among hundreds of speakers.
Takeaways & Limitations
A single neural TTS system can support many voices while sharing most parameters and using less data per speaker than single-speaker systems.
Takeaways & Limitations
Multi-speaker Tacotron performance is sensitive to hyperparameters and preprocessing, and additional tuning may be needed for maximal quality.
Abstract
from arXiv · showhide
We introduce a technique for augmenting neural text-to-speech (TTS) with lowdimensional trainable speaker embeddings to generate different voices from a single model. As a starting point, we show improvements over the two state-ofthe-art approaches for single-speaker neural TTS: Deep Voice 1 and Tacotron. We introduce Deep Voice 2, which is based on a similar pipeline with Deep Voice 1, but constructed with higher performance building blocks and demonstrates a significant audio quality improvement over Deep Voice 1. We improve Tacotron by introducing a post-processing neural vocoder, and demonstrate a significant audio quality improvement. We then demonstrate our technique for multi-speaker speech synthesis for both Deep Voice 2 and Tacotron on two multi-speaker TTS datasets. We show that a single neural TTS system can learn hundreds of unique voices from less than half an hour of data per speaker, while achieving high audio quality synthesis and preserving the speaker identities almost perfectly.
1 Introduction
The paper develops neural TTS systems that support multiple voices within one shared model, addressing the greater data and development effort traditionally required for multi-speaker systems. It introduces improved single-speaker components and trainable speaker embeddings for Deep Voice 2 and Tacotron.
- Multi-speaker TTS traditionally requires distinct speech databases or model parameters for different voices.This increases the data and development effort compared with single-speaker systems.
- A single all-neural TTS model can generate multiple voices while sharing the vast majority of its parameters.
- Deep Voice 2 is an improved architecture based on Deep Voice 1.
- A WaveNet-based spectrogram-to-audio neural vocoder replaces Griffin-Lim audio generation for Tacotron.
- Trainable speaker embeddings extend both Deep Voice 2 and Tacotron to multi-speaker neural speech synthesis.
2 Related Work
The paper situates its approach among neural and traditional multi-speaker TTS methods. Its distinguishing strategy is to train low-dimensional speaker embeddings jointly with a shared synthesis model rather than relying on fixed embeddings, per-speaker output layers, or average-voice adaptation.
- Single-speaker speech synthesis: Earlier neural TTS work addressed components such as duration, fundamental frequency, acoustic modeling, and waveform generation, including Deep Voice 1 and Tacotron.
- Multi-speaker speech synthesis: Traditional and DNN-based multi-speaker TTS methods used average voice models, speaker representations, or speaker-dependent output layers.
- Trainable speaker embeddings: The paper uses trainable speaker embeddings for multi-speaker TTS, trained jointly with the rest of the model from scratch.
- Trainable speaker embeddings: Unlike prior approaches, the method avoids fixed embeddings, per-speaker output layers, and average voice modeling.
- Evaluation: Speaker-discriminator classification accuracy is proposed as an automated measure of generated-voice distinctiveness.
3 Single-Speaker Deep Voice 2
Deep Voice 2 retains Deep Voice 1’s staged inference pipeline while replacing or refining several components. Its stages convert text to phonemes, predict and expand durations, generate F0-related features, and synthesize audio with a vocal model.
- Architecture: Deep Voice 2 separates phoneme-duration prediction from frequency modeling, predicting durations first and then using them as frequency-model inputs.
- Inference pipeline: The inference pipeline converts text to phonemes, predicts phoneme durations, upsamples them to frame-level features, generates F0, and feeds F0 with phonemes to the vocal model.
- Segmentation model: Phoneme segmentation uses a convolutional-recurrent architecture with CTC loss to classify phoneme pairs and extract boundaries.
- Segmentation model: Deep Voice 2 adds batch normalization and residual connections to the segmentation model and corrects silence boundaries with a detection heuristic.
- Duration model: Duration prediction discretizes phoneme durations into log-scaled buckets and models their sequence with a CRF decoded by the Viterbi forward-backward algorithm.
- Frequency model: The frequency model combines bidirectional GRU and convolutional predictions to estimate voicedness and normalized F0 before converting it to true F0.
- Vocal model: The vocal model is based on WaveNet with a two-layer bidirectional QRNN conditioning network and modified residual-conditioning connections.
4 Multi-Speaker Models with Trainable Speaker Embeddings
The paper extends neural TTS architectures with trainable, low-dimensional speaker embeddings, placing them in different model components to synthesize multiple voices from shared parameters. It also evaluates speaker-conditioned Tacotron and replaces Griffin-Lim with a WaveNet-based vocoder for higher-quality waveform generation.
- Shared Speaker Embeddings: Speaker embeddings provide near-complete weight sharing while storing speaker-dependent parameters in low-dimensional vectors.The embeddings can produce recurrent initial states, biases, and multiplicative feature modulation.
- Speaker-Conditioning Mechanisms: The models use site-specific projections, recurrent initialization, input augmentation, and feature gating to inject speaker information.These mechanisms transform a shared embedding for each architectural use site and control recurrent states, inputs, or activations.
- Deep Voice 2: Deep Voice 2 segmentation uses feature gating and shared site-specific embeddings across convolutional layers, while recurrent layers receive a second site-specific initialization embedding.The same site-specific embedding is shared across the relevant layers rather than learned separately per layer.
- Deep Voice 2: The duration model uses speaker-dependent recurrent initialization and input augmentation, whereas the frequency model uses recurrent initialization with a site-specific speaker embedding.Frequency conversion also uses trainable F0 normalization parameters tied to observed dataset statistics.
- Vocal Model: The vocal model concatenates speaker embeddings to each conditioner input frame, allowing speaker information to influence the local conditioning network.This differs from global conditioning and supports speaker-dependent waveform generation.
- Tacotron and Results: Speaker embeddings improve vocal-model audio quality and converge to a meaningful latent space, while multi-speaker Tacotron remains sensitive to hyperparameters and silence preprocessing.The paper focuses on demonstrating multi-speaker capability rather than directly comparing Tacotron and Deep Voice 2 quality.
- Tacotron and Results: A WaveNet-based neural vocoder replaces Griffin-Lim for Tacotron spectrogram-to-waveform conversion and significantly improves single-speaker Tacotron audio quality.The vocoder converts linear-scaled log-magnitude spectrograms into audio waveforms.
5 Results
The experiments evaluate single- and multi-speaker synthesis using MOS and speaker-classification accuracy. Deep Voice 2 approaches ground-truth audio quality while generated voices remain distinguishable, though MOS comparisons include ground-truth samples and may lower scores.
- Evaluation: The evaluation uses crowdMOS to assess synthesized audio quality and speaker classification to assess voice distinctiveness.Table 2 reports both MOS and discriminator classification accuracy for the multi-speaker models.
- Multi-Speaker Speech Synthesis: The multi-speaker models were trained on VCTK with 108 speakers and on an audiobook dataset with 477 speakers receiving 30 minutes of audio each.The observed sample quality indicates that the architectures can learn hundreds of voices with varied accents and cadences.
- Multi-Speaker Speech Synthesis: Deep Voice 2 can approach ground-truth MOS when low sampling rate and companding/expanding are taken into account.Ground-truth samples were included to compare model quality against data quality, but this implicit comparison may lower resulting MOS.
- Multi-Speaker Speech Synthesis: The learned speaker embeddings occupy a meaningful latent space for the VCTK models.Principal components are shown for the 80-layer vocal and character-to-spectrogram models.
- Multi-Speaker Speech Synthesis: Speaker-classification accuracy shows that generated voices from the models are as distinguishable as ground-truth voices.The discriminator evaluates whether synthesized speech preserves speaker identity in addition to audio quality.
6 Conclusion
The paper extends entirely neural TTS to multiple speakers with low-dimensional trainable speaker embeddings. It concludes that neural TTS can learn hundreds of voices from small amounts of speaker-specific data, while identifying open questions about scalability and embedding use.
- Conclusion: The paper uses low-dimensional trainable speaker embeddings to extend entirely neural speech synthesis to multi-speaker TTS.Both multi-speaker Deep Voice 2 and Tacotron are trained with this technique and evaluated through MOS.
- Conclusion: The results show that neural TTS models can learn effectively from small amounts of data distributed across hundreds of speakers.The conclusion describes this as producing high-quality text-to-speech systems.
- Future Work: Future research may test speaker-count limits, minimum data per speaker, adding speakers by training embeddings alone, and whether embeddings form a meaningful vector space.These questions define the stated directions for extending and understanding the technique.
A Training Deep Voice 2
The appendix documents Deep Voice 2’s training diagram, implementation substitutions, and intermediate comparisons with Deep Voice 1. Deep Voice 2 improves intermediate evaluation metrics, although those metrics do not fully quantify synthesis quality.
- Training Procedure: Figure 5 depicts the training procedure for Deep Voice 2.The diagram is presented as a convenience and is similar to the corresponding Deep Voice 1 figure.
- Implementation: The pronunciation dictionary may be replaced by a grapheme-to-phoneme model to support words unseen in the dictionary.This substitution is described as an adaptation of the Deep Voice 1 approach.
- Implementation: Praat is used for frequency extraction during training, but another fundamental-frequency and voicedness estimator may be substituted.The appendix identifies this as an implementation choice rather than a required component.
- Intermediate Evaluation: Deep Voice 2 obtains significant improvements over Deep Voice 1 in the intermediate segmentation, duration, and frequency evaluation metrics.Table 3 compares the models using the same single-speaker data and evaluation procedure.
B Model Hyperparameters
The appendix provides model hyperparameters and implementation conventions for the experiments. It also notes a CTC penalty used to accelerate Tacotron training, without conclusive evidence that it improves audio quality.
- Hyperparameters: Table 4 provides hyperparameters for all models presented in the paper.The appendix identifies this table as the source for the experimental configurations.
- Tacotron Training: A CTC penalty added to Tacotron’s loss was observed to accelerate convergence in some cases, without conclusive evidence of improving audio quality.The penalty is derived from attention hidden states and was introduced to speed training.
- Optimization: Learning rates are specified by initial value, decay factor, and decay interval, and all models use Adam with β1 = 0.9, β2 = 0.99, and ε = 10^-8.The notation ℓ–r–s denotes decay of the initial rate ℓ by r every s iterations.
- Architecture Notation: Convolutional-layer notation specifies layer count, output channels, and filter height and width in frequency bins and time frames.This convention explains how convolutional configurations are represented in the hyperparameter table.
C Training Time for Multi-speaker Models
This section provides training-time details for the multi-speaker Deep Voice 2 and Tacotron models on the VCTK dataset and points to related inference optimizations.
- Training-time details for the multi-speaker Deep Voice 2 and Tacotron models are reported in Table 5.
- The implementation uses the optimized WaveNet inference kernels described in Deep Voice 1.
- Detailed analysis of inference-time optimization techniques is deferred to Arik et al. (2017).
- The models are trained on the VCTK dataset.
D.1 Speaker-Dependent Fundamental Frequency Profiles
The learned speaker embeddings affect intermediate speech representations, including fundamental-frequency profiles and attention, while exhibiting patterns associated with speaker gender and region.
- D.1 Speaker-Dependent Fundamental Frequency Profiles: Speaker embeddings preserve the overall fundamental-frequency profile shape while making the actual frequency values highly speaker dependent.The input phoneme features dominate the profile shape, whereas substituting a different speaker embedding changes speaker-dependent values.
- D.2 Speaker-Dependent Attention: Speaker-dependent attention focuses on different portions of the input text for speakers who speak at different speeds.Figure 7 compares attention plots for three speakers with different speaking speeds.
- D.3 Principal Components of the Embeddings: The learned embeddings show discriminative patterns for speaker gender and region despite random initialization and training only through generative-quality losses.Gender is the most apparent pattern, while Great Britain and North America appear especially separable in the two-dimensional visualization.
- D.4 Speaker Classification Accuracy: Multi-speaker classification accuracy is measured with a speaker-discriminative model trained on ground-truth data from multiple speakers.The authors report that their test-set accuracy is on par with state-of-the-art speaker-classification methods and use their own deep learning discriminator.
- D.4 Speaker Classification Accuracy: The discriminator study evaluates multiple hyperparameter configurations and reports selected models D3 and D8 because they achieved the highest validation accuracy.Tables 6 and 7 provide discriminator hyperparameters, while Tables 8 and 9 report classification accuracy for the evaluated models.