Source-linked AI summary
Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, John Miller
TL;DR
Neural TTS needs architectures that handle pronunciation and attention errors while scaling to large, multispeaker datasets and production traffic. Deep Voice 3 addresses these needs with a fully-convolutional attention-based model, achieving ten-fold faster training alongside large-scale training and deployment capabilities.
Problem
Raw-text TTS can mispronounce rare words or skip and repeat words, while deployed systems need pronunciation modification for proper nouns, foreign words, and domain-specific jargon.
Method
Deep Voice 3 uses a fully-convolutional sequence-to-sequence acoustic model with position-augmented attention, joint character-phoneme representations, and multi-task decoder-converter training.
Results
Ten-fold faster training is achieved than Tacotron, with 820 hours of audio from 2484 speakers and support for multiple waveform synthesis methods.
Takeaways & Limitations
Deep Voice 3 supports large-scale multispeaker TTS, mitigates common attention errors, and can serve ten million queries per day on one single-GPU server.
Takeaways & Limitations
On subsampled LibriSpeech data with only 108 speakers, generated samples have worse quality than VCTK.
Abstract
from arXiv · showhide
We present Deep Voice 3, a fully-convolutional attention-based neural text-to-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training ten times faster. We scale Deep Voice 3 to data set sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on one single-GPU server.
1 INTRODUCTION
Deep Voice 3 introduces a fully-convolutional architecture for speech synthesis, scales TTS to very large datasets, and addresses deployment issues in attention-based systems. Its contributions include faster training, large-scale multispeaker training, monotonic attention, waveform-method comparisons, and high-throughput inference.
- 1 INTRODUCTION: Deep Voice 3 proposes a fully-convolutional character-to-spectrogram architecture with fully parallel computation.The architecture trains an order of magnitude faster than analogous recurrent architectures.
- 1 INTRODUCTION: 820 hours of audio from 2484 speakers are used to scale the system to the LibriSpeech ASR dataset.
- 1 INTRODUCTION: The system generates monotonic attention behavior to avoid repeated words, mispronunciations, and skipped words common in sequence-to-sequence TTS.
- 1 INTRODUCTION: Deep Voice 3 compares WORLD, Griffin-Lim, and WaveNet waveform synthesis methods.
- 1 INTRODUCTION: Ten million queries per day can be served by Deep Voice 3 on one single-GPU server.
2 RELATED WORK
Deep Voice 3 builds on neural and attention-based sequence-to-sequence TTS while replacing recurrent components with fully convolutional processing. It also targets the scale and deployment requirements of production speech synthesis.
- 2 RELATED WORK: Neural TTS research includes Deep Voice, Tacotron, Char2Wav, VoiceLoop, SampleRNN, and WaveNet.
- 2 RELATED WORK: Deep Voice 3 uses an attention-based sequence-to-sequence model, yielding a more compact architecture than Deep Voice 1 and 2.
- 2 RELATED WORK: Deep Voice 3 avoids recurrent neural networks to speed up training relative to Tacotron and Char2Wav.
- 2 RELATED WORK: Deep Voice 3 is presented as the first single-model TTS system known to the authors to scale to thousands of speakers.
- 2 RELATED WORK: Attention mechanisms let sequence-to-sequence decoders adaptively select encoder hidden states while generating target sequences.
3 MODEL ARCHITECTURE
Deep Voice 3 uses a fully convolutional encoder–decoder–converter architecture to transform textual features into vocoder parameters. Its design combines parallel convolutional processing, attention-guided spectrogram prediction, multi-task training, speaker conditioning, and text normalization to address synthesis and deployment challenges.
- Architecture: The architecture converts textual features into vocoder parameters through fully convolutional encoder, causal decoder, and non-causal converter components.The decoder autoregressively predicts mel-scale spectrograms, while the converter uses decoder hidden states and future context to predict final vocoder parameters.
- Training: Multi-task training combines decoder and converter losses because mel-spectrogram prediction supplies gradients that make attention learning easier.The overall objective is a linear combination of decoder and converter losses.
- Conditioning and preprocessing: Speaker-dependent embeddings condition the encoder, decoder, and converter in multi-speaker models, while input normalization and optional phoneme representations target pronunciation, repetition, and skipping errors.Normalization includes character uppercasing, punctuation removal, utterance-final punctuation, and pause-marking separators; phoneme inputs provide explicit pronunciation information.
- Convolution blocks: Fully convolutional layers provide long-term sequence context without sequential dependencies that limit training parallelism in recurrent networks.The convolution block combines one-dimensional convolution, a gated linear unit, residual connection, and scaling factor.
- Encoder and attention: The encoder creates attention keys from processed character or phoneme embeddings and derives value vectors that combine local embedding information with long-term contextual information.Attention uses the keys to compute weights and weighted value vectors to form context.
- Attention constraints: Inference can constrain attention to progress monotonically, while the reported hard monotonic attention alternative sometimes became stuck on the first or last character.The paper preserves unconstrained training when applying monotonic constraints at inference.
4 RESULTS
Deep Voice 3 is evaluated across training speed, attention errors, waveform naturalness, multi-speaker synthesis, and deployment throughput. Results show faster training, mitigated attention errors, competitive quality, and scalable single-GPU inference, with dataset-quality and evaluation-scope caveats.
- Data: 820 hours of LibriSpeech audio from 2484 speakers supports multi-speaker training at substantially larger scale than the VCTK dataset.VCTK contains approximately 44 hours from 108 speakers, while LibriSpeech contains approximately 820 hours from 2484 speakers.
- Fast Training: 0.06 seconds versus 0.59 seconds per training iteration indicates a ten-fold training-speed increase over Tacotron.Deep Voice 3 converges after approximately 500K iterations, while Tacotron requires approximately 2M iterations.
- Attention Error Modes: Repeated words, mispronunciations, and skipped words are common attention errors, and monotonic inference substantially improves error counts.The evaluation uses a custom 100-sentence test set containing challenging cases such as dates, acronyms, URLs, and proper nouns.
- Naturalness: 3.78 MOS for WaveNet exceeds 3.63 for WORLD and 3.62 for Griffin-Lim in waveform-synthesis comparisons.WORLD offers lower inference latency, reaching up to 40X realtime per CPU core versus 3X realtime for the described WaveNet implementation.
- Multi-Speaker Synthesis: 3.44 MOS on VCTK with WORLD is comparable to Deep Voice 2's 3.69 MOS, while LibriSpeech quality is lower under varied recording conditions.The multi-speaker evaluation includes ground-truth samples, and the cited comparison does not train Deep Voice 2 or Tacotron on LibriSpeech.
- Multi-Speaker Synthesis: Worse generated-sample quality is observed when LibriSpeech is subsampled to 108 speakers, matching VCTK's speaker count.
5 CONCLUSION
Deep Voice 3 is a fully convolutional sequence-to-sequence TTS system with position-augmented attention. The paper reports avoidance of common attention errors, waveform-method flexibility, multi-speaker capability, production-scale inference, and state-of-the-art quality through MOS evaluations.
- Deep Voice 3 uses a fully-convolutional sequence-to-sequence acoustic model with position-augmented attention.
- The system avoids common sequence-to-sequence speech-synthesis attention errors and supports Griffin-Lim, WaveNet, and WORLD waveform synthesis.
- Trainable speaker embeddings extend the architecture to multi-speaker speech synthesis.
- The production-ready system includes text normalization and performance characteristics and reports state-of-the-art quality through extensive MOS evaluations.
A DETAILED MODEL ARCHITECTURE OF DEEP VOICE 3
Deep Voice 3 encodes text or phonemes into timestep-specific key and value vectors for an attentional decoder. The decoder predicts mel-band spectrograms, and a converter produces linear spectrograms or waveform-vocoder parameters.
- A deep residual convolutional encoder maps text and/or phonemes to per-timestep key and value vectors for attention.
- The attentional decoder uses these vectors to predict mel-band log magnitude spectrograms corresponding to output audio.
- The decoder hidden state feeds a converter network that outputs linear spectrograms for Griffin-Lim or parameters for WaveNet.
B OPTIMIZING DEEP VOICE 3 FOR DEPLOYMENT
Deep Voice 3 deployment required replacing prohibitively slow TensorFlow-graph inference with custom GPU kernels, while parallelizing WORLD synthesis across CPUs. The resulting system reaches the targeted single-server throughput, with GPU inference remaining the bottleneck.
- Throughput: 115 QPS on one Nvidia Tesla P100 GPU meets the target of ten million queries per day.WORLD synthesis is parallelized across all 20 CPUs.
- Implementation: TensorFlow graph execution averages approximately 1 QPS because graph-evaluator overhead accumulates across hundreds of nodes and timesteps.
- Bottleneck: GPU inference is the bottleneck because WORLD synthesis on 20 cores is faster than 115 QPS.
- Future work: The authors leave faster kernels, smaller models, and fixed-precision arithmetic for future work.
C MODEL HYPERPARAMETERS
The paper reports the hyperparameters used by its best models across the three datasets in Table 4.
- Reference: Model hyperparameters are presented as a reference for the reported best-model configurations.
- Reported settings: Table 4 lists hyperparameters for the best models used in the paper.
- Dataset coverage: The table covers the three datasets used in the paper.
D LATENT SPACE OF THE LEARNED EMBEDDINGS
Principal-component analysis reveals clear male–female separation in learned speaker embeddings for both VCTK and LibriSpeech. This separation suggests the low-dimensional embeddings form a meaningful latent space.
- Embedding structure: PCA shows a clear separation between male and female speakers in the learned embedding space.
- Interpretation: The low-dimensional speaker embeddings constitute a meaningful latent space according to the observed gender separation.
- Visualization: Figure 7 visualizes the first two principal components for VCTK's 108 speakers and LibriSpeech's 2484 speakers.
E 100-SENTENCE TEST SET
The paper uses a 100-sentence test set to quantify results reported in Table 1.
- Evaluation set: A 100-sentence test set provides the basis for quantifying the results in Table 1.
- Documentation: The test sentences are listed in the appendix material for reference.