Source-linked AI summary
Deep Voice: Real-time Neural Text-to-Speech
Sercan O. Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, Shubho Sengupta, Mohammad Shoeybi
TL;DR
Deep Voice addresses the labor and complexity of traditional multi-stage TTS by building every component with neural networks and simpler features. It combines neural text conversion, segmentation, prosody prediction, and WaveNet-based synthesis, achieving faster-than-real-time inference and reporting a 400X speedup over earlier WaveNet implementations. The authors conclude that fully neural TTS can be trained with minimal human involvement, while duration and F0 prediction remain the main barrier to naturalness.
Problem
Traditional TTS systems use complex pipelines with hand-engineered features and heuristics, making new systems labor intensive and difficult to develop.
Method
Deep Voice replaces all TTS components with neural networks, using simpler phoneme-based features and a WaveNet variant for audio synthesis.
Results
Faster-than-real-time inference and a 400X speedup over previous WaveNet inference implementations demonstrate production-oriented neural TTS performance.
Takeaways & Limitations
The system can be retrained on a new audio-only dataset with unaligned transcripts using few hours of manual effort, while duration and F0 prediction remain the main barrier to natural TTS.
Takeaways & Limitations
Model loss is not reliably indicative of perceptual quality below a threshold, and architecture changes can substantially alter loss while scarcely changing audio quality.
Abstract
from arXiv · showhide
We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.
1. Introduction
Deep Voice replaces labor-intensive, hand-engineered TTS components with neural networks while retaining a pipeline structure and simpler features. It also targets production use through faster-than-real-time WaveNet inference.
- Motivation and Contributions: Deep Voice replaces every traditional TTS component with a neural network and uses phonemes, durations, and F0 as its primary features.The system converts text to phonemes before synthesizing speech from linguistic features.
- Motivation and Contributions: A fully neural pipeline is intended to reduce the labor and domain expertise required to develop new TTS systems.Traditional systems rely on complex multistage processing, hand-engineered features, and heuristics.
- Motivation and Contributions: 400X speedup over previous WaveNet inference implementations enables faster-than-real-time generation of high-quality 16 kHz audio.The authors present optimized inference kernels for production-oriented use.
2. Related Work
Deep Voice extends neural TTS work from individual components toward a standalone, fully neural system with simpler features and production-oriented inference speed.
- Scope and Distinctions: Prior work neuralizes individual TTS components, but does not solve the entire TTS problem and often relies on specialized hand-engineered features.Examples include grapheme-to-phoneme, duration, F0, and audio synthesis models.
- Scope and Distinctions: Deep Voice can be trained from scratch using short audio clips and textual transcripts, without requiring a pre-existing TTS system.This distinguishes it from systems that use features from another TTS system during training or inference.
- Scope and Distinctions: Deep Voice minimizes hand-engineered features by using representations obtainable from audio and transcripts with minimal effort.Prior systems use more complex representations that depend on pre-existing TTS systems.
- Scope and Distinctions: Deep Voice targets real-time inference, whereas previous WaveNet results required several minutes to synthesize one second of audio.The system can synthesize audio in fractions of a second and offers a speed-quality trade-off.
3. TTS System Components
The system combines neural models for text conversion, alignment, prosody prediction, and waveform synthesis. Its components include a CTC-based boundary detector, joint duration–F0 predictor, and a more efficient WaveNet variant.
- System Architecture: The five building blocks are grapheme-to-phoneme conversion, segmentation, phoneme duration prediction, fundamental-frequency prediction, and audio synthesis.Together they transform text and learned linguistic features into synthesized speech.
- System Architecture: During inference, text becomes phonemes, predicted durations and F0 form conditioning features, and the synthesis model generates the utterance.The segmentation model is used only to annotate training data with phoneme boundaries and implied durations.
- Grapheme-to-Phoneme Conversion: The grapheme-to-phoneme model uses bidirectional and unidirectional GRU stacks with teacher forcing and beam-search decoding.The decoder uses a beam width of 5 candidates.
- Segmentation: The segmentation model uses CTC to align audio with phoneme sequences, then predicts phoneme pairs to localize precise boundaries.Pair labels encourage outputs near the boundary between adjacent phonemes.
- Prosody Prediction: A single neural network jointly predicts phoneme duration and time-dependent F0 from phonemes with stress annotations.Its joint loss includes duration and F0 errors, voicing likelihood, and an F0 smoothness penalty.
- Audio Synthesis: The audio synthesis model is a WaveNet variant that models discretized audio samples and conditions generation on upsampled linguistic features.It replaces transposed-convolution upsampling with bidirectional QRNN encoding followed by repetition, improving performance while reducing training cost and parameters.
4. Results
The experiments evaluate Deep Voice’s component models, WaveNet configurations, training setup, and perceptual quality. Results show strong grapheme-to-phoneme performance, recognizable speech from sufficiently deep models, and remaining limitations in duration and F0 prediction.
- Grapheme-to-phoneme: 5.8% phoneme error rate and 28.7% word error rate were achieved after approximately 20,000 iterations, matching previous reported results.The model was trained on 124,978 filtered CMUDict grapheme-phoneme pairs without a language model during decoding.
- Duration and F0 prediction: 38 milliseconds mean absolute error for phoneme duration and 29.4 Hz for fundamental frequency were reached after approximately 20,000 iterations.Training used a single TitanX Maxwell GPU with iterations taking approximately 120 milliseconds.
- Audio synthesis: 20-, 30-, and 40-layer WaveNet models produced high-quality recognizable speech, while models below 20 layers produced poor-quality audio.The 40-layer models had less noise than the 20-layer models, detectable with high-quality over-ear headphones.
- Audio synthesis: The 40-layer models generated high-quality audio despite having only 83 milliseconds of receptive field at 48 kHz, suggesting receptive field size was not the source of their quality difference.The authors conjecture that another factor explains the difference between 20- and 40-layer audio quality.
- Perceptual evaluation: Mean opinion scores were high with ground-truth durations and F0, but synthesized F0 and especially synthesized durations reduced perceptual quality.The authors identify duration and fundamental-frequency prediction as the main barrier to more natural TTS and report that their systems had not meaningfully progressed beyond the state of the art there.
- Perceptual evaluation: Models trained with the reduced feature set showed no perceptual difference from models trained with the full feature set of the original WaveNet publication.This supports using the smaller feature representation without an observed perceptual-quality penalty.
5. Optimizing Inference
The paper optimizes autoregressive WaveNet inference by reducing memory and launch overhead while exploiting CPU and GPU hardware. Its CPU kernels achieve real-time or faster-than-real-time performance for some models, whereas GPU kernels remain below real time.
- Performance challenge: 55×10^9 FLOPs are required to synthesize one second of audio with the 40-layer WaveNet model, making meaningful parallelism difficult.The model has approximately 1.6 × 10^6 parameters, and activations depend on both the previous layer and previous timestep.
- Results: CPU kernels run at real-time or faster-than-real-time for a subset of models, while GPU models do not yet match this performance.The benchmarks cover only autoregressive audio generation, excluding linguistic conditioning features that can be generated in parallel.
- CPU implementation: Real-time CPU inference is achieved through avoiding recomputation, cache-friendly accesses, multithreading, thread pinning, and hardware-optimized matrix and convolution routines.The implementation also minimizes nonlinear FLOPs and avoids cache thrashing and thread contention.
- CPU implementation: Two thread groups overlap WaveNet work: auxiliary threads compute Wskip and prepare Wprev while main threads advance through WaveNet layers and the output layer.Spinlocks coordinate waits between the thread groups.
- CPU implementation: 30% higher performance results from pinning threads to physical cores or disabling hyperthreading.The optimization is intended to avoid thread contention and cache thrashing.
- CPU implementation: Approximate nonlinearities increase performance by roughly 30%, with maximum absolute errors of 1.5 × 10^-3 for tanh and 2.5 × 10^-3 for sigmoid.The reported approximation error for exp is 2.4 × 10^-5.
6. Conclusion
Deep Voice demonstrates that a fully neural text-to-speech engine can support faster-than-real-time inference and streaming audio generation. The system is trainable without human involvement, simplifying TTS development, while further optimization and stage integration remain open directions.
- The fully neural system demonstrates that deep learning approaches are viable for all components of a high-quality text-to-speech engine.
- Faster-than-real-time inference enables real-time streaming audio generation.
- Training without human involvement dramatically simplifies the process of creating TTS systems.
- Further directions include inference optimization, model quantization, alternative hardware, and merging stages into a single sequence-to-sequence model.
A. WaveNet Architecture and Details
The modified WaveNet combines a conditioning network with an autoregressive process that predicts audio samples from linguistic features and recent audio context. Figure 3 categorizes its operations by function.
- The WaveNet conditioning network converts low-frequency linguistic features into the native audio frequency.
- The autoregressive process predicts each next audio sample from current conditioning and a context of R previous samples.
- Figure 3 color-codes inputs, convolutions and QRNNs, unary operations and softmax, binary operations, and reshapes, transposes, and slices.
A.1. Auto-regressive WaveNet
The autoregressive WaveNet uses residual convolutional layers, skip projections, and a conditioning network that upsamples linguistic features to audio rate. Upsampling and sampling choices strongly affect pronunciation and output quality.
- The autoregressive network is parameterized by layers, skip channels, and residual channels, with residual connections preserving fixed channel dimensionality.
- Audio is quantized to 256 µ-law values and embedded through an initial convolution to form the first residual-stack input.
- Hidden states are concatenated, projected to skip channels, and passed through ReLU layers to produce the output distribution.
- Every timestep is conditioned on phoneme, stress, and fundamental-frequency features because unconditioned WaveNet lacks sufficient long-range information for recognizable words.
- The conditioning network uses bidirectional QRNN layers and repetition to upsample low-frequency linguistic features to the native audio frequency.
- The model is highly sensitive to upsampling: many convergent conditioning networks still produce phoneme mispronunciations.
- Including phoneme context is crucial for transposed-convolution upsampling and remains important for QRNN-based upsampling because omission causes subset mispronunciations.
- Among sampling methods, only direct sampling produces high-quality converged outputs, while temperature sampling becomes significantly worse after convergence.
A.5. Training
Training begins with noisy outputs and progressively increases the speech signal, while autoregressive errors can create audible disturbances early in training. The loss combines duration, voicing, and fundamental-frequency objectives, and inference uses rational nonlinear approximations.
- The signal-to-noise ratio increases throughout training as white noise decreases and the speech signal becomes dominant.
- Rare autoregressive sampling mistakes can compound into brief periods of loud noise, especially early in training, but do not occur in converged models.
- The phoneme loss uses tradeoff-weighted objectives for duration, voiced-probability cross-entropy, and time-dependent fundamental frequency.
- Inference replaces exact neural nonlinearities with high-accuracy rational approximations.
- A fourth-order polynomial approximation produces accurate tanh and sigmoid values when used in the specified formulas.
C.2. ex approximation
The system approximates the exponential function by manipulating IEEE 754 floating-point representations and applying a rational approximation. This implementation achieves a maximum error of 2.4 × 10^-5 for x ∈ (−∞, 0].
- C.2. ex approximation: The approximation represents 2^x through its 32-bit floating-point exponent and fraction bits.The integer and fractional portions of x map to the exponent and fraction fields of 2^x.
- C.2. ex approximation: A rational approximation estimates g(z) = 2^z − z over z ∈ [0, 1).Interpreting the resulting byte pattern back as a 32-bit float approximates 2^x.
- C.2. ex approximation: 2.4 × 10^-5 is the maximum error for x ∈ (−∞, 0].
D. Persistent GPU Kernels
Persistent GPU kernels partition WaveNet across streaming multiprocessors and coordinate sequential sample generation within one kernel. The design supports high-throughput autoregressive inference but is constrained by register capacity, synchronization, and limited flexibility.
- D. Persistent GPU Kernels: The inference process must generate one sample every 61 µs, motivating a single CUDA kernel with weights loaded across the GPU register files.High kernel-launch latency and the cost of reading small matrices make separate kernel launches impractical.
- D. Persistent GPU Kernels: The model is partitioned across 24 SMs, with SM1–SM20 storing adjacent pairs of residual-stack layers.Each SM generates hidden states and an output for its assigned layers.
- D. Persistent GPU Kernels: Sequential round-robin execution is simulated with 23 thread blocks that spin on locks stored in global memory and cached in L2.Data is transferred between SMs through global memory writes that bypass L1, followed by a memory fence.
- D. Persistent GPU Kernels: The partitioning scheme works only for specific values of l, r, and s, and parameter changes require writing a new kernel.Fixed register-file sizes and costly inter-SM communication make weight-matrix splitting difficult to change.
- D. Persistent GPU Kernels: GPU kernels are slower than CPU kernels because SM synchronization uses expensive busy-waiting and compiler register spills reach L1 cache.Limited WaveNet parallelism also exposes GPU memory-access latencies.
- D. Persistent GPU Kernels: 55 × 10^9 FLOPs are required per second of synthesis under the stated model parameters and a 16384 Hz sampling frequency.The performance model counts the arithmetic cost of generating each autoregressive audio sample.