Source-linked AI summary
Efficient Neural Audio Synthesis
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, Koray Kavukcuoglu
TL;DR
Efficient sampling remains difficult for sequential models because their serial structure can make high-quality audio generation slow. The paper combines WaveRNN, weight sparsity, and subscaling to reduce sampling costs while preserving output quality, achieving GPU acceleration, real-time mobile-CPU synthesis, and 16 samples per step.
Problem
Sequential sampling can become impractical for high-fidelity audio because generation is strictly serial and its time grows with sequence length, model operations, computation, and overhead.
Method
The paper develops a single-layer WaveRNN with dual softmax, prunes its weights into sparse networks, and uses subscaling to batch conditioned subsequences.
Results
Large sparse WaveRNNs outperform small dense networks at fixed parameter count, while Subscale WaveRNN generates B = 16 samples per step without loss in audio fidelity.
Takeaways & Limitations
The methods enable high-fidelity sequential audio synthesis in real time on off-the-shelf mobile CPUs and provide multiple orthogonal routes to faster sampling.
Abstract
from arXiv · showhide
Sequential models achieve state-of-the-art results in audio, visual and textual domains with respect to both estimating the data distribution and generating high-quality samples. Efficient sampling for this class of models has however remained an elusive problem. With a focus on text-to-speech synthesis, we describe a set of general techniques for reducing sampling time while maintaining high output quality. We first describe a single-layer recurrent neural network, the WaveRNN, with a dual softmax layer that matches the quality of the state-of-the-art WaveNet model. The compact form of the network makes it possible to generate 24kHz 16-bit audio 4x faster than real time on a GPU. Second, we apply a weight pruning technique to reduce the number of weights in the WaveRNN. We find that, for a constant number of parameters, large sparse networks perform better than small dense networks and this relationship holds for sparsity levels beyond 96%. The small number of weights in a Sparse WaveRNN makes it possible to sample high-fidelity audio on a mobile CPU in real time. Finally, we propose a new generation scheme based on subscaling that folds a long sequence into a batch of shorter sequences and allows one to generate multiple samples at once. The Subscale WaveRNN produces 16 samples per step without loss of quality and offers an orthogonal method for increasing sampling efficiency.
1. Introduction
The paper targets the serial sampling bottleneck in sequential models by reducing per-sample operations, implementation overhead, computation, and sequence length while preserving audio quality. It introduces WaveRNN, sparse WaveRNN, and subscaling methods for faster text-to-speech generation.
- Sampling bottleneck: Sequential models sample strictly serially, making generation slow when targets, layer counts, computation, or operation-launch overhead are large.High-fidelity audio contains 24,000 16-bit samples per second, while deep architectures such as WaveNet require many operations per sample.
- WaveRNN: WaveRNN uses a single recurrent layer and dual softmax output to match WaveNet-quality speech with N = 5 sequential matrix-vector products per 16-bit sample.WaveNet uses 60 matrix-vector products per sample, compared with WaveRNN's five.
- WaveRNN: 96,000 16-bit samples per second corresponds to 4× real-time 24kHz audio on an Nvidia P100 GPU, versus roughly 0.3× real time for the best WaveNet GPU kernel.Custom GPU operations reduce sampling overhead in the WaveRNN implementation.
- Sparse WaveRNN: For a fixed parameter count, large sparse WaveRNNs significantly outperform small dense WaveRNNs, with the relationship holding beyond 96% sparsity.Weight pruning reduces the computation required for sampling.
- Sparse WaveRNN: Sparse WaveRNN supports real-time, high-quality audio synthesis on off-the-shelf mobile CPUs despite much lower computation and memory bandwidth than GPUs.The mobile CPU has three orders of magnitude less computation and two orders of magnitude less memory bandwidth than a GPU in the reported comparison.
- Subscaling: Subscaling folds a long sequence into shorter conditioned sub-tensors, enabling B = 16 samples per step without loss in audio fidelity.A fused Subscale WaveRNN reaches 10× real-time sampling on an Nvidia P100 GPU.
2. Wave Recurrent Neural Networks
WaveRNN uses a compact recurrent architecture and dual softmax output to reduce per-sample operations while preserving audio quality. Custom GPU execution further addresses launch and memory overhead, enabling faster-than-real-time synthesis.
- Architecture: N = 5 operations per sample for WaveRNN replaces N = 60 for WaveNet with a 16-bit DLM output.
- GPU sampling: A single persistent GPU operation reduces launch overhead and keeps parameters in registers throughout sampling.The entire sampling process for an utterance is executed as one GPU operation.
3. Sparse WaveRNN
Sparse WaveRNN reduces sampling computation by pruning weights while retaining large hidden states. At fixed parameter count, increasing sparsity can improve fidelity, and structured sparse kernels support mobile implementation.
- Pruning: Sparse WaveRNN prunes weight matrices to reduce computation while retaining a large recurrent state.
- Pruning: Every 500 training steps, the smallest-magnitude weights are removed and target sparsity is gradually increased during training.The pruning schedule begins at step 1000, runs for 200k steps, and training lasts 500k steps.
- Structured sparsity: Benchmarks confirm greater speed for 16 × 1 blocks than for 4 × 4 blocks.
- Mobile implementation: Sparse matrix-vector operations are implemented on mobile CPUs using 16-bit weights converted to 32-bit before computation.
4. Subscale WaveRNN
Subscale WaveRNN reshapes a long sequence into interleaving lower-rate sub-tensors, generating them sequentially while batching samples within each sub-tensor. This increases sampling parallelism without requiring local independence assumptions.
- Batched generation: Subscaling trades a small constant number of distant dependencies for generating B samples per step.This avoids the local-independence assumption used by some prior parallel-generation methods.
- Dependency scheme: Subscaling folds a tensor of scale L into B interleaving sub-tensors of scale L/B.For 24kHz audio with B = 16, each sub-tensor corresponds to a 1.5kHz utterance.
- Dependency scheme: Sub-tensors are generated in order, with each conditioned on previously generated sub-tensors.The dependency ordering remains equivalent to the standard joint-distribution factorization.
- Batched generation: The conditioning network uses a finite future horizon F, allowing the next sub-tensor to begin after the first F samples of the previous one.
- Fused variant: Fused Subscale WaveRNN 2× generates 32 bits per step at 10× real time, with only a small quality drop.Audio quality drops quickly for fusion factors beyond 2×.
5. Experiments
Experiments evaluate WaveRNN quality, sparse-model trade-offs, mobile inference, and Subscale WaveRNN batching for efficient audio synthesis.
- WaveRNN Quality Evaluation & Speed: Human A/B tests find no significant quality difference between WaveRNN-896 and WaveNet, and between WaveRNN-2048 and WaveNet.
- WaveRNN Quality Evaluation & Speed: WaveRNN-896 achieves NLL 5.42 and MOS 4.37 ± 0.073, with throughput of 96,000 samples per second for batch size 1.The persistent GPU operations are most efficient for this model.
- Sparse WaveRNN Quality Evaluation & Speed: At fixed parameter count and sampling time, increasing sparsity improves model fidelity beyond 98% sparsity, with larger hidden layers.Higher sparsity monotonically implies lower NLL in the reported experiments.
- Sparse WaveRNN Quality Evaluation & Speed: 4 × 4 blocks generally provide the best NLL, whereas 16 × 1 blocks provide a speed advantage over other block configurations.Both block types outperform unstructured sparsity at low sparsity levels, but unstructured sparsity continues improving at higher levels.
- Sparse WaveRNN Quality Evaluation & Speed: Mobile benchmarks use Snapdragon 808 and 835 CPUs to estimate Sparse WaveRNN sampling speed from sparse matrix-vector multiplication and nonlinearity measurements.The evaluation targets real-time on-device audio synthesis on widely available mobile CPUs.
- Subscale WaveRNN Quality Evaluation: Subscale WaveRNN with B = 8 generates 8 samples per step and reaches MOS 4.39, equivalent to the baseline WaveRNN-896.The experiments also evaluate B = 16 for interleaved generation.
- Subscale WaveRNN Quality Evaluation: Batching by factors of 8 or 16 increases throughput flexibility and can support generation across multiple devices.The passage states that multi-device generation can in principle provide linear speed-up over one device.
6. Conclusion
The paper introduces WaveRNN, sparse WaveRNN inference, and Subscale WaveRNN to improve efficient high-fidelity audio generation. These methods reduce operations, computation, or serial dependence while preserving output quality and enabling mobile-CPU synthesis.
- WaveRNN is a recurrent model for high-fidelity audio with a high-performance GPU implementation.
- Large sparse models provide better quality than small dense models with the same number of parameters.
- High-fidelity audio generation is achievable on widely available low-power mobile CPUs.
- The Subscale dependency scheme generates many samples per step while preserving the original model’s output quality.
- The methods’ underlying ideas are not specific to audio, and sparse-model results have implications for inference in other neural networks.