Source-linked AI summary
LPCNet: Improving Neural Speech Synthesis Through Linear Prediction
Jean-Marc Valin, Jan Skoglund
TL;DR
Neural speech synthesis has often required powerful GPUs for real-time operation, creating a deployment challenge for lower-power devices. LPCNet combines linear prediction with recurrent neural networks to shift spectral-envelope modeling away from the neural network. The resulting system achieves higher quality than WaveRNN at equal complexity, with high-quality synthesis around 2.8 GFLOPS.
Problem
Real-time neural speech synthesis can require GPU-scale computation, motivating lower-complexity models for devices with limited processing power and battery capacity.
Method
LPCNet combines linear prediction with WaveRNN so the neural network can devote most of its capacity to modeling spectrally flat excitation.
Results
2.8 GFLOPS total complexity supports real-time synthesis on a single Apple A8 core or 20% of a 2.4 GHz Intel Broadwell core.
Takeaways & Limitations
LPCNet quality significantly exceeds WaveRNN+ at equal complexity, or achieves the same quality at significantly reduced complexity.
Takeaways & Limitations
Generated samples retain roughness from noise between pitch harmonics; post-denoising was identified as a possible remedy but was not investigated.
Abstract
from arXiv · showhide
Neural speech synthesis models have recently demonstrated the ability to synthesize high quality speech for text-to-speech and compression applications. These new models often require powerful GPUs to achieve real-time operation, so being able to reduce their complexity would open the way for many new applications. We propose LPCNet, a WaveRNN variant that combines linear prediction with recurrent neural networks to significantly improve the efficiency of speech synthesis. We demonstrate that LPCNet can achieve significantly higher quality than WaveRNN for the same network size and that high quality LPCNet speech synthesis is achievable with a complexity under 3 GFLOPS. This makes it easier to deploy neural synthesis applications on lower-power devices, such as embedded systems and mobile phones.
1. INTRODUCTION
Neural speech synthesis can produce high-quality speech but often requires GPU-scale computation, motivating more efficient models for mobile and embedded devices. LPCNet addresses this by shifting spectral-envelope modeling to linear prediction, allowing a smaller neural network to focus on excitation.
- Motivation: High-quality neural speech synthesis has required tens of billions of floating-point operations per second for real-time operation on high-end GPUs.The target is synthesis on mobile phones and other devices with limited GPU capability and battery capacity.
- Motivation: Recent work seeks to reduce speech-synthesis complexity so speech can run on slower CPUs with limited battery impact.
- Motivation: Low-complexity vocoders efficiently model the spectral envelope with linear prediction, but their quality is limited because excitation remains difficult to model.
- Contribution: LPCNet removes spectral-envelope modeling from the neural network, dedicating most of its capacity to spectrally flat excitation.The paper starts from WaveRNN and develops further complexity reductions.
2. WAVERNN
WaveRNN predicts each output sample from the previous sample and conditioning parameters using a GRU and fully connected layers. Its output is a sampled discrete distribution, and block sparsity reduces recurrent computation while preserving vectorization.
- Architecture: WaveRNN takes the previous audio sample and conditioning parameters as input, then generates a discrete probability distribution for the output sample.
- Architecture: The model uses a gated recurrent unit followed by two fully connected layers and a softmax activation.
- Output: The synthesized sample is obtained by sampling from the predicted distribution P(s_t).
- Efficiency: Block-sparse GRU matrices reduce complexity while retaining efficient vectorization, using non-zero blocks such as 4x4 or 16x1.
3. LPCNET
LPCNet combines frame-rate conditioning with a sample-rate network and uses linear prediction and pre-emphasis to make efficient 8-bit µ-law synthesis practical. Its architecture separates slowly varying features from per-sample prediction.
- Architecture: LPCNet uses a sample-rate network at 16 kHz and a frame-rate network processing 10-ms frames of 160 samples.
- Architecture: The synthesis input comprises 20 features: 18 Bark-scale cepstral coefficients and 2 pitch parameters.
- Architecture: The figure’s yellow frame-rate network is computed once per frame, while the blue sample-rate network operates throughout the frame and predicts samples using prior samples and linear-prediction coefficients.
- Architecture: The frame-rate network produces a 128-dimensional conditioning vector held constant throughout each frame.Two convolutional layers provide a receptive field of 5 frames before the conditioning vector is produced.
- Quantization: A first-order pre-emphasis filter with α = 0.85 shapes quantization noise and reduces its power at the Nyquist rate by 16 dB.This makes 8-bit µ-law output viable for high-quality synthesis.
3.3. Linear Prediction
LPCNet uses linear prediction to model the vocal-tract response and lets the neural network focus on excitation. Prediction coefficients are derived from cepstral features, while the network also receives past signal and prediction values.
- Rationale: The vocal-tract response can be represented by a simple all-pole linear filter, reducing the burden placed on neural speech-synthesis models.
- Prediction: The linear prediction p_t is computed from previous samples using Mth-order coefficients a_k for the current frame.
- Coefficient computation: Prediction coefficients are derived by converting the 18-band Bark-frequency cepstrum to a linear-frequency PSD, then using inverse FFT and Levinson-Durbin computation.
- Coefficient computation: Although cepstrum-derived LPC analysis is less accurate than input-signal analysis, its output effect is small because the network learns to compensate.
- Excitation prediction: The network predicts excitation rather than sample values and receives the previous excitation, previous signal, and current prediction as inputs.This slightly eases prediction and can reduce µ-law quantization noise because excitation generally has smaller amplitude than the pre-emphasized signal.
3.4. Output Layer
The output layer uses a DualFC structure that combines two fully connected tanh layers through an element-wise weighted sum to compute output probabilities efficiently.
- 3.4. Output Layer: DualFC combines two fully connected layers with an element-wise weighted sum to compute output probabilities without greatly enlarging the preceding layer.The layer uses weight matrices W1 and W2 with weighting vectors a1 and a2.
- 3.4. Output Layer: The two tanh layers approximate the two comparisons needed to determine whether a value falls within a µ-law quantization interval.Visualized trained weights support this intuition.
3.5. Sparse Matrices
LPCNet reduces recurrent-network complexity with block-sparse matrices designed to preserve efficient vectorization while maintaining accuracy.
- 3.5. Sparse Matrices: Block-sparse matrices replace general element-wise sparsity because fixed blocks preserve efficient vectorization.Training progressively zeros low-magnitude blocks until the target sparsity is reached.
- 3.5. Sparse Matrices: 16x1 non-zero blocks provide good accuracy while making matrix products easy to vectorize.The largest GRU uses these blocks.
- 3.5. Sparse Matrices: Including all diagonal terms avoids allocating full 16x1 blocks when only a diagonal element would otherwise be non-zero.Diagonal contributions remain easy to vectorize as element-wise multiplication.
3.6. Embedding and Algebraic Simplifications
LPCNet uses µ-law embeddings and algebraic precomputation to simplify non-recurrent inputs, making their computational cost negligible in the main GRU.
- 3.6. Embedding and Algebraic Simplifications: The embedding matrix maps each discrete µ-law level to a vector instead of scaling scalar samples to a fixed range.The learned embedding includes the function converting µ-law values to linear scale.
- 3.6. Embedding and Algebraic Simplifications: Precomputing U(u,s)E creates embedding tables that directly map the previous sample to each GRU gate’s non-recurrent update term.The same transformation is applied across gates and embedded inputs.
- 3.6. Embedding and Algebraic Simplifications: The frame-conditioning contribution g(·) = U(·)f is computed once per frame because f remains constant during that frame.This removes repeated computation of the conditioning contribution at sample rate.
- 3.6. Embedding and Algebraic Simplifications: These simplifications make the computational cost of all non-recurrent inputs to the main GRU negligible.The resulting calculations use GRU state updates and a probability output based on DualFC.
- 3.6. Embedding and Algebraic Simplifications: During noise injection, the prediction filter is applied to the noisy, quantized input, while excitation is the clean input minus the prediction.Noise is added in the µ-law domain.
3.7. Sampling from Probability Distribution
LPCNet modifies sampling to reduce excessive and impulse noise while preserving speech naturalness through pitch-dependent sharpening and probability thresholding.
- 3.7. Sampling from Probability Distribution: Probabilities below threshold T are set to zero and the distribution is renormalized to suppress impulse noise caused by low probabilities.The modified distribution applies the threshold after scaling.
- 3.7. Sampling from Probability Distribution: T = 0.002 provides a good trade-off between reducing impulse noise and preserving speech naturalness.The distribution is renormalized between the two processing steps and after thresholding.
3.8. Training Noise Injection
Training noise injection is tuned to make LPCNet robust to the mismatch between generated and training samples while avoiding artifacts caused by linear prediction.
- Training Noise Injection: Adding noise during training improves robustness to the mismatch between generated samples and training samples.Generated samples are more imperfect than training samples, which can amplify distortion during synthesis.
- Training Noise Injection: Noise added to the signal while training on clean excitation produces artifacts shaped by the synthesis filter.These artifacts resemble those associated with pre-analysis-by-synthesis vocoders.
- Training Noise Injection: Adding noise as shown in Fig. 2 teaches the network to minimize error in the signal domain despite predicting the residual.The network receives the same prediction used to compute the residual, paralleling analysis-by-synthesis in CELP and greatly reducing artifacts.
- Training Noise Injection: Noise is injected directly in the µ-law domain so its amplitude remains proportional to the signal amplitude.Its distribution ranges across training data from no noise to a uniform distribution in the [−3, 3] range.
4. EVALUATION
The evaluation measures LPCNet’s complexity and subjective quality in speaker-independent synthesis, comparing it with an improved WaveRNN baseline under matched complexity conditions.
- 4.1. Complexity: 2.8 GFLOPS is the estimated total complexity for LPCNet with NA = 384, NB = 16, Q = 256, and Fs = 16000.Real-time synthesis is achieved on one Apple A8 core or 20% of a 2.4 GHz Intel Broadwell core.
- 4.1. Complexity: 16 GFLOPS is reported for speaker-dependent FFTNet, while sparse mobile WaveRNN is estimated at around 10 GFLOPS.The WaveRNN estimate is an interpretation of data from the WaveRNN paper rather than an explicitly stated value.
- 4.2. Experimental Setup: The evaluation uses speaker-independent synthesis, the more challenging setting, with features computed directly from recorded speech to isolate vocoder quality.The cepstrum follows the band layout in, and pitch uses an open-loop cross-correlation search.
- 4.2. Experimental Setup: 4 hours of speech from a 21-language NTT database are used for training, excluding test speakers.Networks are trained for 120 epochs, or 230k updates, with batch size 64.
- 4.2. Experimental Setup: LPCNet is compared with WaveRNN+ across main-GRU sizes of 192, 384, and 640 units at density d = 0.1.The configurations match dense-equivalent GRU sizes of 61, 122, and 203 units, with NB = 16 in every case.
- 4.3. Quality Evaluation: LPCNet significantly exceeds WaveRNN+ in subjective quality at equal complexity, or achieves the same quality at reduced complexity.The comparison uses a MUSHRA-derived test with 8 utterances evaluated by 100 participants.
- 4.3. Quality Evaluation: Roughness caused by noise between pitch harmonics is the main audible artifact in both LPCNet and WaveRNN+ samples.Post-denoising is suggested as a possible remedy but was not investigated.
5. CONCLUSION
The paper concludes that combining linear prediction with neural synthesis improves the efficiency of speaker-independent speech synthesis, with possible application to speech synthesis and coding.
- 5. CONCLUSION: Combining linear prediction with neural synthesis improves the efficiency of speaker-independent speech synthesis.The conclusion identifies this combination as the main contribution.
- 5. CONCLUSION: The proposed model is believed to apply equally to text-to-speech and low-bitrate speech coding.This is presented as the authors’ supported scope for the model.
- 5. CONCLUSION: Future work will investigate whether longterm pitch prediction can further reduce complexity.The paper presents this as a future extension rather than a demonstrated result.