Source-linked AI summary
A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet
Jean-Marc Valin, Jan Skoglund
TL;DR
Very-low-bitrate parametric codecs struggle to model excitation, while neural speech synthesis has offered high quality at high computational cost. This paper presents LPCNet for real-time 1.6 kb/s speech coding, reporting higher quality than MELP and identifying clean, non-reverberant speech as its tested scope.
Problem
Very-low-bitrate parametric codecs have limited quality because speech excitation is difficult to model, while neural synthesis approaches can be too computationally complex for real-time use.
Source: §1 · p. 1Method
The paper applies LPCNet, combining linear prediction and sparse recurrent networks, to low-bitrate speech coding.
Source: §1 · p. 1, §5.1 · p. 3Results
At 1.6 kb/s, LPCNet exceeds the quality of existing low-bitrate vocoders such as MELP; unquantized LPCNet achieves slightly higher quality than Opus at 9 kb/s.
Source: §6 · p. 4, §5.3 · p. 4Takeaways & Limitations
The authors report a neural vocoder that operates in real time on general-purpose hardware and mobile devices.
Source: §6 · p. 4Evaluation has focused on clean, non-reverberant speech, and further work is needed to test and improve robustness to noise and reverberation.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Neural speech synthesis algorithms are a promising new approach for coding speech at very low bitrate. They have so far demonstrated quality that far exceeds traditional vocoders, at the cost of very high complexity. In this work, we present a low-bitrate neural vocoder based on the LPCNet model. The use of linear prediction and sparse recurrent networks makes it possible to achieve real-time operation on general-purpose hardware. We demonstrate that LPCNet operating at 1.6 kb/s achieves significantly higher quality than MELP and that uncompressed LPCNet can exceed the quality of a waveform codec operating at low bitrate. This opens the way for new codec designs based on neural synthesis models.
1. Introduction
Very-low-bitrate parametric codecs model speech spectra efficiently, but their quality is limited by the difficulty of modeling excitation. Neural synthesis offers high-quality speech generation, motivating simpler models that can run in real time at 1.6 kb/s.
1. Introduction
- Very-low-bitrate parametric codecs have limited quality because no simple model captures speech excitation, restricting their use above 3 kb/s.They model the spectral envelope with linear prediction, but excitation remains a challenge.Source: §1 · p. 1
- Neural synthesis approaches can produce high-quality speech from coded features, but prior approaches require complexity on the order of 100 GFLOPS [10].That complexity typically prevents real-time use without high-end hardware.Source: §1 · p. 1
- The paper targets a simpler, speaker-independent model for real-time communication on general-purpose and mobile hardware at 1.6 kb/s.The stated aim includes operation across speakers and languages.Source: §1 · p. 1
2. LPCNet Overview
LPCNet combines a frame-rate network with a sample-rate network that predicts conditional sample probabilities using prior speech and linear prediction. Sparse recurrent computation and precomputed contributions reduce runtime work.
2. LPCNet Overview
- LPCNet uses a frame-rate network to compute conditioning features every 10 ms and a sample-rate network to compute conditional sample probabilities.The frame-rate output stays constant through each frame while the sample-rate network operates.Source: §2 · p. 1, Figure 1 · p. 2
- The sample-rate network uses the previous speech sample, a 16th-order linear prediction, and the previously generated excitation.The predicted sample is subtracted from speech to obtain excitation.Source: §2 · p. 1
- Precomputed input contributions and once-per-frame conditioning leave recurrent matrices as the sample-rate network's remaining runtime computation.Discrete µ-law inputs allow lookup-based gate and state contributions.Source: §2 · p. 1
- Sparse recurrent matrices use nonzero blocks of 16x1, retaining 20% of hidden-state weights and 5% of reset and update gate weights.This yields an average of 10% of weights retained.Source: §2 · p. 1
3. Features and Quantization
The codec represents four 10-ms LPCNet frames in each 40-ms packet and quantizes pitch and cepstral features to meet a fixed bitrate. Pitch is tracked within subframes, while cepstral prediction reduces redundancy and limits packet-loss propagation.
3. Features and Quantization
- Each 40-ms packet represents four 10-ms frames and uses 64 bits for a constant bitrate of 1.6 kb/s.Its features include 18 cepstral coefficients, pitch period, and pitch correlation.Source: §3 · p. 2
3.1. Pitch
- Pitch errors matter because no residual corrects prediction errors, and LPCNet's ability to compensate is reduced when the cepstrum is quantized.Pitch is encoded over 62.5–500 Hz, with packet-average pitch represented on a logarithmic scale using 6 bits.Source: §3.1 · p. 2
- Pitch search uses 5-ms subframes because a single pitch estimate over a 40-ms packet does not handle pitch variation well.A Viterbi backtrack once per packet gives a consistent path across all eight subframes, though it does not guarantee the global optimum.Source: §3.1 · p. 2
3.2. Cepstrum
- The cepstrum uses prediction across packet vectors to reduce within-packet redundancy while limiting packet-loss error propagation to at most 30 ms.The first component of c4k+3 is independently quantized; c4k+1 uses c4k−1 and c4k+3 as predictors, with single-vector options improving transient and onset quantization.Source: §3.2 · p. 3
4. Training
Training addresses the mismatch between clean-speech training and synthesized-speech inference by injecting noise into the prediction loop. The training procedure also varies speech level and frequency response, then adapts only the frame-rate network for quantized features.
4. Training
- Laplace-distributed noise is added to the excitation inside the prediction loop to make signal and excitation noise consistent and improve robustness to training–inference mismatch.At inference, the sample-rate network operates on synthesized rather than ground-truth clean speech.Source: §4 · p. 3
- Training data augmentation varies speech level over 40 dB and varies frequency response according to the cited procedure [17].Source: §4 · p. 3
- Quantized-feature training starts from an unquantized model and adapts only the frame-rate network, leaving sample-rate weights unchanged.The authors report slightly better quality, faster training for new quantizers, and smaller storage when keeping different quantized models.Source: §4 · p. 3
5. Evaluation
The evaluation reports that 1.6 kb/s LPCNet outperforms MELP and generalizes to samples from recording conditions outside the NTT database. The decoder runs in real time on tested desktop and mobile hardware, and the model was trained using data from 21 languages.
5. Evaluation
- The source code is available under a BSD license, and the evaluation used commit 6fda6b7.Source: §5 · p. 3
5.1. Complexity and Implementation
- The decoder requires 20% of one 2.4 GHz Intel Broadwell core for real-time decoding; on a 2.84 GHz Snapdragon 855, it requires 31% of one core.The proposed system has a total complexity of 3 GFLOPS, including 2.3 GFLOPS for multiply-add operations and 0.6 GFLOPS for activation functions.Source: §5.1 · p. 3
5.3. Quality
- 1.6 kb/s LPCNet clearly outperforms MELP, and testing on samples outside the NTT database shows the model generalizes to other recording conditions.Figure 4 presents subjective MUSHRA results for both listening tests.Source: Figure 4 · p. 4, §5.3 · p. 4
5.2. Experimental Setup
- The model was trained on 4 hours of speech from a 21-language database, with testing speakers excluded from training.The original data was augmented to 14 hours; the unquantized network was trained for 120 epochs, and quantized-feature adaptation for 40 epochs.Source: §5.2 · p. 4
5.3. Quality
- The first listening test used NTT samples from four speakers excluded from training; the second tested samples from the Opus testvector set with 100 listeners per test.The second test measured performance on samples outside the NTT database; only English was formally tested, while French, Spanish, and Swedish were assessed informally.Source: §5.3 · p. 4
6. Conclusions
The paper presents a 1.6 kb/s LPCNet vocoder for real-time mobile communication, with quality exceeding existing low-bitrate vocoders such as MELP. It identifies robustness to noise and reverberation as work still needed and proposes exploring higher LPCNet bitrates and LPCNet-based enhancement of waveform codecs.
6. Conclusions
- The demonstrated 1.6 kb/s LPCNet vocoder supports real-time communication on mobile devices and exceeds the quality of existing low-bitrate vocoders such as MELP.The authors believe it is the first neural vocoder operating in real time on general-purpose hardware and mobile devices.Source: §6 · p. 4
- The evaluation focused on clean, non-reverberant speech; robustness to noise and reverberation still needs testing and improvement.Source: §6 · p. 4
- The authors propose exploring LPCNet at 2–6 kb/s and using encoded LSPs or excitation as inputs to improve low-bitrate waveform codecs.They suggest encoded excitation may help reduce artifacts when LPCNet acts as a neural post-filter.Source: §6 · p. 4