Source-linked AI summary
LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation
Zhen Zeng, Jianzong Wang, Ning Cheng, Jing Xiao
TL;DR
Waveform generation needs models that capture long-term dependencies while improving inference efficiency. The paper introduces location-variable convolution and LVCNet, which predict interval-specific kernels from acoustic conditioning and use LVCNet as a Parallel WaveGAN generator. On LJSpeech, synthesis is four times faster than base Parallel WaveGAN without degradation in sound quality.
Problem
Autoregressive waveform models have limited inference efficiency, while conventional waveform networks use unified convolution kernels despite time-dependent features.
Source: §1 · p. 1, §2.1 · p. 2Method
Location-variable convolution predicts different convolution kernels for waveform intervals from conditioning acoustic features, and LVCNet applies this method in a Parallel WaveGAN generator.
Source: §2.1 · p. 2, §1 · p. 2, §3.1.2 · p. 3Results
Four times faster inference than base Parallel WaveGAN was achieved on LJSpeech without degradation in sound quality.
Source: §4 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
In this paper, we propose a novel conditional convolution network, named location-variable convolution, to model the dependencies of the waveform sequence. Different from the use of unified convolution kernels in WaveNet to capture the dependencies of arbitrary waveform, the location-variable convolution uses convolution kernels with different coefficients to perform convolution operations on different waveform intervals, where the coefficients of kernels is predicted according to conditioning acoustic features, such as Mel-spectrograms. Based on location-variable convolutions, we design LVCNet for waveform generation, and apply it in Parallel WaveGAN to design more efficient vocoder. Experiments on the LJSpeech dataset show that our proposed model achieves a four-fold increase in synthesis speed compared to the original Parallel WaveGAN without any degradation in sound quality, which verifies the effectiveness of location-variable convolutions.
1. INTRODUCTION
Waveform generators need efficient inference without sacrificing speech quality. This paper proposes location-variable convolution and LVCNet, reporting four-fold faster synthesis than Parallel WaveGAN without sound-quality degradation.
1. INTRODUCTION
- Autoregressive waveform models provide strong quality but have limited inference efficiency for real-time applications.This motivates non-autoregressive waveform-generation approaches.Source: §1 · p. 1
- Existing alternatives include distillation, flow-based, GAN, and diffusion waveform generators, each involving training, model-size, or generation trade-offs.The introduction positions LVCNet among these approaches.Source: §1 · p. 1
- Four-fold faster synthesis than Parallel WaveGAN was achieved on LJSpeech without degradation in sound quality.The result supports the proposed model's effectiveness for waveform generation.Source: §1 · p. 1
- Location-variable convolution uses different kernels for different waveform intervals, with coefficients predicted from conditioning acoustic features.The method targets more efficient modeling of time-dependent waveform features.Source: §1 · pp. 1–2
- LVCNet applies location-variable convolutions to waveform generation and implements them as the generator in Parallel WaveGAN.The paper evaluates the resulting vocoder against Parallel WaveGAN.Source: §1 · p. 2
2. PROPOSED METHODS
The proposed method predicts location-specific convolution kernels from conditioning acoustic features and uses them to model waveform intervals. These convolutions are organized into LVCNet and integrated as a Parallel WaveGAN generator.
2.1. Location-Variable Convolution
- Location-variable convolution assigns each conditioning element a kernel set for its associated input interval, then splices the interval outputs.This replaces a single shared convolution-kernel configuration across all waveform locations.Source: §2.1 · p. 2, Figure 1 · p. 2
- The method applies gated activation with separate filter and gate kernels predicted for each input interval.The interval-specific kernels are denoted W_f^(i) and W_g^(i) in the described formulation.Source: §2.1 · p. 2
2.2. LVCNet
- LVCNet stacks location-variable convolution layers with increasing factorial dilation coefficients to expand the receptive field.It uses multiple LVCNet blocks, channel-conversion linear layers, and block-level residual connections.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
- The conditioning sequence feeds a kernel predictor that generates convolution coefficients for each LVC layer.The predictor uses residual linear layers with leaky ReLU activation, with output size determined by the coefficients required.Source: §2.2 · p. 3
2.3. Parallel WaveGAN with LVCNet
- In Parallel WaveGAN with LVCNet, the LVCNet generator transforms noise into waveform while retaining Parallel WaveGAN's discriminator, loss, and training strategy.The generator is conditioned on the mel-spectrum for a controlled comparison.Source: §2.3 · p. 3
- The kernel predictor first aligns mel-spectrum and waveform sequences with a 1D convolution, then uses stacked 1×1 convolutions to output location-variable kernels.The first LVCNet block omits its residual connection for improved model performance.Source: §2.3 · p. 3
3. EXPERIMENTS
Experiments on LJSpeech evaluate LVCNet against Parallel WaveGAN across dataset, model, subjective quality, speed, and TTS settings. LVCNet preserves comparable sound quality while substantially improving synthesis speed, including under reduced residual-channel configurations.
3.1.1. Database
- The evaluation used 12,600 training, 400 validation, and 100 test utterances from LJSpeech, with 22,050 Hz audio and 80-channel mel-spectrogram features.Synthetic audio and ground-truth audio were rated by 20 testers using headphones in a conventional MOS evaluation.Source: §3.1.1 · p. 3, §3.2.1 · p. 4
3.1.2. Model Details
- The LVCNet generator contained three blocks of ten LVC layers, eight residual channels, kernel size three, and a kernel predictor with one 1×5 and three 1×1 convolutional layers.The discriminator retained the original Parallel WaveGAN structure, and the experiments compared residual-channel settings of 4, 8, and 16 against 32, 64, and 128.Source: §3.1.2 · pp. 3–4
3.2.1. Evaluation
- Four-fold faster inference was achieved with almost the same results as Parallel WaveGAN, while unoptimized GPU synthesis reached approximately 300 MHz versus 35 MHz.Table 1 compares the proposed model with Parallel WaveGAN using CPU real-time factor measurements.Source: §3.2.1 · p. 4, Table 1 · p. 4
- LVCNet's performance degraded more slowly than Parallel WaveGAN as residual channels decreased, including configurations with 4 or 8 channels.The authors attribute this to convolution coefficients adjusted according to mel-spectrums.Source: §3.2.1 · p. 4
3.2.2. Text-to-Speech
- In feed-forward TTS systems using Transformer TTS and AlignTTS, LVCNet significantly improved synthesis speed without degrading sound quality relative to Parallel WaveGAN.Both systems predicted mel-spectrums that were then used by the two vocoders for MOS evaluation; Table 2 reports the comparison.Source: §3.2.2 · p. 4, Table 2 · p. 4
- LVCNet uses adjusted convolution kernels across time intervals to obtain more effective feature modeling and a faster vocoder without sound-quality degradation.The text connects this design to the mutual independence of acoustic features such as mel-spectrums.Source: §3.2.2 · p. 4
4. CONCLUSION
The paper proposes location-variable convolution for time-dependent waveform modeling and uses it in LVCNet as the Parallel WaveGAN generator. On LJSpeech, the resulting model is four times faster than baseline Parallel WaveGAN without sound-quality degradation.
4. CONCLUSION
- Four times faster inference was obtained than with the base Parallel WaveGAN model without any degradation in sound quality.The result is reported for experiments on the LJSpeech dataset.Source: §4 · p. 4