Source-linked AI summary
Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, Lei Xie
TL;DR
MelGAN-based waveform generation has limited efficiency for real-time TTS applications. The paper improves MelGAN with a larger receptive field, multi-resolution STFT loss, pre-training, and multi-band processing, producing a smaller model that retains audio quality while running 7 times faster.
Problem
Low efficiency limits waveform-generation models' practical use in efficiency-sensitive and real-time TTS applications.
Method
The paper combines a larger generator receptive field, multi-resolution STFT loss with pre-training, and multi-band processing that predicts sub-band signals and synthesizes them into full-band audio.
Results
7 times faster than basic MelGAN and FB-MelGAN, MB-MelGAN retains comparable generation performance with a much smaller architecture and better real-time factor.
Takeaways & Limitations
The proposed improvements substantially improve speech-generation quality, training stability, and efficiency, while TTS experiments support their effectiveness.
Abstract
from arXiv · showhide
In this paper, we propose multi-band MelGAN, a much faster waveform generation model targeting to high-quality text-to-speech. Specifically, we improve the original MelGAN by the following aspects. First, we increase the receptive field of the generator, which is proven to be beneficial to speech generation. Second, we substitute the feature matching loss with the multi-resolution STFT loss to better measure the difference between fake and real speech. Together with pre-training, this improvement leads to both better quality and better training stability. More importantly, we extend MelGAN with multi-band processing: the generator takes mel-spectrograms as input and produces sub-band signals which are subsequently summed back to full-band signals as discriminator input. The proposed multi-band MelGAN has achieved high MOS of 4.34 and 4.22 in waveform generation and TTS, respectively. With only 1.91M parameters, our model effectively reduces the total computational complexity of the original MelGAN from 5.85 to 0.95 GFLOPS. Our Pytorch implementation, which will be open-resourced shortly, can achieve a real-time factor of 0.03 on CPU without hardware specific optimization.
1. INTRODUCTION
The paper targets the low inference efficiency of autoregressive waveform models in real-time TTS and proposes multi-band MelGAN as a faster, high-quality alternative. It improves MelGAN with a larger receptive field, multi-resolution STFT loss, pre-training, and multi-band processing.
- 1. INTRODUCTION: Autoregressive waveform models generate samples serially, limiting their practical use in efficiency-sensitive and real-time TTS applications.
- 1. INTRODUCTION: Non-autoregressive models offer highly parallelizable waveform generation and can be substantially faster than autoregressive counterparts.
- 1. INTRODUCTION: GAN-based models such as MelGAN, Parallel WaveGAN, and GAN-TTS provide non-autoregressive approaches to fast audio generation.
- 1. INTRODUCTION: MB-MelGAN expands the generator receptive field, replaces feature matching with multi-resolution STFT loss, uses pre-training, and introduces multi-band processing.The generator predicts sub-band signals that are combined into full-band audio.
- 1. INTRODUCTION: 1.91M parameters and 0.95 GFLOPS characterize the proposed MB-MelGAN in the reported introduction results.The cited passage contrasts its 0.95 GFLOPS complexity with 7.6 GFLOPS for the improved full-band comparison.
2. THE MODEL
The model retains MelGAN’s adversarial vocoder framework while replacing its feature-matching objective and extending generation to multiple frequency bands. A shared generator predicts sub-bands, which are synthesized into full-band audio for discrimination and multi-resolution STFT training.
- 2. THE MODEL: The proposed generator takes mel-spectrograms and predicts signals in multiple frequency bands instead of directly generating full-band audio.Each sub-band is upsampled and passed through synthesis filters before the signals are summed.
- 2. THE MODEL: MelGAN upsamples mel-spectrograms with transposed convolutions and uses dilated residual blocks to enlarge the generator receptive field.Multiple discriminators operate at different audio scales to capture fine-grained structures at different frequency levels.
- 2. THE MODEL: The paper replaces MelGAN’s feature matching loss with multi-resolution STFT loss to measure differences between real and generated speech more effectively.The multi-resolution objective averages STFT losses computed with different FFT, window, and hop-size parameters.
- 2. THE MODEL: MB-MelGAN applies multi-resolution STFT loss at both full-band and sub-band scales.Sub-band targets come from an analysis filter, while synthesized sub-bands are combined for full-band loss calculation.
- 2. THE MODEL: Pre-training initializes the generator with multi-resolution STFT loss before alternating discriminator and generator training.The procedure loops discriminator and generator updates until the complete model converges.
3. EXPERIMENTS
Experiments evaluate model details, training strategies, waveform generation, complexity, and TTS quality. The results show that the proposed improvements and multi-band processing improve quality, stability, and generation efficiency.
- Experimental setup: 12 hours of 16 kHz Chinese female-speaker audio were used, with 20 sentences held out for testing and MOS as the evaluation measure.Mel-spectrograms used 50 ms frames, 12.5 ms shifts, and 1024-point Fourier transforms.
- Model details: 81 timesteps versus 27 in basic MelGAN is the receptive field used in each residual dilated convolution stack.The stack uses four layers with dilation 1, 3, 9, and 27 and kernel size 3.
- Training strategy: About one-third of basic MelGAN training time was required after replacing feature matching with multi-resolution STFT loss, while pre-training reduced it to about two-thirds.Increasing the receptive field then produced the best MOS among the evaluated full-band systems and reduced audible jitter and metallic artifacts.
- Training strategy: 4.22 MOS was obtained by MB-MelGAN with full-band multi-resolution STFT loss, while combining sub-band and full-band losses further improved MOS and training stability.Multi-band processing also reduced training time by about one-half compared with full-band models.
- Evaluation results: 4.35 versus 3.98 in MOS was reported for improved full-band MelGAN versus basic MelGAN, while MB-MelGAN generated speech about 7 times faster.MB-MelGAN retained comparable generation performance with a smaller architecture and better real-time factor; the TTS evaluation found improved versions closer to real recordings.
4. CONCLUSION
The paper improves MelGAN quality and proposes a smaller, faster multiband version that retains audio quality. Text-to-speech experiments support these improvements.
- The improved MelGAN achieves significant quality improvement in speech generation.
- Multiband processing produces a smaller, faster MelGAN while retaining the same level of audio quality.The model runs 7 times faster than the original.
- Text-to-speech experiments justify the proposed improvements.