Source-linked AI summary
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, Aaron Courville
TL;DR
Generating coherent raw audio waveforms with GANs has been challenging. MelGAN introduces architectural changes and training techniques for non-autoregressive conditional waveform synthesis, achieving high-quality results across audio tasks with substantially faster inference. Its scope is constrained by time-aligned conditioning and other practical requirements.
Problem
Generating coherent raw audio waveforms with GANs has been challenging, despite GANs' progress in other domains.
Method
MelGAN uses a non-autoregressive feed-forward convolutional GAN with mel-spectrogram conditioning, window-based discriminators, and architectural changes for waveform synthesis.
Results
MelGAN produces high-quality waveform generation results across speech synthesis, music translation, and unconditional music synthesis while providing substantially faster inference than competing alternatives.
Takeaways & Limitations
MelGAN can serve as a lightweight, fast decoder for higher-level audio tasks and can replace autoregressive decoders with only slight quality degradation in the reported experiments.
Takeaways & Limitations
The model requires time-aligned conditioning information and is designed for cases where output length is a factor of input length.
Abstract
from arXiv · showhide
Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.
1 Introduction
Raw-audio generation is difficult because high temporal resolution and multi-timescale structure make direct modelling challenging. MelGAN addresses this gap with a non-autoregressive GAN-based waveform generator that targets fast, parallel audio synthesis.
- Motivation: Raw audio is difficult to model because it contains at least 16,000 samples per second and dependencies across different timescales.These properties motivate lower-resolution intermediate representations that preserve enough information for faithful inversion.
- Motivation: Mel-spectrogram inversion is the second stage of audio modelling, transforming an intermediate representation back into raw audio.The paper focuses on this stage rather than modelling the intermediate representation from text.
- Prior approaches: Autoregressive models generate realistic audio but are inherently slow because samples must be produced sequentially.WaveNet, SampleRNN, and WaveRNN represent major autoregressive approaches discussed in the paper.
- Prior approaches: Non-autoregressive models are much faster because their computations are highly parallelizable and exploit modern deep-learning hardware.Existing approaches include distillation into flow-based convolutional student models and other parallel decoders.
- MelGAN: MelGAN introduces a non-autoregressive feed-forward convolutional GAN architecture for raw waveform generation without additional distillation or perceptual loss functions.The authors report applications in music translation, text-to-speech, and unconditional music synthesis, with substantial speed advantages over competing mel-spectrogram inversion models.
2 The MelGAN Model
MelGAN uses a fully convolutional generator and multi-scale, window-based discriminators to invert mel-spectrograms into raw waveforms. Architectural choices target long-range coherence, artifact reduction, efficient adversarial training, and fast inference.
- Generator: MelGAN maps a mel-spectrogram to a raw waveform with a fully convolutional feed-forward generator and no global noise vector.The generator upsamples the 256× lower-resolution input using transposed convolutions followed by residual dilated convolution blocks.
- Generator: Residual blocks with dilated convolutions increase receptive-field overlap between distant output timesteps, encouraging long-range audio correlation.They are inserted after each upsampling layer, and the receptive field grows exponentially with the number of dilated layers.
- Generator: 256× upsampling uses four stages with ratios 8×, 8×, 2×, and 2×, while each residual stack uses dilations 1, 3, and 9.The architecture uses transposed-convolution kernel sizes twice the stride and gives each residual stack a 27-timestep receptive field.
- Generator: The generator chooses transposed-convolution kernel sizes as multiples of stride to reduce repeated patterns associated with audible high-frequency hissing.This addresses checkerboard-like artifacts without using the PhaseShuffle layer described in prior work.
- Discriminator: Three discriminators operate on raw audio and versions downsampled by factors of 2 and 4, biasing them toward different frequency ranges.The downsampled discriminators lack access to high-frequency components and therefore focus on lower-frequency features.
- Discriminator: Window-based discriminators classify overlapping local audio chunks, helping preserve coherence across patches while reducing parameters and supporting variable-length audio.Grouped convolutions enable large kernels with fewer parameters, and feature matching minimizes L1 distances between real and synthetic discriminator feature maps.
- Training objective: The model combines hinge-loss adversarial training with feature matching and uses λ = 10 in the final generator objective.The feature-matching term compares discriminator feature maps rather than applying a loss directly in raw-audio space.
- Efficiency: 2500kHz on GTX1080 Ti and 50kHz on CPU make MelGAN more than 10× and 25× faster, respectively, than the fastest competing models.The comparison uses the same hardware across models, and the architecture is also described as significantly smaller in parameter count.
3 Results
MelGAN was evaluated across speech synthesis, mel-spectrogram inversion, music translation, and unconditional music generation. The reported results indicate competitive audio quality, generalization to unseen speakers, and substantial speed advantages in music translation.
- 3.1 Ground truth mel-spectrogram inversion: MelGAN is comparable in quality to state-of-the-art high-capacity WaveNet-based models such as WaveNet and WaveGlow for mel-spectrogram inversion.
- 3.1 Ground truth mel-spectrogram inversion: MelGAN generalizes to completely new speakers outside the training set and learns a speaker-invariant mapping from mel-spectrograms to raw waveforms.
- 3.3 Non autoregressive decoder for music translation: 160 milliseconds are required to translate 1 second of music audio on an RTX2080Ti GPU, making the MelGAN-augmented network about 2500 times faster than the original model.
- 3.4 Non-autoregressive decoder for VQ-VAE: Replacing the VQ-VAE autoregressive WaveNet decoder with MelGAN provides an adversarially learned decoder for unconditional piano music generation.The adapted VQ-VAE uses a local encoder with a 64× temporal downsampling and a codebook of 512 quantized vectors, plus a global continuous latent vector.
4 Conclusion and future work
The authors introduce a GAN architecture for conditional audio synthesis and report qualitative and quantitative evidence for its effectiveness and generality. They identify conditioning alignment and sequence-length relationships as important limitations, while noting feature-matching and unconditional synthesis constraints.
- MelGAN is a GAN architecture tailored for conditional audio synthesis, with qualitative and quantitative results supporting the proposed methods.
- The model is lightweight, trains quickly on a single desktop GPU, and offers fast inference.
- The model requires time-aligned conditioning information and assumes output length is a factor of input length.
- Feature matching with paired ground-truth data can be infeasible, and unconditional synthesis requires another method to learn conditioning variables.
Appendix A Model Architecture
The appendix presents the generator and discriminator architecture for mel-spectrogram inversion through an architecture table and a residual stack figure.
- Table 6 specifies the generator and discriminator architecture for the mel-spectrogram inversion task.
- Figure 4 presents the residual stack architecture.
Appendix B Hyper-parameters and Training Details
The appendix reports shared optimization and training settings, including batch size, Adam parameters, and the feature-matching loss coefficient.
- All experiments used batch size 16 and Adam with learning rate 0.0001, β1 = 0.5, and β2 = 0.9 for both models.
- The feature-matching loss coefficient was set to 10.
- VQGAN experiments used a 256-dimensional global latent vector and clamped the KL term below by 1.0 to avoid posterior collapse.
Appendix C Evaluation Method - MOS
The MOS evaluation compared model-generated and original samples using blind ratings from 200 individuals, with each participant scoring 15 randomly selected samples.
- MOS tests compared samples generated by different models with original samples, using generated samples that were unseen during training.
- 200 individuals blindly rated random subsets of 15 samples on a 1-to-5 scale.