Source-linked AI summary
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
Jungil Kong, Jaehyeon Kim, Jaekyoung Bae
TL;DR
Speech synthesis needs waveform generators that are both fast and high fidelity, since autoregressive and flow-based models involve speed or model-size trade-offs. HiFi-GAN models speech periodicity with multi-period and multi-scale discriminators plus a multi-receptive-field generator, achieving high MOS, human-quality similarity, and fast inference across GPU, CPU, unseen-speaker, and end-to-end settings.
Problem
Existing speech-synthesis methods have not simultaneously matched the quality of autoregressive and flow-based models while improving sampling efficiency and memory usage.
Method
HiFi-GAN uses adversarial training with multi-period and multi-scale discriminators and generator residual blocks that model speech patterns at multiple periods and lengths.
Results
HiFi-GAN achieves higher MOS than WaveNet and WaveGlow, with a 0.92M-parameter version and a fastest version running 13.44 times faster than real-time on CPU.
Takeaways & Limitations
HiFi-GAN generalizes to unseen-speaker mel-spectrogram inversion and end-to-end speech synthesis, while its periodic-pattern discriminator materially contributes to perceptual quality.
Takeaways & Limitations
End-to-end quality depends on matching input representations; mismatches with Tacotron2 produced low-quality audio.
Abstract
from arXiv · showhide
Several recent work on speech synthesis have employed generative adversarial networks (GANs) to produce raw waveforms. Although such methods improve the sampling efficiency and memory usage, their sample quality has not yet reached that of autoregressive and flow-based generative models. In this work, we propose HiFi-GAN, which achieves both efficient and high-fidelity speech synthesis. As speech audio consists of sinusoidal signals with various periods, we demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality. A subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method demonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen speakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.
1 Introduction
Neural speech synthesis has advanced quality and efficiency, but autoregressive and flow-based waveform models retain important speed or size trade-offs. HiFi-GAN addresses these trade-offs by modeling periodic speech patterns with specialized discriminators and achieves high-quality, efficient synthesis across settings.
- 1 Introduction: Autoregressive WaveNet surpasses conventional methods in quality but generates one sample per operation, making high-temporal-resolution synthesis prohibitively slow.Flow-based models accelerate sampling through parallel waveform generation, although WaveGlow requires over 90 layers.
- 1 Introduction: Flow-based models improve sampling speed through parallel computation, but WaveGlow requires many parameters for its deep architecture.Parallel WaveNet improves synthesis speed to 1,000 times or more than its WaveNet teacher without quality degradation.
- 1 Introduction: HiFi-GAN targets both computational efficiency and sample quality by modeling periodic speech patterns with multi-period discriminators and parallel multi-receptive-field residual blocks.Its discriminator uses sub-discriminators focused on specific periodic parts of raw waveforms, while the generator observes patterns at multiple lengths.
- 1 Introduction: HiFi-GAN achieves higher MOS than WaveNet and WaveGlow while synthesizing human-quality speech at 3.7 MHz on a single V100 GPU.The model is also evaluated for unseen-speaker mel-spectrogram inversion and end-to-end speech synthesis.
- 1 Introduction: A tiny HiFi-GAN version uses 0.92M parameters, while its fastest version runs 13.44 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.The paper also provides audio samples and an open-source implementation for reproducibility and future work.
2 HiFi-GAN
HiFi-GAN combines a convolutional generator with multi-period and multi-scale discriminators to synthesize realistic speech efficiently. Its design targets both long-term dependencies and diverse periodic patterns, while additional losses stabilize training and improve fidelity.
- 2.2 Generator: HiFi-GAN uses a fully convolutional generator that upsamples mel-spectrograms until their temporal resolution matches raw waveforms.Transposed convolutions perform upsampling, and multi-receptive field fusion modules follow them.
- 2.2 Generator: MRF modules sum residual-block outputs with different kernel sizes and dilation rates to observe patterns of varied lengths in parallel.This design provides diverse receptive field patterns within the generator.
- 2.3 Discriminator: HiFi-GAN targets realistic speech by modeling both long-term dependencies and diverse periodic patterns in sinusoidal audio.The paper notes that phoneme durations can exceed 100 ms, correlating more than 2,200 adjacent waveform samples.
- 2.3 Discriminator: MPD uses sub-discriminators that process equally spaced waveform samples at periods [2, 3, 5, 7, 11] to capture different periodic structures.Each input is reshaped into 2D data and processed with convolutions restricted to width 1.
- 2.3 Discriminator: MSD complements MPD by evaluating audio consecutively at raw, ×2 average-pooled, and ×4 average-pooled scales.The two discriminator families therefore assess disjoint periodic samples and consecutively evaluated smoothed sequences.
- 2.4 Training Loss Terms: Training combines LS-GAN objectives with mel-spectrogram and feature-matching losses for adversarial stability, training efficiency, and generated-audio fidelity.The mel-spectrogram loss is an L1 distance between generated and ground-truth mel-spectrograms; feature matching compares discriminator features.
3 Experiments
Experiments compare HiFi-GAN with publicly available speech-synthesis systems on single-speaker and unseen-speaker settings. They assess audio quality, synthesis speed, and efficiency-quality trade-offs across generator variants.
- 3 Experiments: HiFi-GAN is evaluated on LJSpeech, a 13,100-clip single-speaker dataset totaling approximately 24 hours at a 22 kHz sample rate.Comparisons use publicly available WaveNet, WaveGlow, and MelGAN implementations with provided pretrained weights.
- 3 Experiments: Generality is evaluated through mel-spectrogram inversion on VCTK, containing approximately 44,200 clips from 109 English speakers with varied accents.The approximately 44-hour dataset is downsampled from 44 kHz to 22 kHz.
- 3 Experiments: Audio quality is measured with five-point MOS tests conducted through Amazon Mechanical Turk, with 95% confidence intervals.Samples are randomized, each clip is rated once per rater, and clips are volume-normalized.
- 3 Experiments: Synthesis speed is measured on a single NVIDIA V100 GPU and a 2.6 GHz Intel i7 MacBook Pro CPU using 32-bit floating-point operations without optimization.The setup follows recent efficiency-evaluation practices.
- 3 Experiments: Three generator variants, V1, V2, and V3, are evaluated with a shared discriminator configuration to examine the trade-off between synthesis efficiency and sample quality.V2 reduces the hidden dimension to 128 while preserving V1's receptive fields.
4 Results
HiFi-GAN’s evaluations show strong synthesis quality and speed, component-level benefits from MPD, MRF, and mel-spectrogram loss, generalization to unseen speakers, and effective fine-tuned end-to-end synthesis.
- 4.1 Audio Quality and Synthesis Speed: All HiFi-GAN variations scored higher than the compared models; V1 achieved the highest MOS, while V3 reached 13.44 times real-time on CPU.V1 was 0.09 MOS below ground truth; V3 used 1,186 times real-time speed on a single V100 GPU and had perceptual quality similar to MoL WaveNet.
- 4.2 Ablation Study: MPD removal caused a significant perceptual-quality decrease, while removing MSD caused a smaller but noticeable degradation.The ablation results also found that retaining only one wide-receptive-field residual block per MRF module worsened performance.
- 4.2 Ablation Study: Mel-spectrogram loss improved synthesis quality and made quality improvements more stable during training.The ablation study evaluated MPD, MRF, and mel-spectrogram loss using the V3 generator.
- 4.2 Ablation Study: MelGAN with MPD outperformed the original MelGAN by 0.47 MOS, a statistically significant improvement.This experiment tested whether the multi-period discriminator benefited another GAN-based speech synthesizer.
- 4.3 Generalization to Unseen Speakers: The three HiFi-GAN generator variations scored 3.77, 3.69, and 3.61 on unseen-speaker mel-spectrogram inversion and all outperformed autoregressive and flow-based models.The results were obtained from 50 utterances by nine VCTK speakers excluded from training.
- 4.4 End-to-End Speech Synthesis: MelGAN and MoL WaveNet were excluded from one comparison because preprocessing differences, including frequency clipping, produced mismatched input representations.The mismatch led to low-quality audio when those systems were combined with Tacotron2.
- 4.4 End-to-End Speech Synthesis: In end-to-end synthesis, all proposed models initially outperformed WaveGlow, while fine-tuning raised all proposed-model MOS scores above 4 and did not improve WaveGlow.The input mel-spectrograms were generated by Tacotron2; the authors conclude that HiFi-GAN adapts well after fine-tuning.
- 4.4 End-to-End Speech Synthesis: Tacotron2’s predicted mel-spectrograms were already noisy, and fine-tuning increased mel-spectrogram error while improving perceptual quality.The pixel-wise difference between generated waveforms and Tacotron2 mel-spectrograms was insignificant before fine-tuning.
5 Conclusion
The paper presents HiFi-GAN as an efficient, high-quality speech synthesizer whose periodic-pattern modeling supports strong quality, generalization, and deployment efficiency. Its small-footprint version combines comparable quality with substantially faster CPU synthesis, supporting progress toward low-latency, low-memory on-device speech synthesis.
- 5 Conclusion: HiFi-GAN efficiently synthesizes high-quality speech and outperforms publicly available models in quality while substantially improving synthesis speed.The paper reports quality comparable to human level and emphasizes both synthesis quality and speed.
- 5 Conclusion: Modeling speech patterns with various periods and using the proposed discriminator substantially influences speech-synthesis quality.The conclusion links this design choice to the ablation-study findings.
- 5 Conclusion: HiFi-GAN generalizes to unseen speakers and can produce near-human-quality speech from noisy inputs in an end-to-end setting.The conclusion summarizes the paper’s unseen-speaker and end-to-end experiments.
- 5 Conclusion: The small-footprint model offers comparable quality to the best publicly available autoregressive counterpart while synthesizing an order of magnitude faster than real-time on CPU.The authors identify this combination as progress toward on-device natural speech synthesis requiring low latency and memory footprint.
A.1. Details of the Model Architecture
The appendix details the HiFi-GAN generator and multi-period discriminator architectures, alongside three generator variants. ResBlocks differ substantially across variants, with V3 using fewer layers than V1 and V2.
- Figure 4 depicts the generator and an MPD sub-discriminator with period p.
- Table 5 lists hyper-parameters for generator variants V1, V2, and V3.
- V1 and V2 stack two convolution layers and one residual connection three times in each ResBlock.
- V3 stacks one convolution layer and one residual connection two times per ResBlock, yielding substantially fewer layers than V1 and V2.
B.1. Periodic signal discrimination experiments
The appendix evaluates periodic-signal discrimination using a simple dataset and reports classification accuracy comparisons across experimental settings. The supplied passages identify the experiment and table but do not provide the accuracy values.
- The experiment evaluates discriminator classification accuracy for periodic signals.The associated table is titled “Comparison of classification accuracy for periodic signals.”
- The study generates 40,000 sinusoidal signals with random frequency, phase, and energy for the periodic-signal experiment.Frequencies are randomly selected from 1~8,000Hz.
B.2. Frequency response analysis of input signals of discriminators
The frequency-response analysis contrasts MPD and MSD on a sinc-function toy task. MPD preserves periodic structure more effectively, and its trained generator produces outputs closer to the ground truth than MSD.
- The toy experiment trains separate generators with MPD and MSD to mimic a ground-truth sinc function.MPD uses periods [1, 2, 4] in the matched-structure comparison.
- MPD sub-discriminator inputs retain undistorted frequency responses except for aliasing, whereas MSD inputs become progressively smoother after down-sampling.MSD smoothing results from average-pooling low-pass filtering that diminishes high-frequency amplitudes.
- MPD-trained synthesis resembles the ground truth, while MSD-trained synthesis is noisy.
- Average pooling can make MSD inputs resemble the ground truth despite large differences between the ground-truth and generated signals.MPD remains similar to the ground truth across down-sampling factors, supporting stronger periodic-pattern capture.
Appendix C
The appendix compares RWD and MPD at the architectural level, emphasizing differences in weight sharing and input coverage. MPD shares more parameters at each layer because it processes evenly spaced samples rather than the entire sequence.
- With grouped 1d convolution first, RWD resembles MPD’s first 2D convolution with k×1 kernels but does not share weights across groups.
- RWD processes the entire input sequence, whereas MPD operates on down-sampled, evenly spaced samples.
- MPD shares more parameters at each layer than RWD, which uses additional parameters to mix down-sampled representations.