Source-linked AI summary
UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, Juntae Kim
TL;DR
Band-limited conditioning omits high-frequency information, while full-band vocoders can suffer from over-smoothed spectrograms. UnivNet addresses this with multi-resolution spectral discrimination and full-band mel-spectrogram conditioning. It reports the best objective and subjective results for seen and unseen speakers, including text-to-speech subjective evaluation.
Problem
Band-limited mel-spectrograms omit high-frequency information, while some full-band vocoder models generate over-smoothed spectrograms.
Method
UnivNet uses full-band mel-spectrograms with a multi-resolution spectrogram discriminator and combines spectral and temporal discriminator components during training.
Results
UnivNet achieved the best objective and subjective results among competing models for seen and unseen speakers, including the best text-to-speech subjective score.
Takeaways & Limitations
The results demonstrate potential for high-fidelity real-time waveform generation and rapid adaptation to new speakers without training from scratch.
Abstract
from arXiv · showhide
Most neural vocoders employ band-limited mel-spectrograms to generate waveforms. If full-band spectral features are used as the input, the vocoder can be provided with as much acoustic information as possible. However, in some models employing full-band mel-spectrograms, an over-smoothing problem occurs as part of which non-sharp spectrograms are generated. To address this problem, we propose UnivNet, a neural vocoder that synthesizes high-fidelity waveforms in real time. Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.
1. Introduction
UnivNet addresses over-smoothing in full-band neural vocoders by combining full-band mel-spectrogram conditioning with multi-resolution spectral discrimination. It achieves strong results across seen and unseen speakers, including text-to-speech adaptation.
- Motivation: Band-limited mel-spectrograms omit high-frequency acoustic information, while some full-band models generate over-smoothed spectrograms.Full-band features can provide information up to half the sampling rate, but prior full-band models exhibited non-sharp outputs.
- Proposed approach: Multiple spectral and temporal resolutions are expected to improve generated spectrogram resolution, based on their usefulness in voice activity detection.The discriminator treats multiple-resolution spectral and temporal features as inputs to a binary classifier.
- Proposed approach: UnivNet adds a multi-resolution spectrogram discriminator using linear spectrogram magnitudes computed with varied parameter sets.Its discriminators combine spectral and temporal modeling through MRSD and a multi-period waveform discriminator.
- Results: UnivNet achieved the best objective and subjective scores among GAN-based vocoders for both seen and unseen speakers.The model was trained from scratch on a large multi-speaker English dataset.
- Results: In text-to-speech evaluation, UnivNet achieved the best subjective score and demonstrated potential for rapid adaptation without training from scratch.The model was fine-tuned using predicted features from an acoustic model.
2. Related work
The related work positions GAN-based neural vocoders as fast alternatives to traditional waveform synthesis, while describing representative discriminator and loss designs. UnivNet’s architecture is presented against this progression.
- Representative GAN vocoders: MelGAN combines a non-autoregressive generator with a multi-scale waveform discriminator for GAN-based mel-spectrogram inversion.It is described as one of the first vocoders to successfully achieve this setup.
- Representative GAN vocoders: Parallel WaveGAN combines a non-autoregressive WaveNet with adversarial and multi-resolution STFT losses, achieving fast convergence and real-time inference without distillation.Its training objective uses a weighted sum of the adversarial and multi-resolution STFT loss terms.
- UnivNet architecture: Figure 1 depicts UnivNet’s architecture, including STFT-based linear spectrogram computation and reshaping of one-dimensional signals into two-dimensional signals.The caption defines STFT #m as the m-th parameterized spectrogram computation and reshape2d(p) as producing height T/p and width p.
3. Description of the proposed model
UnivNet uses a MelGAN-inspired generator conditioned on log-mel-spectrograms and discriminator components that model multiple spectral, temporal, and periodic views. Training combines adversarial objectives with multi-resolution STFT losses.
- 3.1. Generator: The generator receives a noise sequence and log-mel-spectrogram condition, producing an output waveform with the target waveform’s length.Transposed convolutions preserve matching output and target lengths.
- 3.1. Generator: Location-variable convolution captures local conditioning information, with kernels predicted from the log-mel-spectrogram by kernel predictors.LVC improves sound quality and speed while maintaining model size, according to the passage.
- 3.2. Discriminators: The multi-resolution spectrogram discriminator compares real and generated linear spectrogram magnitudes computed from the same waveform using multiple STFT parameter sets.Each parameter set specifies Fourier-transform points, frame-shift interval, and window length.
- 3.2. Discriminators: MRSD uses spectrograms with varied temporal and spectral resolutions, whereas filter-bank approaches divide one spectral representation into bands.Its architecture uses strided two-dimensional convolutions and leaky rectified linear units.
- 3.2. Discriminators: The multi-period waveform discriminator extracts periodic waveform components at intervals defined by a set of prime numbers.It was added to improve detailed adversarial modeling in the temporal domain.
- 3.3. Training losses: Multi-resolution STFT loss sums spectrogram losses computed with various STFT parameter sets as an auxiliary training loss.The auxiliary loss comprises spectral convergence and log STFT magnitude losses.
- 3.3. Training losses: UnivNet uses least-squares GAN objectives combining adversarial losses from all sub-discriminators with the auxiliary loss.The discriminator-output sum is divided by the number of sub-discriminators to keep its balance with the auxiliary loss unchanged.
4. Experiments
The experiments train UnivNet and comparison vocoders on full-band, multi-speaker speech, then evaluate objective quality, spectrogram reconstruction, and subjective audio quality.
- 4.1. Data configurations: The LibriTTS train-clean-360 subset provided 192 hours, 116k utterances, and 904 speakers for training and seen-speaker evaluation.
- 4.1. Data configurations: The testclean subset evaluated unseen speakers, while LJSpeech supported text-to-speech evaluation after vocoder fine-tuning with predicted log-mel-spectrograms.
- 4.2. Evaluation metrics: Quality was measured with wideband PESQ, spectrogram-magnitude RMSE, and MOS ratings from 1 to 5.
- 4.3. Model details: Experiments used full-band 0–12 kHz log-mel-spectrograms at a 24 kHz sampling rate and compared MelGAN, Parallel WaveGAN, and HiFi-GAN.
5. Results
Ablations show that UnivNet’s discriminator configuration is important, while comparisons indicate robust quality across seen and unseen speakers with real-time generation.
- 5.1. Ablation study: Removing MRSD produced over-smoothing, especially in the high-frequency band, and significantly reduced MOS because of an audible metallic artifact.
- 5.1. Ablation study: Removing MPWD reduced the score, indicating support for detailed temporal prediction by the generator.
- 5.1. Ablation study: Replacing MPWD with MSWD significantly reduced MOS, whereas adding MSWD showed little difference from UnivNet-c16 despite lower training speed.
- 5.2. Comparison with existing models: HiFi-GAN and UnivNet maintained scores for seen and unseen speakers, while MelGAN and Parallel WaveGAN declined significantly for unseen speakers.
- 5.2. Comparison with existing models: UnivNet-c16 generated at 200 times real time with consistently high objective and subjective scores; UnivNet-c32 achieved the highest objective scores in all scenarios.
6. Conclusion
The paper concludes that UnivNet combines multi-resolution spectrogram discrimination with full-band conditioning for high-fidelity, real-time waveform generation.
- 6. Conclusion: UnivNet uses a multi-resolution spectrogram discriminator that accepts multiple spectrograms and alleviates over-smoothing to improve waveform spectral resolution.
- 6. Conclusion: Table 2 reports comparisons with existing models, with Speed defined as generation speed relative to real time.