PaperlayerView exact revision on arXiv

Source-linked AI summary

BigVGAN: A Universal Neural Vocoder with Large-Scale Training

Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, Sungroh Yoon

arXiv:2206.04658v2cs.SDcs.CLcs.LGeess.AS

TL;DR

Existing GAN vocoders struggle to synthesize high-fidelity audio across many speakers and recording environments. BigVGAN combines periodic, anti-aliased generator design with large-scale GAN training, reaching 112M parameters and strong zero-shot performance across diverse out-of-distribution conditions. Its reported quality advantages include improved subjective scores over HiFi-GAN, while some stabilization and regularization alternatives reduced perceptual quality.

  • Problem

    Existing GAN vocoders degrade for unseen speakers and recording environments because of limited model capacity.

  • Method

    BigVGAN combines periodic activations, anti-aliased representation, and large-scale GAN training up to 112M parameters.

  • Results

    BigVGAN achieves strong zero-shot performance across unseen speakers, languages, environments, singing voice, music, and instrumental audio, with 112M BigVGAN exceeding HiFi-GAN by +0.11 SMOS.

  • Takeaways & Limitations

    A large GAN vocoder trained on clean LibriTTS can generalize to diverse out-of-distribution audio without fine-tuning.

  • Takeaways & Limitations

    Alternative architecture changes, spectral normalization, and data augmentation resulted in worse perceptual quality in the authors’ study.

Abstract

from arXiv · show

Despite recent progress in generative adversarial network (GAN)-based vocoders, where the model generates raw waveform conditioned on acoustic features, it is challenging to synthesize high-fidelity audio for numerous speakers across various recording environments. In this work, we present BigVGAN, a universal vocoder that generalizes well for various out-of-distribution scenarios without fine-tuning. We introduce periodic activation function and anti-aliased representation into the GAN generator, which brings the desired inductive bias for audio synthesis and significantly improves audio quality. In addition, we train our GAN vocoder at the largest scale up to 112M parameters, which is unprecedented in the literature. We identify and address the failure modes in large-scale GAN training for audio, while maintaining high-fidelity output without over-regularization. Our BigVGAN, trained only on clean speech (LibriTTS), achieves the state-of-the-art performance for various zero-shot (out-of-distribution) conditions, including unseen speakers, languages, recording environments, singing voices, music, and instrumental audio. We release our code and model at: https://github.com/NVIDIA/BigVGAN

1 INTRODUCTION

BigVGAN targets the limited out-of-distribution robustness of existing GAN vocoders by scaling GAN training and introducing architectural changes for universal, high-fidelity generation without fine-tuning.

  • Existing GAN vocoders generate high-fidelity audio quickly but degrade for unseen speakers and recording environments because of limited model capacity.
  • Scaling model size for zero-shot performance had been explored in text and image synthesis but not audio synthesis.
  • BigVGAN introduces periodic activations to provide an audio-specific inductive bias for waveform synthesis.
  • The anti-aliased multi-periodicity composition module combines learnable periodic components and low-pass filtering to reduce high-frequency artifacts.
  • BigVGAN scales to 112M parameters while addressing large-scale GAN training failures without regularizing both generator and discriminator.
  • BigVGAN-base with 14M parameters outperforms comparable state-of-the-art vocoders in-distribution and BigVGAN with 112M parameters achieves a large zero-shot margin.

2 RELATED WORK

Prior audio GAN work largely emphasized discriminator and loss design, while BigVGAN focuses on generator architecture and scaling to improve universal neural vocoding under challenging conditions.

  • Image GAN research has achieved strong results through optimized architectures and large-scale training.
  • Audio GAN vocoders have primarily improved discriminator architectures and added auxiliary losses.
  • BigVGAN introduces periodic inductive bias and addresses feature aliasing within a non-autoregressive generator.
  • Improving generator architecture is crucial for universal neural vocoding in challenging conditions.
  • Universal neural vocoding has had limited success with WaveRNN, flow-based, and GAN-based approaches.

3 METHOD

BigVGAN modifies a HiFi-GAN-style generator with periodic, anti-aliased components and scales it to 112M parameters while addressing training instabilities and artifact formation.

  • The generator transforms mel spectrograms or other features into raw waveforms, using HiFi-GAN as the baseline architecture.
  • 3.2 PERIODIC INDUCTIVE BIAS: Snake activations add channel-wise trainable periodic frequencies, enabling the generator to model multi-periodic waveform components.
  • 3.3 ANTI-ALIASED REPRESENTATION: Snake activations improve the generator’s periodic bias but can create high-frequency details that cause aliasing artifacts in discrete-time outputs.
  • 3.3 ANTI-ALIASED REPRESENTATION: The anti-aliased nonlinearity upsamples by 2×, applies Snake and low-pass filtering, then downsamples by 2×.
  • 3.3 ANTI-ALIASED REPRESENTATION: The AMP module applies filtered Snake nonlinearities within residual dilated-convolution layers to reduce high-frequency artifacts.
  • 3.4 BIGVGAN WITH LARGE SCALE TRAINING: BigVGAN scales from a 14M-parameter base using six upsampling blocks and 1536 AMP channels to reach 112M parameters.
  • 3.4 BIGVGAN WITH LARGE SCALE TRAINING: Large-scale training required lowering the learning rate, increasing batch size, and addressing gradient amplification that could cause early collapse.
  • 3.4 BIGVGAN WITH LARGE SCALE TRAINING: Alternative architecture changes, spectral normalization, and data augmentation produced worse perceptual quality in the authors’ study.

4 RESULTS

BigVGAN is evaluated on LibriTTS and diverse out-of-distribution speech, music, and instrumental audio, with experiments covering quality, speed, robustness, and ablations. It consistently improves objective or subjective quality over baselines, especially in challenging recording environments and non-speech audio.

  • Experimental setup: BigVGAN and baselines are trained on the full LibriTTS dataset, including diverse recording environments, using 24 kHz audio and 100-band log-mel spectrograms.The evaluation includes in-distribution and out-of-distribution scenarios, with training on LibriTTS train-full rather than only clean subsets.
  • LibriTTS results: BigVGAN significantly improves all objective LibriTTS metrics, and BigVGAN-base consistently outperforms equal-sized HiFi-GAN (V1).The authors attribute the equal-size advantage to a better periodic inductive bias for waveform data.
  • LibriTTS results: +0.05 SMOS for BigVGAN-base and +0.11 SMOS for 112M BigVGAN over HiFi-GAN are observed when samples are compared side-by-side with ground truth.Without side-by-side ground-truth comparison, the models perform comparably in MOS.
  • Out-of-distribution robustness: BigVGAN outperforms baselines by a large margin on unseen recording environments and improves zero-shot quality for singing voice, instrumental audio, and full-song mixtures.For clean under-resourced languages, the performance gap is not substantially large; improvements on drums and bass are less significant.
  • Ablation and scaling: 58% of pairwise ratings favor 112M BigVGAN over a 112M HiFi-GAN on MUSDB18-HQ mixtures, with statistical significance at p-value < 0.01.The comparison uses models with matched 112M parameter counts and supports an architectural advantage for BigVGAN in the large-scale setting.

5 CONCLUSIONS

BigVGAN scales universal neural vocoding through an anti-aliased periodic generator and large-scale GAN training, achieving strong zero-shot performance across diverse out-of-distribution conditions.

  • 5 CONCLUSIONS: BigVGAN combines learned-frequency periodic activation with anti-aliased representation in an improved generator architecture.The design injects an inductive bias for waveform generation and addresses feature aliasing in the non-autoregressive generator.
  • 5 CONCLUSIONS: BigVGAN is the largest GAN vocoder described, scaling to 112M parameters while maintaining high-fidelity output without over-regularization.
  • 5 CONCLUSIONS: Training on clean LibriTTS speech supports strong zero-shot performance for unseen speakers, languages, environments, singing voice, and instrumental audio.
  • 5 CONCLUSIONS: The study evaluates universal neural vocoding at unprecedented scale across data, model, evaluations, and diverse scenarios.

A ARCHITECTURAL DETAILS

BigVGAN uses a HiFi-GAN-like generator that progressively upsamples mel spectrograms through AMP residual blocks, while MRD and MPD provide complementary waveform representations for discrimination.

  • A ARCHITECTURAL DETAILS: The generator converts a mel spectrogram into a waveform through an initial convolution followed by N hierarchical upsampling blocks.Each block halves the preceding channel count through transposed convolution and uses upsampling rate u_i.
  • A ARCHITECTURAL DETAILS: Each upsampling block contains M AMP residual blocks with dilated convolutions using varied kernel sizes and dilation rates.AMP blocks contain anti-aliased periodic activations within the residual dilated-convolution stack.
  • A ARCHITECTURAL DETAILS: The low-pass filter uses a cutoff frequency tied to the sampling rate and up/down-sampling ratio, then serves as the convolution kernel for resampling.The described design uses m = 2, window length n = 6 · m, and transition-band half-width f_h = 0.6/m.
  • A ARCHITECTURAL DETAILS: MRD converts waveforms into 2D linear spectrograms with varied STFT parameters, whereas MPD reshapes and pads waveforms into 2D representations with varied widths and heights.Both discriminator submodules use multiple 2D-convolution subdiscriminators.

B TRAINING OBJECTIVE DETAILS

BigVGAN retains the HiFi-GAN training objective while replacing MSD with MRD, combining adversarial, feature-matching, and mel-spectrogram regression losses.

  • C TRAINING OBJECTIVE DETAILS: BigVGAN uses generator and discriminator objectives based on the HiFi-GAN formulation, with MRD replacing MSD as a discriminator submodule.
  • C TRAINING OBJECTIVE DETAILS: The adversarial loss uses least-square GAN objectives for the generator and discriminator across MPD and MRD submodules.D_k denotes the k-th MPD or MRD discriminator submodule.
  • C TRAINING OBJECTIVE DETAILS: Feature matching minimizes the ℓ1 distance between intermediate discriminator features for real and generated audio.The loss sums feature distances across the T layers of each sub-discriminator.
  • C TRAINING OBJECTIVE DETAILS: The generator also minimizes spectral ℓ1 regression between synthesized and ground-truth mel spectrograms, using λ_fm = 2 and λ_mel = 45.The mel spectrogram is obtained through the STFT function φ.

C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO

Large-scale audio GAN training required selective stabilization: several seemingly reasonable changes caused collapse, artifacts, over-regularization, or degraded perceptual quality.

  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: Without clipping, BigVGAN’s gradient norm explodes during training, while BigVGAN-base shows a sharp early increase.
  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: Anti-aliased or nearest-neighbor upsampling alternatives introduced training instability and early collapse.
  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: Periodic discriminator activations degraded quality because the feature-matching loss diverged.
  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: Spectral normalization stabilized training but caused phase-mismatch artifacts by over-regularizing MPD gradients.The generator consequently relied primarily on mel regression loss, worsening audio quality.
  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: Larger discriminators partially alleviated early collapse but usually degraded audio quality without clear improvement.
  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: Deeper or wider generators produced high-frequency rattling artifacts or early collapse rather than better quality.
  • C PRACTICAL LESSONS FOR LARGE-SCALE GAN TRAINING ON AUDIO: SpecAugment caused over-smoothing, while waveform mixup sometimes mixed two speaker identities during single-speaker inference.

D VISUALIZATION

Visual examples show how BigVGAN’s periodic activation and anti-aliased representation improve harmonic accuracy over HiFi-GAN and an ablated model.

  • HiFi-GAN’s feature aliasing introduces blurry harmonics in singing-voice synthesis by aggregating incorrect frequency components.The resulting error is amplified during generation.
  • Periodic activation improves harmonic components in BigVGAN-base even without the filtered nonlinearity.The comparison also uses an advanced discriminator.
  • Continuous feature representation further improves harmonic accuracy in BigVGAN models by reducing feature aliasing.

E ADDITIONAL RESULTS

Additional evaluations compare BigVGAN with prior vocoders and ablations across clean, unseen, and varied recording environments. BigVGAN-base consistently improves over its ablations, while the final 112M model further improves universal vocoding accuracy.

  • Flow-based vocoder quality degrades heavily in multi-speaker setups, making these models unsuitable as universal neural vocoders in this evaluation.
  • UnivNet’s larger LibriTTS training set did not improve quality, and subjective quality was indistinguishable between its train-full and train-clean-360 checkpoints.The authors conjecture that the architecture is harder to generalize to unseen recording environments.
  • BigVGAN-base consistently improves all metrics over its ablation models across clean and other recording environments.
  • MRD sharpens spectral structure, Snake activation improves accuracy and periodicity error, and filtering suppresses aliasing and high-frequency artifacts.
  • The final 112M BigVGAN substantially improves accuracy for state-of-the-art universal neural vocoding.
  • All GAN-based models performed comparatively well on unseen VCTK and LJSpeech speech in subjective listening tests.The passage attributes this result to sufficiently diverse training data.

F LINGUISTIC ACCURACY EVALUATION

The linguistic evaluation uses multilingual ASR-based character error rates to assess synthesized speech. BigVGAN achieves consistently low CERs, close to ground truth, across the evaluated languages.

  • Character error rates from a multilingual Conformer-based ASR evaluation are used to measure linguistic accuracy.
  • BigVGAN has the consistently lowest CERs, which are also close to ground truth.
  • BigVGAN-base outperforms the largest HiFi-GAN on four of six languages, ties on one, and underperforms on one.The languages are English, German, Catalan, French, Chinese, and Spanish.

G ADDITIONAL DETAILS OF MECHANICAL TURK EVALUATION

The Mechanical Turk MOS and SMOS evaluations collect 450 unique ratings per model with worker-selection and sample-assignment procedures intended to improve evaluation reliability.

  • Each model receives 450 unique ratings in the MOS and SMOS tests.
  • Workers may evaluate up to three random samples to encourage participant diversity.
  • The study restricts listeners to native English speakers in the United States and requires an evaluation acceptance rate above 98%.
Loading 2206.04658v2…