Source-linked AI summary
Adversarial Audio Synthesis
Chris Donahue, Julian McAuley, Miller Puckette
TL;DR
Audio synthesis must model high-resolution signals across multiple timescales, while GANs had seen limited application to audio and spectrogram-based generation can require lossy inversion. The paper presents WaveGAN for unsupervised raw-waveform audio generation, finding promise in both waveform and spectrogram approaches across speech and other sound domains.
Problem
Audio synthesis requires high-dimensional, high-temporal-resolution representations, while GANs had limited audio application and spectrogram representations can be non-invertible.
Method
WaveGAN applies GANs to unsupervised raw-waveform audio generation and evaluates models using quantitative metrics alongside human judgments.
Results
WaveGAN generated speech-like and domain-specific audio, while SpecGAN achieved a higher inception score (6.0 vs. 4.7) and human labeling accuracy (66% vs. 58%), but WaveGAN received higher subjective ratings for sound quality and speaker diversity.
Takeaways & Limitations
WaveGAN provides a parallelizable template for unsupervised audio generation applicable to creative sound design, with promise shown for waveform and spectrogram methods.
Takeaways & Limitations
Inception score can be high for models that output one example per class or overfit the training data, so it cannot alone establish generative quality.
Abstract
from arXiv · showhide
Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that, without labels, WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.
1 INTRODUCTION
Audio synthesis could support creative sound design, data augmentation, and rapid audio sampling, but high temporal resolution makes learning useful representations difficult. The paper investigates waveform and spectrogram GANs for one-second audio slices, including whether unsupervised models learn intelligible speech structure.
- Motivation: High-resolution audio makes learning representations that capture structure across relevant timescales difficult.Audio signals have high temporal resolution, requiring models to operate effectively in high dimensions.
- Motivation: GANs could support creative sound design, speech-recognition data augmentation, and rapid sampling of large audio collections.The motivation includes exploring compact latent spaces for sound effects and generating audio efficiently.
- Existing approaches: Spectrogram-based GANs are problematic when perceptually informed representations are non-invertible and require lossy or learned inversion.This limits direct listening to generated spectrograms.
- Existing approaches: Autoregressive raw-audio models avoid engineered features but generate slowly because samples must be fed back one at a time.The paper contrasts this sequential generation process with GAN-based synthesis.
- Approach: The paper studies SpecGAN and WaveGAN for one-second audio slices, using approximate spectrogram inversion and a one-dimensional adaptation of DCGAN.WaveGAN is presented as a recipe for modifying image-generation methods to operate on waveforms.
- Evaluation: Without conditioning, both models generate human-intelligible spoken digits, while human judges prefer WaveGAN for sound quality and speaker diversity.The speech benchmark uses the English digits “zero” through “nine.”
2 GAN PRELIMINARIES
The paper reviews GANs as latent-to-data minimax models and introduces Wasserstein-based training as an alternative to the original objective. It also motivates audio-specific architectural changes by contrasting periodic audio structure with image structure.
- GAN formulation: GANs map low-dimensional latent vectors to data by training a generator against a discriminator in a two-player minimax game.The generator minimizes the value function while the discriminator maximizes it.
- GAN formulation: The original GAN trains the discriminator to distinguish real from fake examples while the generator learns to fool it.This formulation corresponds to minimizing Jensen-Shannon divergence between data and generator distributions.
- Wasserstein GAN: WGAN replaces the original divergence objective with a Wasserstein distance computed using 1-Lipschitz functions.The critic-like function is constrained to have Lipschitz constant at most one.
- Wasserstein GAN: In WGAN, D_w assists in computing Wasserstein distance rather than identifying examples as real or fake.Weight clipping and gradient penalty are described as ways to enforce the 1-Lipschitz constraint.
- Audio versus images: Natural-image components generally capture intensity, gradients, and edges, whereas speech components form a periodic basis representing frequency bands.The contrast motivates treating audio differently from image data.
- Audio versus images: WaveGAN uses length-25 one-dimensional filters and larger upsampling than DCGAN while keeping the same parameter and operation counts.DCGAN uses 5x5 two-dimensional filters, whereas WaveGAN uses a larger upsampling factor.
3 WAVEGAN
WaveGAN adapts DCGAN to raw one-dimensional audio by enlarging receptive fields and using audio-specific training refinements. Phase shuffle regularizes the discriminator by making it invariant to waveform phase.
- Architecture: Audio periodicity creates long-range correlations, so raw-waveform models need filters with larger receptive fields.At 16 kHz, a 440 Hz sinusoid takes over 36 samples to complete one cycle.
- Architecture: WaveGAN replaces DCGAN’s two-dimensional filters with length-25 one-dimensional filters and increases each layer’s upsampling factor from 2 to 4.These changes widen the receptive field while adapting the architecture to one-dimensional waveforms.
- Architecture: The architecture adds a layer to produce 16384 samples, slightly more than one second of audio at 16 kHz.This output length is intended for domains such as sound effects and voice commands.
- Architecture: WaveGAN’s recipe flattens convolutions into one dimension, increases stride, removes batch normalization, and uses WGAN-GP training.The paper describes this straightforward modification as already producing reasonable audio.
- Phase shuffle: Transposed-convolution artifacts can become pitched noise in audio, making them harder for the discriminator to reject when artifact frequencies overlap real frequencies.The artifacts occur at a particular phase, creating a discriminator shortcut.
- Phase shuffle: Phase shuffle randomly shifts discriminator feature-map phases by −n to n samples, using reflection to fill missing samples.It requires the discriminator to become invariant to input-waveform phase.
4 SPECGAN: GENERATING SEMI-INVERTIBLE SPECTROGRAMS
SpecGAN adapts image-oriented GANs to spectrograms by designing a representation that remains approximately invertible. Generated spectra are converted back to waveforms with preprocessing inversion and Griffin-Lim phase estimation.
- Representation: Common discriminative audio spectrograms are often unsuitable for generation because they cannot be inverted into audio.This motivates a representation designed specifically for approximate inversion.
- Representation: SpecGAN uses a spectrogram representation designed for image-based GANs that can also be approximately inverted.Its dimensionality is matched to WaveGAN: 16384 samples correspond to a 128x128 spectrogram.
- Preprocessing: SpecGAN preprocessing applies a short-time Fourier transform with 16 ms windows and 8 ms stride, producing 128 linearly spaced frequency bins from 0 to 8 kHz.The FFT’s Nyquist bin is discarded during training and replaced by the dataset mean during resynthesis.
- Evaluation: Figure 4 compares real samples with WaveGAN and SpecGAN outputs across five datasets in the frequency domain.WaveGAN operates in the time domain, while both models’ outputs are displayed spectrally for visual comparison.
- Preprocessing: Magnitude spectra are logarithmically scaled, normalized per frequency bin, clipped to three standard deviations, and rescaled to [−1, 1].Clipping addresses the unbounded values produced by the preprocessing pipeline.
- Resynthesis: Generated spectrograms are transformed back by reversing preprocessing, then applying 16 Griffin-Lim iterations to estimate phase and produce 16384 audio samples.The resulting waveforms have the same target length as WaveGAN outputs.
5 EXPERIMENTAL PROTOCOL
Experiments use SC09 for human evaluation, compare WaveGAN configurations with SpecGAN, and extend training to four additional audio domains.
- Speech Commands Dataset: SC09 contains one-second recordings of the digits zero through nine from many speakers and uncontrolled conditions.The training set has 1,850 utterances per word, totaling 5.3 hours of speech.
- Model configurations: The study compares baseline WaveGAN, phase-shuffle variants, nearest-neighbor upsampling, post-processing filters, and SpecGAN using WGAN-GP.Phase shuffle uses n ∈ {2, 4}; the baseline excludes phase shuffle.
- Additional domains: Additional experiments train WaveGAN and SpecGAN on drums, bird vocalizations, piano, and TIMIT.These datasets span sound effects, wildlife recordings, music, and large-vocabulary speech.
- Training procedure: WaveGAN training uses batches of 64 on one NVIDIA P100 GPU, with SC09 convergence within four days and SpecGAN convergence within two days.WaveGAN produces speech-like audio within the first hour of training.
6 EVALUATION METHODOLOGY
The evaluation combines inception score and nearest-neighbor indicators with human judgments to assess semantic capture, diversity, memorization, and perceptual quality.
- Evaluation strategy: The study uses multiple quantitative metrics and human judges because generative-model metrics can disagree with one another and with human judgment.Human evaluation is applied to the most promising models after quantitative validation.
- Inception score: Inception score is exp(E_x D_KL(P(y | x)||P(y))) and ranges from 1 to n for n classes.It is estimated over many generated samples and used for quantitative evaluation and early stopping.
- Inception score: The audio classifier for inception score uses a log-normalized Mel-scale spectrogram and a four-layer convolutional network with 10 output classes.The input representation uses 64 ms windows, 8 ms stride, and 128 frequency bins spanning 40–7800 Hz.
- Metric caveats: Inception score can be inflated by producing one example per class or memorizing training examples.These are identified as trivial failure cases of the metric.
- Nearest-neighbor indicators: |D|self measures within-generated-set nearest-neighbor distance, while |D|train measures nearest-neighbor distance to the training data.The indicators are compared with test-set values to assess diversity and overfitting.
- Human judgments: Human evaluation labels 3,000 generated digit examples and rates sound quality, intelligibility, and speaker diversity on 1–5 scales.Random labeling accuracy is 10%, and mean opinion scores are computed from 300 examples.
7 RESULTS AND DISCUSSION
WaveGAN and SpecGAN both generate speech-like audio and transfer across domains, but their quantitative and perceptual strengths differ, leaving no decisive overall winner.
- Speech results: 4.7 is the inception score of the best WaveGAN model, below the SC09 test-set score of 8 used as an overfitting reference.The model uses phase shuffle with n = 2.
- Ablation results: Phase shuffle lowers SpecGAN’s inception score, possibly because it has an exaggerated effect on spectrograms’ compact temporal axis.Using 50% discriminator dropout also results in a lower score than phase shuffle in the reported comparison.
- Nearest-neighbor analysis: Most experiments have |D|self and all experiments have |D|train above the test-data values, without indicating the metric’s identified trivial solutions.The generated examples therefore differ statistically from real data while avoiding those specific failure patterns.
- Waveform versus spectrogram: SpecGAN achieves a higher inception score than WaveGAN, 6.0 vs. 4.7, and higher human labeling accuracy, 66% vs. 58%.Human judges nevertheless prefer WaveGAN on sound quality and speaker diversity.
- Cross-domain results: Across other domains, WaveGAN captures drum modes, varied bird sounds, consonant piano motifs, and speech-like TIMIT babbling.Its generated spectra are visually more consistent with training data than SpecGAN’s in examples such as sharpness.
8 RELATED WORK
Prior audio-generation work includes text-to-speech systems, autoregressive raw-audio models, and supervised GAN applications, whereas this paper studies unsupervised GAN synthesis.
- Text-to-speech: Traditional text-to-speech systems are concatenative or parametric, using prerecorded speech fragments or vocoder parameters.Concatenative systems sequence phonetically indexed units; parametric systems synthesize salient speech parameters.
- Text-to-speech: End-to-end neural text-to-speech methods learn vocoder features from text or phonetic embeddings before converting them to raw audio.Conversion uses methods such as WORLD, Griffin-Lim, or neural vocoders.
- Unsupervised raw audio: WaveNet and related work explore unsupervised raw-audio generation through autoregressive modeling or autoencoding.Unconditional autoregressive models had not demonstrated cohesive-word generation in the cited setting.
- GANs for audio: Before this work, audio GAN applications were primarily supervised and combined GAN objectives with traditional or unstructured losses.Applications included speech enhancement and mappings between spectrogram domains.
9 CONCLUSION
WaveGAN targets short sound-effect generation with parallelizable audio synthesis, while addressing artifacts introduced by upsampling. The paper explores learned filtering and alternative interpolation strategies to mitigate these artifacts.
- WaveGAN is presented as a fully parallelizable approach for generating hours of audio in only a few seconds.
- Transposed-convolution artifacts create pitched noise in audio, with measured response peaks at 250 Hz, 1 kHz, 4 kHz, and higher multiples.
- A learned 512-sample post-processing filter rejects artifact bands while boosting frequency regions prominent in the target domain.
- Speech filters boost prominent speech bands, whereas bird-vocalization filters mainly reduce noise because bird sounds are more uniformly distributed in frequency.
- Nearest-neighbor, linear, and cubic interpolation attenuate aliasing, but linear and cubic interpolation produce qualitatively poor audio compared with transposed convolution.
B EXPERIMENTS WITH AUTOREGRESSIVE WAVEFORM MODELS
The paper compares WaveGAN with autoregressive waveform models for unconditional one-second digit generation. The tested autoregressive implementations were slow and did not produce cohesive words with default settings, limiting their competitiveness for creative sound-effect generation.
- Autoregressive models can potentially produce high-quality audio but generate waveforms several orders of magnitude more slowly and lack a compact latent space.
- Three public WaveNet and SampleRNN implementations were evaluated on the SC09 digit-generation task using default parameters and one-second examples.
- The tested implementations failed to produce cohesive words and obtained inception scores of 1.07 ± 0.05, 1.29 ± 0.03, and 2.28 ± 0.19.
- The authors qualify the comparison because these implementations used default parameters and were developed for a different task.
E TRAINING HYPERPARAMETERS
This section points to the paper’s hyperparameter documentation and the architecture tables for WaveGAN and SpecGAN.
- Table 6 lists the WaveGAN and SpecGAN hyperparameter values used in the experiments and presented as out-of-the-box recommendations.
- Tables 4 and 5 list the SpecGAN generator and discriminator architectures.
- Table 6 is specifically identified as the WaveGAN hyperparameters table.