Source-linked AI summary
GANSynth: Adversarial Neural Audio Synthesis
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, Adam Roberts
TL;DR
Efficient audio synthesis must capture both global structure and fine-scale waveform coherence, while existing autoregressive and GAN approaches have complementary limitations. The paper models log magnitudes and instantaneous frequencies with sufficient spectral frequency resolution, showing on NSynth that GANs achieve strong quality and diversity evaluations while generating audio far faster than autoregressive models. The study’s scope remains limited to controlled data and requires validation on broader signal classes.
Problem
Audio synthesis must model temporal scales from approximately 0.1ms to 100s while preserving high fidelity and global structure, but autoregressive and GAN approaches each have important limitations.
Method
The paper trains GANs on log-magnitude spectrograms and instantaneous frequencies represented with sufficient spectral frequency resolution.
Results
On NSynth, GANs outperform a strong WaveNet baseline in automatic and human evaluations and generate examples approximately 54,000 times faster.
Takeaways & Limitations
Carefully controlling the audio representation enables GANs to generate high-quality, locally coherent audio on the NSynth dataset.
Takeaways & Limitations
The study focuses on a specific controlled dataset, and further work is needed to validate and expand it to speech and other natural sounds.
Abstract
from arXiv · showhide
Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative Adversarial Networks (GANs), have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.
1 INTRODUCTION
Neural audio synthesis must capture global structure and fine-scale coherence across vastly different timescales. The paper investigates whether GANs can overcome waveform-coherence challenges through spectral representations and evaluates them on the structured NSynth dataset.
- 1 INTRODUCTION: Neural audio synthesis spans temporal scales from ∼0.1ms to ∼100s, making simultaneous high-fidelity and global-structure modeling difficult.Autoregressive models address fine-scale structure but sample slowly and often rely on external conditioning for global structure.
- 1 INTRODUCTION: NSynth contains aligned, isolated instrumental notes with controlled pitch, timbre, volume, and attribute labels, focusing evaluation on fine-scale audio details.The dataset's constrained structure parallels focused image datasets used to study generative quality.
- 1.2 EFFECTIVE AUDIO REPRESENTATIONS FOR GANS: Periodic audio requires preserving phase coherence because frame-stride alignment precesses when waveform periodicity and output stride differ.Synthesis networks must represent appropriate frequency and phase combinations over short to intermediate timescales.
- 1.2 EFFECTIVE AUDIO REPRESENTATIONS FOR GANS: Instantaneous frequency is obtained by unwrapping STFT phase and differentiating it, producing a measure of the true signal oscillation.For trumpet harmonics, instantaneous-frequency spectra form solid bands where harmonic frequencies occur.
- 1.3 CONTRIBUTIONS: GANs generate more coherent audio from log-magnitude spectrograms and phase than from directly generated waveforms, with instantaneous frequency performing better than phase.The study investigates how architecture and representation interact in coherent GAN audio synthesis.
- 1.3 CONTRIBUTIONS: Separating harmonics improves performance, achieved by increasing STFT frame size or switching to a mel frequency scale.Low pitches have tightly spaced harmonic multiples that can blur together.
- 1.3 CONTRIBUTIONS: ∼54,000 times faster generation and improved automatic and human evaluations show GANs can outperform a strong WaveNet baseline on NSynth.Global latent and pitch conditioning also support smooth timbre interpolation and consistent timbral identity across pitch.
2 EXPERIMENTAL DETAILS
The experiments train progressively growing, pitch-conditioned GANs on selected NSynth notes and compare spectral representations against waveform and WaveNet baselines. Representations vary in phase treatment, frequency resolution, and mel scaling.
- 2.1 DATASET: The study uses 70,379 acoustic-instrument examples with fundamental pitches from MIDI 24–84 for human audio-quality evaluation.Each original NSynth sample is four seconds long at 16kHz.
- 2.2 ARCHITECTURE AND REPRESENTATIONS: The GANs use progressive training to generate audio spectra, with a mirrored downsampling discriminator and gradient penalty.The discriminator estimates divergence between real and generated distributions.
- 2.2 ARCHITECTURE AND REPRESENTATIONS: The generator maps a spherical-Gaussian latent vector and one-hot pitch representation through transposed convolutions to produce output data.An auxiliary discriminator loss predicts pitch to encourage use of the conditioning signal.
- 2.2 ARCHITECTURE AND REPRESENTATIONS: STFT inputs use log magnitudes and phase or instantaneous frequency, with 256 stride, 1024 frame size, 75% overlap, and 513 frequency bins.Instantaneous-frequency variants unwrap phase and take finite differences; high-resolution variants double frame size and stride.
- 2.2 ARCHITECTURE AND REPRESENTATIONS: WaveGAN and pitch-conditioned WaveNet baselines are retrained on the same NSynth subset for comparison.The more stable 8-bit WaveNet model outperforms the 16-bit mixture-of-logistics variant.
3 METRICS
Because perceptual audio quality is difficult to formalize, the study evaluates models with multiple metrics covering human judgments, diversity, and distributional or classifier-based behavior.
- 3 METRICS: Human evaluation is treated as the gold standard because automated audio-quality measurement is notoriously difficult.Participants compare two 4s examples of the same pitch on a five-level Likert scale.
- 3 METRICS: NDB measures generated-example diversity by assigning samples to the nearest of 50 k-means Voronoi cells defined in log-spectrogram space.The cells are fitted from training examples and reused for generated examples.
4 RESULTS
Human and automated evaluations favor high-frequency instantaneous-frequency representations, with IF-Mel judged near real data and waveform or phase variants performing worse. The same results also expose diversity and sample-quality differences across models.
- Human evaluation ranked output quality from IF-Mel, IF, Phase, to Waveform, with IF-Mel slightly below real data.The WaveNet baseline produced high-fidelity sounds but sometimes broke down into feedback and self-oscillation, scoring comparably to IF GANs.
- Pair-wise perceptual wins correlated well with the quantitative rankings in Table 1.Figure 2 compares the highest-performing models for each output representation and baseline; higher scores indicate better perceptual quality.
- NDB followed the human-evaluation trend, while higher frequency resolution improved sample diversity across model types.The WaveNet baseline received the worst NDB score because autoregressive sampling repeatedly produced similar oscillations for a pitch condition.
- FID was significantly lower for high-resolution IF models, whereas phase models retained high FID even at high frequency resolution.Mel scaling had less effect on FID than in the listener study.
- Classifier metrics showed limited discrimination among high-scoring models because explicit pitch conditioning made many outputs accurately classifiable.These metrics mainly identified less reliable pitch classification in some low-frequency models and baselines.
5 QUALITATIVE ANALYSIS
Qualitative analyses show that instantaneous-frequency representations improve waveform phase coherence and preserve meaningful global structure. Their latent space supports smooth interpolation and stable timbral identity across pitch changes.
- 5.1 PHASE COHERENCE: IFGAN waveforms remain coherent from cycle to cycle, unlike the phase irregularities of WaveGAN and PhaseGAN.Rainbowgrams show consistent harmonic colors for real data and IF models, while PhaseGAN has speckles and WaveGAN is irregular.
- 5.2 INTERPOLATION: Global conditioning lets IF-Mel GAN interpolation remain on the prior and produce high-fidelity intermediate sounds.WaveNet autoencoder interpolation leaves its complex latent prior, producing less realistic sounds with feedback harmonics.
- 5.2 INTERPOLATION: Spherical interpolation in the GAN prior produces smooth perceptual changes without additional artifacts.In a Bach prelude example, interpolated timbre morphs across instruments while pitch follows the composed piece.
- 5.3 PITCH CONTROL: Fixing the latent variable while varying pitch over five octaves largely preserves the generated instrument’s timbral identity.A Bach prelude rendered with one latent vector maintains a consistent harmonic structure across pitches.
6 FAST GENERATION
GAN upsampling-convolution models enable parallel audio training and generation, making them amenable to GPU hardware. On a TitanX, the reported four-second sample latency drops from 1077.53 seconds for WaveNet to 20 milliseconds for the IF- model.
- GAN upsampling convolutions process training and generation in parallel for the entire audio sample.
- Parallel processing is well suited to modern GPUs, unlike iterative autoregressive algorithms that can be I/O bound.
- 1077.53 seconds for WaveNet versus 20 milliseconds for the IF- model is the reported TitanX latency for synthesizing one four-second sample.
7 RELATED WORK
Prior audio-generation work largely emphasizes autoregressive or recurrent approaches, while this paper extends adversarial generation to coherent, high-fidelity musical audio. The related literature also motivates NSynth as a focused benchmark for controlled audio variation.
- Speech synthesis dominates much deep generative audio work, often using recurrent or autoregressive models for variable-length inputs and outputs.Adapting GANs to variable-length conditioning or recurrent generators is left as future work because it is non-trivial.
- Earlier musical-instrument models include autoregressive WaveNet systems with slow generation and GAN audio models with coherent but lower-fidelity results.This positions the paper’s approach against both slow high-fidelity autoregressive baselines and earlier lower-fidelity GAN synthesis.
- Recent GAN advances contributing to this work include improved loss functions, gradient penalties, progressive training, and architectural techniques.These developments target training stability, architectural robustness, and generation quality under limited training time.
- NSynth provides aligned, cropped musical notes with pitch, timbre, volume, and attribute labels for controlled generative modeling.Earlier NSynth work used WaveNet autoencoders for timbre interpolation but lacked unconditional sampling from a prior.
8 CONCLUSION
The study demonstrates high-quality GAN audio generation on NSynth, but its evidence is limited to a controlled dataset and broader signals remain for future validation.
- GANs generated high-quality audio on NSynth, exceeding a strong WaveNet baseline in fidelity while producing samples tens of thousands of times faster.The authors attribute this result to carefully controlling the audio representation used for generative modeling.
- The study focused on a specific controlled dataset, so broader signals including speech and other natural sounds require further validation.
- GAN audio generation still faces mode collapse and diversity issues common to GANs.The authors leave combining adversarial losses with encoders or regression losses for future work to better capture the full data distribution.
A MEASURING DIVERSITY ACROSS GENERATED EXAMPLES
The appendix figures visualize NDB bin proportions for GAN models and the WaveNet baseline, using examples conditioned on pitch 60.
- Figure 5 shows NDB bin proportions for the IF-Mel + H model and the WaveGAN baseline.
- Both figures evaluate examples with pitch 60, enabling comparison of the displayed NDB distributions under the same pitch conditioning.
- Figure 6 shows NDB bin proportions for the WaveNet baseline.
B TIMBRAL SIMILARITY ACROSS PITCH
The appendix examines timbral consistency across synthesized pitches and contrasts direct waveform generation with WaveNet alternatives.
- For direct waveform models, the Waveform GAN baseline performs similarly to WaveGAN, while progressive training does not improve performance.
D TRAINING DETAILS
The training setup combines spectral normalization and GAN architectural components with specified optimization, normalization, architecture, and compute choices.
- The GANs use an AC-GAN-style pitch classifier, ADAM optimization, and sweeps over learning rates and auxiliary-classifier-loss weights.The tested learning rates are 2e-4, 4e-4, and 8e-4; classifier-loss weights are 0.1, 1.0, and 10.
- The models use box upscaling or downscaling, while generators apply pixel normalization.
- Pixel normalization divides activations using the total channel count, with n, h, w, and c denoting batch, height, width, and channel dimensions.
- Real log-magnitudes and phases are independently shifted and scaled to [-0.8, 0.8] before discriminator input to accommodate outliers and use more of Tanh’s linear regime.
- Each GAN variant trains for 4.5 days on one V100 GPU with batch size 8, using approximately 5M examples for nonprogressive and 11M for progressive models.
- The WaveNet decoder has 30 dilated-convolution layers with 512 channels, arranged in three stacks of ten layers.
- The high-frequency-resolution architecture uses a 256-dimensional latent vector and 61-dimensional one-hot pitch conditioning.