GANSynth: Adversarial Neural Audio Synthesis

1 INTRODUCTION

• Autoregressive models use external conditioning for global structure but sample slowly, whereas GANs offer efficient parallel sampling and global latent control yet straightforward waveform generation lacks perceptual fidelity.Source: §1 · p. 1

1.1 GENERATING INSTRUMENT TIMBRES

• NSynth provides aligned, cropped instrument notes across pitches, timbres, and volumes with attribute labels; this work adds adversarial training and explores representations for noncausal convolutional generation.The original NSynth autoencoders did not support unconditional sampling from a prior.Source: §1.1 · pp. 2–3

1.2 EFFECTIVE AUDIO REPRESENTATIONS FOR GANS

• Audio periodicity makes phase alignment drift across frames, challenging waveform coherence; unwrapped-phase derivatives provide instantaneous frequency, a measure of true signal oscillation.Figure 1 illustrates this phase precession and the instantaneous-frequency representation.Source: Figure 1 · p. 2, §1.2 · p. 3

1.3 CONTRIBUTIONS

• Generating log-magnitude spectrograms and phases improves coherence over waveform generation; estimating instantaneous frequency helps further, while larger STFT frames and mel scaling separate harmonics.Low pitches have tightly spaced harmonics, which can blur or overlap.Source: §1.3 · p. 3
• On NSynth, GANs outperform a strong WaveNet baseline in automatic and human evaluations and generate examples ∼54,000 times faster.Source: §1.3 · p. 3
• Global conditioning on latent and pitch vectors enables perceptually smooth timbre interpolation and consistent timbral identity across pitch.Source: §1.3 · p. 3

2.1 DATASET

• The study uses 70,379 acoustic-instrument notes at MIDI pitches 24–84, with a shuffled 80/20 train/test split.NSynth contains 300,000 isolated notes from 1,000 instruments; samples are four seconds at 16 kHz. The original split was divided by instrument type, which the authors considered undesirable for this task.Source: §2.1 · pp. 3–4

2.2 ARCHITECTURE AND REPRESENTATIONS

• The GAN maps spherical-Gaussian latent vectors through transposed convolutions and uses a mirrored downsampling discriminator, gradient penalty, and pixel normalization.Progressive and nonprogressive variants had comparable quality; progressive training showed slightly better convergence time and sample diversity, so subsequent comparisons use progressive training.Source: §2.2 · p. 4
• Pitch conditioning appends a one-hot pitch vector to the latent input, while an auxiliary discriminator classification loss encourages pitch use and aims to separate pitch control from timbre.The auxiliary classification loss follows Odena et al. (2017).Source: §2.2 · p. 4
• Spectral inputs use 1024-frame, 256-stride STFTs with 513 frequency bins, log magnitudes, and either scaled phase or instantaneous frequency.The two channels are magnitude and phase; instantaneous frequency is computed by unwrapping phase and taking finite differences. Higher-resolution “+ H” variants double frame size and stride, while “IF-Mel” variants use 1024 mel bins without dimensional compression; approximate inverse linear transformation restores STFTs without significantly harming audio quality.Source: §2.2 · p. 4
• The waveform GAN baseline adapts pitch-conditioned WaveGAN (Donahue et al., 2019); the authors primarily report its results because their own waveform GANs performed similarly.The adapted WaveGAN is retrained on the NSynth subset; the paper directs readers to appendix Table 2 for additional details.Source: §2.2 · p. 4
• The WaveNet baselines accept the same one-hot pitch signal, and the 8-bit mu-law variant is more stable and outperforms the 16-bit mixture-of-logistics variant.The authors distinguish these generative baselines from prior NSynth WaveNet autoencoder work, which required conditioning on the original audio (Engel et al., 2017).Source: §2.2 · pp. 4–5

Human Evaluation

• Human evaluators provide the gold-standard audio-quality assessment through pairwise comparisons of generated samples.Amazon Mechanical Turk participants compared two 4-second examples at the same pitch on a five-level Likert scale; the study collected 3,600 ratings, with each model involved in 800 comparisons, including hold-out examples.Source: §3 · p. 5

Number of Statistically-Different Bins (NDB)

• NDB measures diversity by counting log-spectrogram clusters whose training and generated sample counts differ significantly.Training examples are grouped into 50 k-means Voronoi cells, generated examples are assigned to the nearest cell, and a two-sample Binomial test identifies statistically different counts (Richardson & Weiss, 2018).Source: §3 · p. 5

Inception Score (IS)

• IS uses pitch-classifier features to assess whether generated samples are individually classifiable and collectively cover classes (Salimans et al., 2016).The metric replaces Inception features with features from a pitch classifier trained on spectrograms of the acoustic NSynth dataset; it is used in consistency with GAN literature (Gulrajani et al., 2017; Miyato et al., 2018; Karras et al., 2018a).Source: §3 · p. 5

Pitch Accuracy (PA) and Pitch Entropy (PE)

• PA and PE separately measure pitch-classifier accuracy and output-distribution entropy to distinguish pitch accuracy from pitch diversity.These measures address the possibility that IS conflates models that fail to produce distinct pitches with models that produce only a few pitches.Source: §3 · p. 5

Fréchet Inception Distance (FID)

• FID measures the 2-Wasserstein distance between Gaussian distributions fitted to pitch-classifier features (Heusel et al., 2017).The original metric uses pretrained Inception-classifier features and is reported to correlate with perceptual quality and diversity on synthetic distributions; this evaluation substitutes pitch-classifier features.Source: §3 · p. 5

4 RESULTS

• Human evaluation ranks quality from IF-Mel through IF and Phase to Waveform; IF-Mel is slightly below real data, while WaveNet’s feedback failures yield scores comparable to IF GANs.Figure 2 summarizes pairwise wins among the highest-performing models of each type.Source: §4 · p. 5, Figure 2 · p. 6
• NDB follows human-evaluation rankings, improves with high-frequency resolution across model types, and is worst for WaveNet.Despite high likelihood on training data, WaveNet sampling tends to repeat the same oscillation for a given pitch, causing an extreme lack of diversity.Source: §4 · p. 6
• High-resolution IF models achieve significantly lower FID; Mel scaling matters less than in listener judgments, while Phase models retain high FID even at high resolution.The high FID of Phase models reflects their poor sample quality.Source: §4 · p. 6
• High-resolution models match real-data classification accuracy, but conditioning and distribution shifts make high IS, pitch accuracy, and pitch entropy weak sample-quality discriminators; low-resolution models and baselines are less reliable.These classifier metrics offer only a rough indication of which models generate classifiable pitches less reliably.Source: §4 · p. 6
• WaveGAN and PhaseGAN samples show phase irregularities, whereas IFGAN is more cycle-coherent; rainbowgrams show harmonic consistency for real and IF models but speckles for PhaseGAN.The illustrated examples use waveforms modulo the fundamental periodicity of MIDI C60.Source: Figure 3 · p. 7

5.1 PHASE COHERENCE

• Real data and IFGAN show cycle-to-cycle phase coherence, while PhaseGAN has discontinuities and WaveGAN appears irregular and incoherent.Rainbowgrams depict log magnitude as brightness and instantaneous frequency as color; consistent harmonic colors are visible for real data and IFGAN.Source: §5.1 · p. 7

5.2 INTERPOLATION

• IF-Mel GAN interpolation stays aligned with its global prior and produces smooth, high-fidelity intermediate sounds, unlike WaveNet autoencoder interpolation that can fall apart, oscillate, and whistle.The accompanying Bach prelude example morphs timbre across 15 random latent points while pitches follow the composed piece.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3 CONSISTENT TIMBRE ACROSS PITCH

• Across five octaves, varying pitch while fixing the latent condition largely preserves the GAN’s timbral identity and harmonic structure.A Bach prelude rendered with one latent vector retains a consistent harmonic structure across pitches.Source: §5.3 · pp. 8–9

• On a TitanX GPU, synthesizing one four-second sample took 20 milliseconds with IF-Mel GAN versus 1077.53 seconds with WaveNet, about 53,880 times faster.Source: §6 · p. 9
• Upsampling convolutions allow training and generation to run in parallel across the audio sample, opening the possibility of real-time neural audio synthesis on device.The passage notes that iterative autoregressive algorithms are often I/O-bound on modern GPU hardware.Source: §6 · p. 9

7 RELATED WORK

• Speech synthesis dominates deep generative audio research, but adapting GANs to variable-length conditioning or recurrent generators remains non-trivial and is left for future work (van den Oord et al., 2018; Wang et al., 2017).Source: §7 · p. 9
• Music instrument synthesis has used autoregressive models with slow generation, while early GAN audio results were coherent but below autoregressive likelihood models in fidelity (van den Oord et al., 2016; Mehri et al., 2017; Donahue et al., 2019).Source: §7 · p. 9
• The work adopts architectural techniques from Karras et al. (2018a) and builds on GAN loss modifications proposed by Gulrajani et al. (2017).Gulrajani et al. report improved training stability and architectural robustness; Karras et al. introduce progressive training and architectural tricks to improve generation quality under limited training time.Source: §7 · p. 9
• NSynth enabled timbre interpolation with WaveNet autoencoders, while later work explored adversarial domain confusion and faster frame-based waveform regression (Liu et al., 2015; Engel et al., 2017; Defossez et al., 2018).Defossez et al. (2018) report ∼2,500x sampling speedups over WaveNet autoencoders, but their approach lacks incentives for phase coherency or multimodal distributions and uses many channels, slowing generation and training.Source: §7 · p. 9

8 CONCLUSION

• On NSynth, GANs exceeded a strong WaveNet baseline in fidelity while generating samples tens of thousands of times faster.The result depended on carefully controlling the audio representation used for generative modeling.Source: §8 · p. 9
• The study focused on a specific controlled dataset; broader validation on speech and other natural sounds remains future work.The authors also identify domain transfer and applications of adversarial losses to audio as possible avenues.Source: §8 · p. 10
• Audio GANs share mode-collapse and diversity issues, and future work could combine adversarial losses with encoders or regression losses to better capture the full data distribution.Source: §8 · p. 10

• Figure 5 reports NDB bin proportions for IF-Mel + H and the WaveGAN baseline, evaluated with pitch-60 examples.Source: Figure 5 · p. 13
• Figure 6 reports NDB bin proportions for the WaveNet baseline, evaluated with pitch-60 examples.Source: Figure 6 · p. 13

B TIMBRAL SIMILARITY ACROSS PITCH

• In the Bach Suite No. 1 in G major prelude, pitches synthesized with one latent vector retain a consistent timbre.The figure shows the first 20 seconds, with 10 seconds per row.Source: Figure 7 · p. 14
• Spherical interpolation in latent space makes timbre vary dramatically across the synthesized pitches.The figure contrasts this interpolation with synthesis using a single latent vector.Source: Figure 7 · p. 14

C BASELINE MODEL COMPARISONS

• The 8-bit categorical WaveNet outperforms the 16-bit mixture-of-logistics model, despite the latter’s increased fidelity.The passage attributes this likely to decreased stability of the 16-bit model with pitch conditioning alone.Source: Table 2 · p. 15
• The Waveform GAN baseline performs similarly to WaveGAN on direct waveform generation.The paper therefore uses WaveGAN as its waveform-generation baseline for comparison.Source: Table 2 · p. 15
• Progressive training does not improve performance for the Waveform GAN baseline.The comparison consequently does not include the progressively trained variant.Source: Table 2 · p. 15
• The 16-bit model’s instability with pitch conditioning alone is the proposed reason it underperforms, notwithstanding its increased fidelity.This explanation is presented as likely rather than definitive.Source: Table 2 · p. 15

D TRAINING DETAILS

• A learning rate of 8e-4 and auxiliary-classifier loss weight of 10 performed best across all tested GAN variants.The sweep covered learning rates 2e-4, 4e-4, and 8e-4 and classifier-loss weights 0.1, 1.0, and 10.Source: D TRAINING DETAILS · p. 16
• GANs used box upscaling and downscaling, generator pixel normalization, and a discriminator minibatch-standard-deviation channel; the latent vector had 256 dimensions.The discriminator also appended a pitch classifier, following AC-GAN; the pitch-conditioning vector was a 61-dimensional one-hot vector.Source: D TRAINING DETAILS · p. 16, Table 3 · p. 17
• Real log-magnitudes and phases were independently shifted and scaled to [-0.8, 0.8] before discriminator input to accommodate outliers and the generator’s Tanh output.The maximum range was measured over 100 examples.Source: D TRAINING DETAILS · p. 16
• Each GAN variant trained for 4.5 days on one V100 GPU with batch size 8, processing about 5M examples for nonprogressive models and 11M for progressive models.Progressive models trained on 1.6M examples per stage across seven stages, plus 800k during alpha blending and 800k after blending.Source: D TRAINING DETAILS · p. 16
• The WaveNet baseline used a 30-layer, 512-channel dilated-convolution decoder and converged in 150k iterations over two days on 32 V100 GPUs.The 8-bit model used mu-law encoding with categorical loss, while the 16-bit model used a quantized mixture of 10 logistics; training used synchronous SGD with batch size 1 per GPU.Source: D TRAINING DETAILS · p. 16