PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffWave: A Versatile Diffusion Model for Audio Synthesis

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro

arXiv:2009.09761v3eess.AScs.CLcs.LGcs.SDstat.ML

TL;DR

Waveform synthesis needs high-fidelity generation across conditional and difficult unconditional settings without the sequential cost of autoregressive models. DiffWave uses a non-autoregressive diffusion process trained with an ELBO variant, and achieves strong speech quality, fast synthesis, and superior unconditional-generation evaluations.

  • Problem

    Previous waveform models mainly rely on local conditioning, while unconditional generation must produce very long sequences without conditional information.

  • Method

    DiffWave converts Gaussian noise into waveforms through a non-autoregressive reverse diffusion Markov chain and trains it with a single ELBO-based objective.

  • Results

    DiffWave significantly outperforms WaveGAN and WaveNet in unconditional and class-conditional generation, while matching WaveNet speech quality at MOS 4.44 versus 4.43 and synthesizing orders of magnitude faster.

  • Takeaways & Limitations

    DiffWave provides a versatile raw-audio generator for conditional and unconditional waveform tasks, including neural vocoding and class-conditional generation.

Abstract

from arXiv · show

In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

1 INTRODUCTION

DiffWave addresses the difficulty of unconditional waveform generation with a non-autoregressive diffusion model for conditional and unconditional raw-audio synthesis. It achieves high-quality speech and diverse unconditional samples while synthesizing much faster than WaveNet.

  • Motivation: Unconditional waveform generation is challenging because very long sequences must be generated without conditional information.For one second of speech, this can require 16,000 time-steps, and prior models sometimes produce inferior or word-like samples.
  • Contributions: DiffWave is a non-autoregressive diffusion model for high-fidelity conditional and unconditional raw-audio generation.It uses a single ELBO-based objective without auxiliary losses and avoids architectural constraints required by flow-based models.
  • Results: MOS: 4.44 versus 4.43 for WaveNet, while DiffWave synthesizes orders of magnitude faster using only a few sequential steps.The cited comparison concerns speech quality against a strong WaveNet vocoder.
  • Results: 2.64M parameters and MOS: 4.37 characterize a small DiffWave model that synthesizes 22.05 kHz speech more than 5× faster than real-time on a V100 GPU.The model remains slower than state-of-the-art flow-based models but has a much smaller footprint.
  • Results: DiffWave significantly outperforms WaveGAN and WaveNet in unconditional and class-conditional generation on audio quality and sample diversity.The evaluations include several automatic and human measures.

2 DIFFUSION PROBABILISTIC MODELS

Diffusion probabilistic models add noise through a fixed forward Markov chain and learn a reverse chain that denoises Gaussian latents into data. DiffWave trains this reverse process with an ELBO variant and accelerates synthesis by collapsing many training steps into fewer inference steps.

  • Diffusion process: The diffusion process gradually converts data x0 into whitened latents xT by adding Gaussian noise under a variance schedule.Each transition is fixed as q(xt|xt−1) = N(xt; √(1−βt)xt−1, βtI).
  • Reverse process: The reverse process samples from isotropic Gaussian noise and repeatedly applies pθ(xt−1|xt) to produce the sampled data x0.Its transition distribution is parameterized by a mean and standard deviation conditioned on xt and the diffusion step t.
  • Training: The model is trained by maximizing a variational lower bound because the data likelihood is generally intractable.The ELBO can be expressed using tractable Gaussian KL divergences under the stated parameterization.
  • Training: DiffWave uses an unweighted ELBO variant in which the diffusion step t is sampled uniformly during training.The paper adopts this objective because prior work reported higher generation quality with it.
  • Fast sampling: 6 inference denoising steps can replace 200 training steps by collapsing the reverse process with a carefully designed variance schedule.The design concentrates sampling effort near t = 0, where denoising is observed to be most effective.

3 DIFFWAVE ARCHITECTURE

DiffWave uses a bidirectional dilated-convolution network rather than an autoregressive architecture. This removes the autoregressive generation constraint while retaining a small number of sequential diffusion rounds for long waveforms.

  • Architecture: DiffWave models ϵθ with a feed-forward, bidirectional dilated-convolution architecture because generation is not autoregressive.The architecture is motivated by WaveNet but differs from it by removing the causal generation constraint.
  • Architecture: Generating a waveform of length L requires T rounds of forward propagation, with T much smaller than the waveform length L.The cited example gives T = 50, illustrating that computation proceeds in diffusion rounds rather than one autoregressive step per waveform sample.

C. These layers are grouped into m blocks and each block has n = N

The architecture combines bidirectional dilated residual layers with diffusion-step and optional conditioning inputs. Enlarged receptive fields across reverse-process iterations make the model suitable for unconditional generation.

  • Network layers: Each residual block uses bidirectional dilated convolutions with kernel size 3 and exponentially increasing dilation within the block.The dilation pattern is [1, 2, 4, · · · , 2^(n−1)], and skip connections are summed across residual layers.
  • Diffusion-step conditioning: The network receives the diffusion step t so it can produce different noise predictions for different denoising steps.A 128-dimensional encoding is transformed and broadcast into every residual layer.
  • Conditioning: DiffWave supports local mel-spectrogram conditioning and global discrete-label conditioning through embeddings injected into residual layers.Global labels can represent information such as speaker IDs or word IDs.
  • Unconditional generation: Unconditional generation requires receptive field r ≥ 2L so output units cover the full utterance input.This requirement creates a substantial architecture-design challenge for long waveforms.
  • Unconditional generation: A 30-layer dilated convolution has receptive field r = 6139, covering only 0.38s of 16 kHz audio.The paper notes that deeper layers and larger dilation cycles degraded quality in its experiments.
  • Unconditional generation: Iterating from xT to x0 can increase the effective receptive field up to T × r, supporting unconditional generation.This expands context across reverse-process iterations rather than requiring the entire receptive field in one network pass.

4 RELATED WORK

Prior waveform-synthesis work largely emphasized conditional neural vocoding, while unconditional time-domain audio generation remained difficult. DiffWave is positioned against autoregressive, GAN-based, and likelihood-based approaches through its reported quality, diversity, and efficiency advantages.

  • Most prior waveform models condition synthesis on informative local representations such as mel spectrograms or aligned linguistic features.
  • Unconditional time-domain audio generation is challenging because models must learn all dataset variations from very long waveforms without conditional information.
  • Autoregressive models can produce made-up word-like sounds or inferior samples in unconditional settings, while GAN-based models are considered suitable because of mode-seeking behavior.
  • DiffWave's variational lower-bound objective can focus on major data variations and alleviate model-capacity requirements for unconditional waveform generation.
  • DiffWave requires fewer vocoder parameters than WaveGrad, using 2.64M versus 15M for Base models and 6.91M versus 23M for Large models.
  • DiffWave is easier to train than GAN- or VAE-based models because it avoids mode collapse, posterior collapse, and instability from jointly training two networks.

5 EXPERIMENTS

DiffWave is evaluated across neural vocoding, unconditional generation, and class-conditional generation, with comparisons spanning quality, speed, diversity, and model footprint. Across these tasks, it achieves high speech quality, faster-than-real-time synthesis, and stronger unconditional and class-conditional results than the cited baselines.

  • DiffWave is evaluated on neural vocoding, unconditional generation, and class-conditional waveform generation.
  • 5.1 NEURAL VOCODING: 5.6× faster than real-time: DiffWave BASE (Fast) retains good audio fidelity on a V100 GPU.DiffWave BASE (T = 20) and BASE (T = 40) achieve 2.1× and 1.1× faster-than-real-time synthesis without engineering optimization.
  • 5.2 UNCONDITIONAL GENERATION: 3.39 versus 1.43 and 2.03 MOS: DiffWave produces higher-quality unconditional samples than WaveNet and WaveGAN.Automatic metrics also indicate better quality, diversity, and matching of the training data's marginal label distribution.
  • 5.3 CLASS-CONDITIONAL GENERATION: 3.50 versus 1.58 MOS: DiffWave significantly outperforms WaveNet in class-conditional generation.DiffWave also exceeds 91% classification accuracy and achieves an mIS six times higher than WaveNet.
  • 5.4 ADDITIONAL RESULTS: DiffWave's unconditional model performs speech denoising despite not being trained on denoising or exposed to the evaluation noise types.Noisy utterances are fed into the reverse process at t = 25 to obtain outputs.

6 CONCLUSION

DiffWave is presented as a versatile raw-waveform generator that performs strongly across vocoding and generation tasks while retaining clear opportunities for further optimization.

  • DiffWave models fine waveform details conditioned on mel spectrograms and matches a strong autoregressive neural vocoder in speech quality.The conclusion also reports realistic voices and consistent word-level pronunciations in unconditional and class-conditional generation.
  • DiffWave captures large data variation in unconditional and class-conditional generation while producing realistic voices and consistent word-level pronunciations.
  • The authors identify longer-utterance generation as an open problem because DiffWave potentially has very large receptive fields.
  • Inference-speed optimization remains important because DiffWave is still slower than flow-based models.
  • The authors observe that the most effective reverse-process denoising steps occur near x0, suggesting that a smaller T may be possible.

A PROOF OF PROPOSITION 1

The proposition proof derives the forward diffusion marginal and expands the ELBO into tractable Gaussian KL-divergence terms, completing its computation.

  • The ELBO expansion consists of tractable KL divergences, including Gaussian terms with shared covariance matrices.
  • The ELBO is expressed using the reverse-process likelihood, the latent prior, and the forward-process distribution.
  • The forward marginal q(xt|x0) is derived by composing Gaussian diffusion steps and remains Gaussian with mean √¯αtx0 and covariance (1 −¯αt)I.
  • Bayes rule and the Markov-chain property are used to derive q(xt−1|xt, x0).
  • Using xt = √¯αtx0 + √1 −¯αtϵ and the boundary expression for x1, the proof finishes the ELBO computation.

B DETAILS OF THE FAST SAMPLING ALGORITHM

DiffWave accelerates sampling by using a user-defined, shorter reverse schedule aligned with the training noise levels, without retraining the model.

  • Tinfer ≪ T denotes the number of reverse-process sampling steps, making inference shorter than the training diffusion chain.
  • A user-defined variance schedule can be independent of the training schedule, with corresponding constants computed analogously.
  • Each sampling noise level is aligned to the training schedule by matching cumulative noise levels or interpolating between adjacent training levels.
  • The fast sampler begins from xTinfer sampled from the isotropic Gaussian latent prior and iteratively denoises toward the waveform.
  • In neural vocoding, the reported schedules use six variance values for both DiffWave LARGE and DiffWave BASE, with different final values.LARGE uses a final value of 0.7, whereas BASE uses 0.5.
  • The fast algorithm reuses a checkpoint trained with large T and does not modify the training procedure.

C DETAILS OF THE MODEL ARCHITECTURE

The architecture figures document DiffWave’s tensor-processing network and the receptive fields of its output units.

  • Figure 3 depicts the DiffWave network architecture for modeling ϵθ(xt, t), including tensor shapes and activation functions at each stage.
  • Figure 3 defines B as batch size, C as the number of residual/skip channels, and L as data dimension.
  • Figure 4 shows the receptive fields of the output units within the DiffWave network.

D DETAILS OF AUTOMATIC EVALUATION METRICS IN SECTION 5.2 AND 5.3

The paper evaluates generated audio using metrics that compare feature distributions, label distributions, and sample allocation across clusters. These measures capture complementary aspects of generation quality and diversity.

  • Feature-distribution metrics: FID measures the Wasserstein-2 distance between Gaussian distributions fitted to training and generated audio features.The fitted distributions are parameterized by their mean vectors and covariance matrices.
  • Label-distribution metrics: IS evaluates generated samples using the feature-derived multinomial label distributions and their marginal label distribution.The marginal label distribution is computed over generated samples.
  • Label-distribution metrics: mIS is presented as a modified Inception Score for evaluating generated audio.
  • Label-distribution metrics: AM Score accounts for the prior distribution of the training data’s feature-derived labels, unlike IS.This incorporates the marginal label distribution of the training data into evaluation.
  • Cluster-allocation metrics: NDB clusters training features into 50 bins and counts bins with statistically different training-versus-generated sample proportions.Generated samples are assigned to their nearest training-data cluster before the comparison.
Loading 2009.09761v3…