Source-linked AI summary
WaveGrad: Estimating Gradients for Waveform Generation
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, William Chan
TL;DR
Waveform generation must balance the high fidelity of autoregressive models against their many sequential computations. WaveGrad estimates conditional data-density gradients and iteratively refines Gaussian noise with a gradient-based sampler. It produces high-fidelity audio in as few as six iterations, matching WaveRNN quality with fewer sequential operations while outperforming adversarial non-autoregressive baselines.
Problem
Autoregressive waveform models offer high fidelity but require many sequential computations, creating a challenge for faster-than-real-time audio generation.
Method
WaveGrad combines score matching and diffusion probabilistic modeling to estimate conditional waveform data-density gradients and sample from Gaussian noise using gradient-based updates.
Results
WaveGrad generates high-fidelity audio in as few as six iterations, matches WaveRNN sample quality with fewer sequential operations, and outperforms non-autoregressive baselines.
Takeaways & Limitations
A single continuously noise-conditioned model can trade inference computation for output quality by changing the number of inference iterations without retraining.
Takeaways & Limitations
The noise schedule must keep the terminal distribution close to standard normal while beginning with small β values to preserve fine-grained details and reduce background static noise.
Abstract
from arXiv · showhide
This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion probabilistic models. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram. WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality. We find that it can generate high fidelity audio samples using as few as six iterations. Experiments reveal WaveGrad to generate high fidelity audio, outperforming adversarial non-autoregressive baselines and matching a strong likelihood-based autoregressive baseline using fewer sequential operations. Audio samples are available at https://wavegrad.github.io/.
1 INTRODUCTION
WaveGrad addresses the need for faster waveform generation by estimating data-density gradients and using gradient-based, non-autoregressive sampling. It combines score matching and diffusion-model ideas to produce high-fidelity audio with few refinement steps.
- Motivation: Autoregressive waveform models provide high-fidelity audio but require many sequential computations, challenging faster-than-real-time deployment.This limitation matters for applications such as digital voice assistants on smart speakers.
- WaveGrad: WaveGrad estimates gradients of the conditional waveform data log-density instead of estimating the density itself.The model is described as simple to train and implicitly optimizing a weighted variational lower-bound on log-likelihood.
- WaveGrad: WaveGrad is non-autoregressive and uses a constant number of generation steps during inference.This provides a sampling approach distinct from models requiring a large number of sequential waveform computations.
- WaveGrad: WaveGrad combines score matching and diffusion probabilistic models for conditional speech synthesis.Its sampler begins from Gaussian noise and uses gradient-based updates during inference.
- Results: WaveGrad outperforms adversarial non-autoregressive models and matches a strong autoregressive model in subjective naturalness.The paper reports high-fidelity samples using as few as six refinement steps.
2 ESTIMATING GRADIENTS FOR WAVEFORM GENERATION
WaveGrad adapts score matching and diffusion probabilistic models to conditional waveform generation by learning data-density gradients and sampling through iterative denoising. Its continuous noise-level conditioning supports flexible inference trajectories and explicit computation–quality trade-offs.
- Score matching: The Stein score function is the gradient of the data log-density, and Langevin dynamics uses it for gradient-based sampling.Score matching trains a neural network to learn this function directly for inference.
- Diffusion formulation: The diffusion forward process adds Gaussian noise through a Markov chain, while inference starts from Gaussian noise and progressively denoises the waveform.The model is trained on noisy waveform pairs and predicts an update direction at each iteration.
- WaveGrad model: WaveGrad extends denoising score matching to conditional waveform generation p(y | x), learning the gradient of the conditional data density.The conditioning features x can include mel-spectrograms, linguistic features, or predicted acoustic features.
- Noise schedule and conditioning: Noise-schedule and iteration-count tuning are critical for high-fidelity audio because poorly tuned training support can cause poor sampling convergence.The issue is especially important when using a small number of denoising steps.
- Noise schedule and conditioning: WaveGrad conditions on a continuous noise level rather than a discrete iteration index, using hierarchical sampling over intervals of a predefined noise schedule.This replaces direct uniform sampling over the full continuous range, which performed poorly empirically.
- Noise schedule and conditioning: A single continuously conditioned model can use different iteration counts at inference, trading computation for output quality without retraining.This enables inference over multiple sampling trajectories and faster hyperparameter search.
3 RELATED WORK
WaveGrad connects conditional waveform generation to diffusion and score-matching methods while distinguishing its continuous noise-level conditioning from related approaches.
- WaveGrad applies diffusion probabilistic models to conditional waveform generation, whereas related diffusion work addressed unconditional image synthesis.
- Its objective resembles Noise Conditional Score Networks, which condition models on a continuous noise-level scalar.
- Unlike denoising and sliced score matching, WaveGrad conditions on the noise level while learning conditional waveform distributions.
- Concurrent diffusion waveform work used a discrete iteration index and reported 20 refinement steps, whereas WaveGrad uses continuous noise-level conditioning and as few as six.
- WaveGrad’s architecture is inspired by GAN-TTS’s generator but omits batch normalization.
4 EXPERIMENTS
Experiments compare WaveGrad with neural vocoders under objective and subjective evaluations, including noise-schedule and iteration-count studies. Continuous noise-level conditioning supports efficient synthesis while preserving high reported naturalness.
- WaveGrad matched autoregressive WaveRNN quality and outperformed the non-autoregressive baselines in subjective evaluation.
- 100 times slower: WaveRNN achieved RTF 20.1, while six-iteration WaveGrad achieved RTF 0.2 on the same NVIDIA V100 GPU.
- Six-iteration WaveGrad achieved an MOS above 4.4 while retaining the reported speed advantage.
- Inference schedules should combine small initial β values with sufficiently large β values to keep terminal KL divergence small and reduce background static noise.
- Continuous noise-level conditioning generalized better than discrete-index conditioning, especially with few iterations, and six-iteration degradation was not significant.
- RTF 0.2 on an NVIDIA V100 GPU and RTF 1.5 on a 16-core Intel Xeon CPU were reported for the six-iteration model.
5 CONCLUSION
WaveGrad estimates data-density gradients to iteratively refine Gaussian noise into mel-spectrogram-conditioned waveforms. The paper reports high-fidelity audio with six iterations and quality matching a strong autoregressive baseline.
- WaveGrad starts from Gaussian white noise and iteratively updates the signal with a mel-spectrogram-conditioned gradient-based sampler.
- Six iterations suffice for WaveGrad to generate high-fidelity audio samples.
- WaveGrad matched a strong autoregressive baseline while requiring a constant number of non-autoregressive generation steps.
A NEURAL NETWORK ARCHITECTURE
WaveGrad’s architecture combines multiscale waveform processing with mel-spectrogram and noise-level conditioning. Its FiLM modules inject conditioning into residual upsampling blocks without batch normalization.
- Five upsampling blocks convert 80 Hz mel-spectrogram features toward 24 kHz raw audio using factors 5, 5, 3, 2, and 2.
- UBlocks contain residual blocks and dilated convolutions, using larger dilation patterns in early blocks to support broad receptive fields.
- Downsampling blocks process noisy waveforms before the network prediction and use strided convolutions to reduce temporal resolution.
- FiLM combines noisy-waveform and mel-spectrogram information, then supplies scale and bias vectors for feature-wise affine transformations.
- The noise level is represented through sinusoidal positional embeddings, with continuous conditioning replacing the discrete iteration index.
- Batch normalization is omitted because minibatches contain samples with different noise levels, making batch statistics dependent on sampled noise.
B NOISE SCHEDULE
WaveGrad evaluates linear and Fibonacci noise schedules for different iteration counts and explores schedule choice during inference. A mismatch between training and inference schedules degraded performance.
- Schedule choices: For 1000 and 50 iterations, WaveGrad uses linearly increasing variance schedules, while 25 iterations use a Fibonacci-based schedule.The schedules are Linear(1 × 10−4, 0.005, 1000), Linear(1 × 10−4, 0.05, 50), and Fibonacci(N), respectively.
- Schedule consistency: A mismatch between the training and inference noise schedules degraded performance.When a fixed schedule was used during training, the same schedule was used during inference.
- Inference schedules: Inference can use a manually selected schedule, including a six-iteration schedule explored by sweeping βs.The maximal iteration S is set to 1000, with noise levels precomputed from Linear(1 × 10−6, 0.01, 1000).
- Inference schedules: LS-MSE was used to tune inference-schedule hyperparameters without training separate models.
C RESULTS FOR LJ SPEECH
The LJ Speech experiments train and evaluate WaveGrad on subsets of a publicly available audiobook dataset, using mel-spectrograms from ground-truth audio as conditioning features.
- Dataset and setup: WaveGrad was trained on 12,764 LJ Speech utterances totaling 23 hours and evaluated on a held-out 130-utterance subset.The utterances were audiobook recordings segmented into clips of up to 10 seconds.
- Dataset and setup: Mel-spectrograms computed from ground-truth audio were used as conditioning features during training.
D SUBJECTIVE LISTENING TEST PROTOCOL
The subjective evaluation uses a 1,000-sentence test set and asks native English-speaking listeners to rate stimulus naturalness on a five-point Likert scale.
- Rating procedure: Subjects rated the naturalness of 1,000 test sentences using a five-point Likert scale from Bad to Excellent.Ratings used 0.5-point increments, with 1 denoting Bad and 5 denoting Excellent.
- Reported scores: The reported MOS results are organized in tables for the LJ Speech dataset and for prior models with confidence intervals and conditioning-feature labels.The prior-work table identifies linguistic and mel-spectrogram conditioning features.
- Presentation procedure: Each subject evaluated up to six stimuli, which were randomly selected and presented in isolation.
- Participants and setting: Participants were paid native English speakers living in the United States and were instructed to use headphones in a quiet room.
E SUBJECTIVE SCORES REPORTED IN THE PRIOR WORK
Prior-work MOS values are reported for models using the same female speaker, but the studies are not directly comparable because their experimental conditions differ.
- Comparability: Prior-work MOS comparisons are limited by differences in training data, sampling rates, conditioning features, and evaluation sentences.