PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior

1 INTRODUCTION

• A standard Gaussian prior may make speech diffusion inefficient when waveform segments, including voiced and unvoiced regions, have highly variable distributions.The model may struggle to cover all data modes, leading to training inefficiency and potentially spurious diffusion trajectories.Source: §1 · p. 2
• PriorGrad computes an instance-level Gaussian prior from conditional data statistics, such as mel-spectrogram frames for vocoders or phonemes for acoustic models.The resulting mean and variance are intended to make the noise more similar to the target data distribution.Source: §1 · p. 2
• Experiments on LJSpeech report faster convergence, improved perceptual quality, and greater tolerance to reduced network capacity for PriorGrad.The implementations use recently proposed diffusion-based speech generative models.Source: §1 · p. 2
• The study examines adaptive non-standard Gaussian priors for conditional generation and reports accelerated inference and improved quality in spectral and waveform speech models.The authors describe this as a systematic investigation of non-standard Gaussian forward priors for conditional generative models.Source: §1 · p. 2

2 BACKGROUND

• DDPMs define a forward Markov process that progressively destroys data into noise and a learned reverse process that transforms prior noise back into data.The reverse transition is parameterized by a deep neural network.Source: §2 · pp. 2–3
• A user-defined noise schedule determines the forward transitions, and the noisy data distribution approaches a standard Gaussian when the terminal cumulative signal coefficient is sufficiently small.The forward transition at step t uses noise level β_t.Source: §2 · pp. 2–3
• The original DDPM setup fixes a standard-Gaussian prior and uses a simplified training objective that Ho et al. (2020) report provides higher sample quality.Its loss omits the weighting factor for each term.Source: §2 · p. 3
• Conditional speech diffusion includes waveform generation from mel-spectrograms and text-to-spectrogram generation; PriorGrad targets training efficiency in these models.The section describes existing methods as requiring very large computing resources to train and generate high-quality samples.Source: §2 · p. 3
• Prior work explores several informative-prior designs, while PriorGrad addresses diffusion training inefficiency with a data-dependent prior intended to improve trajectories and reduce spurious modes.Unlike Popov et al. (2021), PriorGrad directly augments the forward prior from data without restricting encoder representations, and uses a fixed informative prior.Source: §2 · pp. 3–4

3 METHOD

• PriorGrad generalizes diffusion models to a non-standard Gaussian prior, with the method motivated by whether noise closer to the data distribution can improve efficiency.The section presents training and sampling algorithms and a theoretical analysis of the proposed prior.Source: §3 · p. 4

3.1 GENERAL FORMULATION

• The forward prior N(µ, Σ) is computed adaptively from conditional data at the instance level and used as the forward-diffusion target.The training and sampling procedures are augmented by the data-dependent prior; its computation depends on the application.Source: §3.1 · pp. 4–5, §3.2 · p. 5
• Proposition 1 extends the ELBO in Equation 6 to use a non-standard Gaussian prior N(µ, Σ), whose mean and variance are extracted from data, unlike original DDPM’s N(0, I) prior, used without any assumption on data.Source: §3.1 · p. 4

3.2 THEORETICAL ANALYSIS

• Under a linear noise-prediction function and equal prior determinants, Proposition 2 states that the adaptive-prior ELBO loss is no greater than the standard-prior loss.Equal determinants give the two Gaussian priors equal entropy, making this a fair comparison.Source: §3.2 · p. 5
• Under a linear predictor, the adaptive-prior loss Hessian is proportional to I, giving condition number 1 and the smallest condition number described for this comparison.The standard-prior Hessian is proportional to Σ + I; the analysis links a smaller condition number to faster convergence.Source: §3.2 · p. 5

4.1 PRIORGRAD FOR VOCODER

• For the vocoder, PriorGrad estimates noise with covariance Σ rather than identity covariance, while synthesizing a waveform conditioned on a mel-spectrogram.The model also receives a diffusion-step embedding and mel-spectrogram conditioning through conditional projection layers.Source: §4.1 · pp. 5–6
• A normalized frame-level mel-spectrogram energy supplies the adaptive diagonal variance, serving as a proxy for waveform standard deviation.The frame-level variance is upsampled to waveform level; the prior is zero-mean and its minimum standard deviation is clipped to 0.1 for numerical stability.Source: §4.1 · p. 6

4.2 EXPERIMENTAL SETUP

• Experiments used LJSpeech: 13,000 training clips, 5 validation clips, and 95 test clips, with a 2.62M-parameter model trained for 1M iterations.Training took approximately seven days on one NVIDIA A40 GPU; inference used the default 50-step and fast 6-step schedules.Source: §4.2 · p. 6

4.3 EXPERIMENTAL RESULTS

• PriorGrad showed significantly faster LS-MAE convergence, and its auditory samples removed background white noise early in training.The baseline instead had to learn the reverse process from standard Gaussian noise, which the authors describe as containing little information.Source: §4.3 · p. 7, Figure 4 · p. 7
• PriorGrad outperformed baseline DiffWave in subjective MOS with 2 times fewer training iterations and on the reported objective speech and distance metrics.The objective measures included MR-STFT, MCD, F0 RMSE, and debiased Sinkhorn divergence; the supplied passages give no metric values.Source: §4.3 · p. 7, Table 1 · p. 7, Table 3 · p. 7
• With reduced capacity, PriorGrad’s performance degradation was smaller than DiffWave’s, and the small PriorGrad model maintained quality close to the larger baseline.The reduced model used 1.23M parameters, with the dilated convolutional layer width halved; both small models were trained for 1M steps.Source: §4.3 · p. 7, Table 2 · p. 7

5.1 PRIORGRAD FOR ACOUSTIC MODEL

• For each phoneme, the adaptive prior uses 80-dimensional diagonal mean and variance statistics aggregated from aligned training frames, then upsamples them to frame level.Phoneme-to-frame alignment comes from the Montreal forced alignment toolkit.Source: §5.1 · p. 8
• The decoder receives mean-shifted noisy mel-spectrograms as input and estimates injected noise, while the aligned phoneme-encoder output conditions its dilated convolution layers.Source: §5.1 · p. 8

5.2 EXPERIMENTAL SETUP

• Experiments used a 12-layer DiffWave decoder, 400 training diffusion steps, and a fast reverse-sampling schedule with 6 steps.The setup compared 3.5M- and 10M-parameter decoder capacities.Source: §5.2 · p. 8

5.3 EXPERIMENTAL RESULTS

• A 10M-parameter PriorGrad generated high-quality mel-spectrograms after 60K iterations, while the baseline did not match that performance after 300K.The comparison is reported in Table 4.Source: §5.3 · p. 8, Table 4 · p. 9
• The appendix comparison reports PriorGrad as a new state-of-the-art acoustic model.Source: §5.3 · p. 9
• The 3.5M-parameter PriorGrad outperformed both small and large baselines and achieved perceptual quality almost identical to the 10M PriorGrad.The authors describe this as improved parameter efficiency and tolerance to reduced network capacity.Source: §5.3 · p. 9

• Across speech generative models, PriorGrad is reported to improve convergence, denoising of white background noise, perceptual quality, and parameter efficiency.The authors state these benefits make diffusion-based generative models more practical for deployment.Source: §6 · p. 9
• Applying PriorGrad beyond speech, such as to image super-resolution or depth-map-conditioned synthesis, is presented as a potential direction for future work.Source: §6 · p. 9
• Computing data-dependent statistics or proxies may require task-specific design, and suitability may depend on the granularity of conditional information.The authors identify generalizing this design as future work.Source: §6 · p. 9

• The derivation centers the clean input as x̃0 = x0 − µ before forming the noisy sample.Source: §A.1 · p. 13
• The conditional noisy-sample distribution has mean √ᾱt x̃0 and covariance (1 − ᾱt)Σ.Source: §A.1 · p. 13
• A KL term in the ELBO is expressed as a timestep-weighted squared noise-prediction error measured with Σ^-1.Source: §A.1 · p. 13

• With a linear noise predictor and det(Σ) = det(I), the minimum loss using (µ, Σ) is no greater than the minimum loss using (0, I).Source: §A.2 · p. 13
• The proof assumes a scalar-function noise predictor with parameter freedom d and a diagonalized parameter constraint in Σ’s eigenbasis.Source: §A.2 · p. 14
• The adaptive-prior loss Hessian has condition number 1, whereas the standard-prior Hessian is expressed as c1I + c2Σ.Source: §A.2 · p. 14

A.3 EXPLORATION ON THE CONDITIONAL INFORMATION SOURCE OF PRIORGRAD VOCODER

• Frame-level spectral energy produced the best alignment between prior labels and unobserved waveform data, with consistent quality.Figure 6 compares waveform scatter plots for V/UV-, phoneme-, and energy-based priors.Source: §A.3 · p. 15, Figure 6 · p. 15
• V/UV labels provide only two prior variances, making them an overly coarse assumption about the waveform distribution.Source: §A.3 · p. 15
• Phoneme-level priors offer finer granularity than V/UV priors, but their training-set variance statistics can misalign with unobserved waveforms.Source: §A.3 · p. 15

A.4 DESCRIPTION OF OBJECTIVE METRICS FOR PRIORGRAD VOCODER

• LS-MAE measures spectral regression error between synthesized and ground-truth log-mel spectrograms.The same STFT function is used for the conditional input of DiffWave and PriorGrad.Source: §A.4 · p. 15
• MR-STFT measures spectral distance across multiple STFT resolutions, whose windows capture speech time-frequency distributions.The resolution hyperparameter follows Parallel WaveGAN.Source: §A.4 · p. 15
• MCD measures distance between mel cepstra and is a widely adopted speech metric.Source: §A.4 · p. 15
• F0 RMSE measures fundamental-frequency accuracy, with F0 estimated using SWIPE at a hop size of 128.Source: §A.4 · pp. 15–16
• Debiased Sinkhorn divergence approximates Wasserstein distance between distributions and assesses distances from prior noise or generated samples to real data.The test-set average uses a Monte Carlo estimate with 100 samples per data point and T_infer = 6 for DiffWave and PriorGrad.Source: §A.4 · p. 16

A.5 ADDITIONAL DETAILS OF PRIORGRAD ACOUSTIC MODEL

• The baseline acoustic model was sensitive to training and sampling noise schedules, whereas PriorGrad was more robust to schedule choices.The baseline used a grid search over sampling schedules; relatively high final-step beta values were important for baseline quality.Source: §A.5 · p. 16

A.6 COMPARISON TO TRAINABLE DIFFUSION PRIOR

• Jointly trainable prior approaches failed to estimate or refine informative priors; acoustic estimates were noisy or collapsed, while the vocoder estimator’s variance collapsed to zero.The authors caution that these results do not establish that jointly trainable diffusion priors are impossible; the compared acoustic-model results are described in Table 6.Source: §A.6 · p. 17, Table 6 · p. 17

A.7 COMPARISON TO STATE-OF-THE-ART

• At Tinfer = 6, PriorGrad matched baseline DiffWave at Tinfer = 50; acoustically, PriorGrad at Tinfer = 12 matched Grad-TTS quality at Tinfer = 10 with approximately 1.75 times faster RTF.At Tinfer = 2, PriorGrad remained robust, while the baseline acoustic model with N(0, I) suffered significant degradation.Source: §A.7 · pp. 18–19

A.8 MOS EVALUATION DETAILS

• The MOS protocol used 20 randomly selected test clips and 450 evaluations per model, with 150 unique qualified listeners.Listeners scored naturalness on a five-point scale; qualification required an approval rate above 98% and more than 100 previously approved evaluations.Source: §A.8 · p. 19

A.9 AUDIO SAMPLES DEMO PAGE

• An audio-samples demo page is provided for the study.Source: §A.9 · p. 19