PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Probabilistic Model Made Slim

Xingyi Yang, Daquan Zhou, Jiashi Feng, Xinchao Wang

arXiv:2211.17106v1cs.CVeess.IV

TL;DR

Diffusion probabilistic models deliver strong image synthesis but are costly in model size and inference, limiting resource-constrained use. Spectral Diffusion slims them by combining wavelet-gated frequency adaptation with spectrum-aware distillation, achieving substantial efficiency reductions while retaining competitive fidelity.

  • Problem

    Diffusion probabilistic models require massive model sizes and hundreds or thousands of inference steps, motivating slimming beyond test-time acceleration.

  • Method

    Spectral Diffusion combines step-adaptive wavelet gating with spectrum-aware knowledge distillation to improve high-frequency generation in compact diffusion models.

  • Results

    8-18× slimmer and 2-5× faster than latent diffusion models, with negligible performance drop across conditional and unconditional image-generation tasks.

  • Takeaways & Limitations

    Frequency-aware design can preserve competitive image fidelity while substantially reducing diffusion-model complexity and size.

  • Takeaways & Limitations

    SD still produces distorted faces and shapes, especially in crowded instance generation such as bananas.

Abstract

from arXiv · show

Despite the recent visually-pleasing results achieved, the massive computational cost has been a long-standing flaw for diffusion probabilistic models (DPMs), which, in turn, greatly limits their applications on resource-limited platforms. Prior methods towards efficient DPM, however, have largely focused on accelerating the testing yet overlooked their huge complexity and sizes. In this paper, we make a dedicated attempt to lighten DPM while striving to preserve its favourable performance. We start by training a small-sized latent diffusion model (LDM) from scratch, but observe a significant fidelity drop in the synthetic images. Through a thorough assessment, we find that DPM is intrinsically biased against high-frequency generation, and learns to recover different frequency components at different time-steps. These properties make compact networks unable to represent frequency dynamics with accurate high-frequency estimation. Towards this end, we introduce a customized design for slim DPM, which we term as Spectral Diffusion (SD), for light-weight image synthesis. SD incorporates wavelet gating in its architecture to enable frequency dynamic feature extraction at every reverse steps, and conducts spectrum-aware distillation to promote high-frequency recovery by inverse weighting the objective based on spectrum magni tudes. Experimental results demonstrate that, SD achieves 8-18x computational complexity reduction as compared to the latent diffusion models on a series of conditional and unconditional image generation tasks while retaining competitive image fidelity.

1. Introduction

Diffusion models offer strong generation quality but remain difficult to deploy because of their large size and computational demands. This paper attributes slim-model fidelity loss to frequency-dependent denoising and proposes Spectral Diffusion to preserve high-frequency detail at lower cost.

  • Motivation: State-of-the-art DPMs can require billions of parameters and hundreds or thousands of inference steps, limiting deployment on resource-constrained platforms.DALL·E 2 is cited as requiring 5.5B parameters and 356 sampling steps.
  • Motivation: Existing efficient-DPM methods mainly accelerate sampling or reduce input size, while largely overlooking model size.These approaches shorten training or inference time but do not remove the deployment burden of large models.
  • Observed limitation: A channel-reduced latent diffusion model preserves coarse structure but poorly recovers skin and hair textures, revealing high-frequency deficiency in slim networks.The deficiency is also visible in the model’s discrete Fourier transform coefficients.
  • Frequency analysis: DPM denoising follows frequency evolution: low-frequency components are recovered first, while high-frequency details are added during later denoising stages.The optimal denoiser is described as a cascade of Wiener filters with growing bandwidths.
  • Proposed method: Spectral Diffusion combines wavelet gating with spectrum-aware distillation to adapt frequency processing across reverse steps and strengthen high-frequency recovery.The distillation objective inversely weights spectrum magnitudes so compact students emphasize weak high-frequency bands.

2. Related Work

The related work positions this study at the intersection of diffusion modeling, efficient inference, and frequency analysis. Unlike prior efficiency efforts centered on sampling or latent-space reduction, it examines frequency behavior to design compact image generators.

  • Diffusion probabilistic models: DPMs are score-based models with annealed noise schedules whose reverse process can be formulated through reverse stochastic differential equations.High-performing implementations commonly use time-conditioned UNets with self- and cross-attention.
  • Efficient diffusion: Efficient diffusion research has reduced sampling steps or operated in lower-dimensional latent spaces, including distillation, shortened reverse processes, and cascade structures.These approaches primarily target low-resource inference efficiency.
  • Frequency analysis: Frequency bias has been observed in neural networks and generative models, which tend to fit low-frequency signals before high-frequency details.GAN generators likewise struggle to construct natural high-frequency details.
  • Paper positioning: This paper examines DPM frequency behavior and uses those properties to achieve realistic image generation at low cost.The focus addresses small diffusion models, a comparatively underexplored setting.

3. Background

Diffusion models perturb data with scheduled Gaussian noise and learn a time-conditioned reverse denoiser to reconstruct samples. Frequency-domain representations then separate image structure and texture into components that can be analyzed across denoising steps.

  • Denoising diffusion: The forward process adds Gaussian noise with variance β_t over T steps until x_T approaches Gaussian white noise.The schedule uses t ∈ [1,T] and 0 < β_1:T < 1.
  • Denoising diffusion: Because Gaussian perturbation admits a closed-form arbitrary-time sample, the forward process avoids tedious numerical integration over intermediate steps.The reverse process is instead modeled as a variational Markov chain.
  • Denoising diffusion: A time-conditioned denoising network parameterizes reverse transitions and is trained through a re-weighted evidence lower bound related to score estimation.The learned score approximates the gradient of the perturbed-data log density.
  • Denoising diffusion: The reverse process can be viewed as following the score from x_T toward higher data likelihood.This provides an intuitive interpretation of iterative denoising.
  • Frequency representation: DFT represents an image using sine and cosine waves at different frequencies and phases, while DWT represents it with multiresolution wavelets.The formulation is introduced for grayscale images but extends to multichannel inputs.
  • Frequency behavior: Diffusion models recover low-frequency structure first and progressively add high-frequency detail, while biased recovery can miss minority frequency patterns.These dynamics motivate analyzing denoising behavior in the frequency domain.
  • Frequency representation: A single-scale DWT decomposes an image into LL, LH, HL, and HH coefficients, with LL capturing low frequency and the other bands capturing textural high-frequency detail.IDWT reconstructs the original-resolution input from these coefficients.

4. Frequency Perspective for Diffusion

Diffusion models recover low-frequency structure before adding high-frequency detail, but small models struggle with minority frequencies and realistic textures. Frequency-domain analysis explains this behavior through evolving denoising filters and frequency-biased objectives.

  • Lite-LDM performs poorly on high-frequency generation despite recovering overall image structure.Its deficiencies include skin and hair textures, reflected by frequency-domain coefficients.
  • Frequency Evolution: Diffusion models recover low-frequency components first and gradually add photorealistic high-frequency details.This progression corresponds to a transition from rough structure to details such as hair, wrinkles, and pores.
  • Frequency Evolution: The optimal denoiser behaves as a cascade of Wiener filters with growing bandwidths across denoising steps.Early steps use narrow-banded filters, while later steps restore additional high-frequency components.
  • Frequency Bias: Diffusion objectives bias recovery toward high-density, low-frequency patterns while underrepresenting long-tail high-frequency components.The loss expectation over p(x0) favors dominant patterns and makes complex textures difficult for small models to generate.
  • Frequency Bias: A toy experiment shows that a small model with M = 64 struggles to recover minority frequencies, whereas a large model with M = 1024 denoises all frequency bands more smoothly.The difference provides concrete evidence of intrinsic high-frequency recovery defects in small diffusion models.

5. Spectral Diffusion Model

Spectral Diffusion addresses frequency evolution and high-frequency bias in slim diffusion models through wavelet gating and spectrum-aware distillation. The design adapts frequency processing across reverse steps while emphasizing minority frequency recovery during training.

  • Spectral Diffusion combines wavelet gating with spectrum-aware distillation to preserve high-frequency details in compact diffusion models.Both modifications target the frequency dynamics and priors identified as central to slim-model degradation.
  • Dynamic Wavelet Gating: Wavelet Gating dynamically selects frequency bands at different reverse steps by decomposing features into wavelet coefficients.WG replaces UNet downsampling and upsampling with DWT and IDWT operations plus soft coefficient gating.
  • Dynamic Wavelet Gating: WG-Down and WG-Up use gating scores to reweight wavelet sub-bands before reconstruction.WG-Down fuses DWT coefficients, while WG-Up reweights four coefficient chunks before IDWT reconstruction.
  • Spectrum-Aware Knowledge Distillation: Spectrum-aware distillation transfers high-frequency knowledge from a teacher diffusion model to a compact student.The student matches teacher outputs and features while using spatial and frequency-aware losses.
  • Spectrum-Aware Knowledge Distillation: Inverse spectrum weighting strengthens minority-frequency recovery by down-weighting majority components.With α < 0, Lfreq emphasizes frequency bands with smaller magnitudes; the full objective uses λs = 0.1 and λf = 0.1.

6. Experiments

Across unconditional, class-conditional, and text-to-image tasks, Spectral Diffusion substantially reduces model cost while retaining competitive image fidelity, though failures remain for some complex visual content. Ablations further show that wavelet gating and frequency-aware distillation improve the recovery of high-frequency details.

  • Unconditional Image Generation: SD achieves 8–14× parameter and computation reduction versus official LDM on unconditional benchmarks while remaining competitive in image fidelity.On one example, SD uses a 21.1M-parameter U-Net and 6.7G MACs to obtain 5.2 FID versus DDPM’s 4.9 FID at 1/37 of its computation cost.
  • Unconditional Image Generation: 4.6× CPU and 3.6× GPU throughput gains over LDM are reported on three of four unconditional datasets.Throughput is measured as time steps per second with batch size 64, averaged over 30 runs.
  • Class-conditional Image Generation: SD reaches 10.6 FID on class-conditional ImageNet, close to ADM’s 10.9 FID despite using a much smaller architecture.ADM uses 553.8M parameters and 1114.2 MACs, whereas Lite-LDM reaches 20.1 FID despite comparable speed.
  • Text-to-Image Generation: On zero-shot MS-COCO text-to-image generation, SD obtains 18.43 FID with a 77.6M model that is 18.7× smaller than LDM.Qualitatively, it composes vivid abstract or cartoon-style images with minimal computational cost, but struggles with human bodies and faces.
  • Ablation Study and Analysis: Removing wavelet gating increases FID from 10.5 to 12.4, while wavelet gating alone improves Lite-LDM’s FID by 2.6.These ablations indicate that wavelet gating improves sample quality in small diffusion models.
  • Ablation Study and Analysis: The frequency term contributes 1.8 FID improvement versus 0.9 FID from the spatial term, and restores hair and architectural textures absent without it.The frequency-aware distillation term produces larger improvements in high-frequency components.

7. Conclusion

Spectral Diffusion targets the computational burden of diffusion models by improving high-frequency generation in compact networks. Its components contribute to improved FFHQ fidelity, while the reported model is substantially slimmer and faster than latent diffusion models.

  • On FFHQ 256 × 256, the complete ablated configuration reaches FID↓ 10.5, improving over the 17.3 baseline.
  • 8-18× slimmer and 2-5× faster than latent diffusion models, with negligible performance drop.
  • High-frequency generation is identified as a primary weakness of diffusion models, linked to frequency evolution and diffusion-process bias.
  • Wavelet gating enables spectrum-dynamic denoising by enhancing different frequency bands at different reverse steps.
  • A large pre-trained network improves high-frequency generation through knowledge distillation.
Loading 2211.17106v1…