PaperlayerView exact revision on arXiv

Source-linked AI summary

Scale Space Diffusion

Soumik Mukhopadhyay, Prateksha Udhayanan, Abhinav Shrivastava

arXiv:2603.08709v1cs.CVcs.AI

TL;DR

Diffusion timesteps and scale-space resolutions exhibit related information hierarchies, raising the question of why highly noisy states require full-resolution processing. The paper integrates scale transitions into diffusion through generalized linear degradations, introduces SSD and Flexi-UNet, and reports faster training and inference with reasonable FID scores.

  • Problem

    Diffusion and scale spaces degrade information hierarchically, but prior scale-changing approaches do not mathematically integrate scale transitions into the diffusion process.

  • Method

    The paper formulates generalized linear diffusion processes, uses resizing for Scale Space Diffusion, and introduces Flexi-UNet to process only relevant network levels across resolutions.

  • Results

    Across CelebA resolutions and ImageNet experiments, the framework is faster during training and inference while achieving reasonable FID scores.

  • Takeaways & Limitations

    Scale Space Diffusion provides an end-to-end diffusion formulation that natively incorporates multiple resolutions rather than approximating scale changes only during inference.

  • Takeaways & Limitations

    The authors identify newer diffusion samplers and progressive curriculum learning as future directions for improving performance and inference speed.

Abstract

from arXiv · show

Diffusion models degrade images through noise, and reversing this process reveals an information hierarchy across timesteps. Scale-space theory exhibits a similar hierarchy via low-pass filtering. We formalize this connection and show that highly noisy diffusion states contain no more information than small, downsampled images - raising the question of why they must be processed at full resolution. To address this, we fuse scale spaces into the diffusion process by formulating a family of diffusion models with generalized linear degradations and practical implementations. Using downsampling as the degradation yields our proposed Scale Space Diffusion. To support Scale Space Diffusion, we introduce Flexi-UNet, a UNet variant that performs resolution-preserving and resolution-increasing denoising using only the necessary parts of the network. We evaluate our framework on CelebA and ImageNet and analyze its scaling behavior across resolutions and network depths. Our project website ( https://prateksha.github.io/projects/scale-space-diffusion/ ) is available publicly.

1. Introduction

Diffusion timesteps and scale-space resolutions organize image information hierarchically, but through different degradation mechanisms. The paper formalizes their connection and proposes integrating scale changes directly into diffusion while reducing unnecessary high-resolution computation.

  • Diffusion noise progressively removes fine facial details, then coarse structure, establishing an information hierarchy across timesteps.
  • Scale spaces produce a similar hierarchy through progressive low-pass filtering, blurring, or downsampling rather than iterative noising.
  • The formal analysis suggests highly noisy states can contain information equivalent to tiny images, motivating scale-aware processing instead of full-resolution computation.
  • Scale Space Diffusion integrates scale transitions into diffusion through generalized linear degradations, using image resizing as the practical degradation.
  • Flexi-UNet uses only relevant network levels and supports resolution-preserving denoising plus next-resolution upscaling across Gaussian-pyramid stages.
  • Across CelebA resolutions, the models are faster during training and inference while achieving reasonable FID scores.

2. Related Work

Prior work uses cascades, multiresolution processes, or added noise to handle scale changes, but does not directly model scale-space transitions within diffusion. This paper positions SSD as an end-to-end pixel-domain formulation addressing that gap.

  • Diffusion Models: DDPM established high-quality image generation without adversarial training but required many Markov-chain sampling steps; DDIM and latent diffusion later targeted acceleration or compact representations.
  • Scale-Space Theory: Cascaded diffusion generates low-resolution images first, then uses super-resolution models to refine upsampled images conditionally.
  • Scale-Space Theory: Existing approaches do not directly incorporate scale-space theory because upsampling noisy states creates correlated noise pixels, often requiring additional high-resolution noise.
  • Scale-Space Theory: UDPM incorporates blurring and subsampling but assumes isotropic posterior covariance, which may fail when resizing kernels overlap.
  • Scale-Space Theory: SSD instead trains a single pixel-domain diffusion model that handles multiple resolutions through a generalized mathematical formulation.

3. Scale Spaces vis-`a-vis Diffusion Timesteps

The paper models information loss in standard diffusion and scale spaces, finding similar degradation trends across increasing noise and decreasing resolution. This correspondence motivates a mathematically integrated multiresolution diffusion process.

  • 3.1. Preliminary: Standard Diffusion Process: Standard DDPM progressively noises data through a Markov chain, and its noisy state combines the original signal with Gaussian noise.
  • 3.1. Preliminary: Standard Diffusion Process: A neural network reverses the process by approximating the posterior and predicting noise, although it may alternatively predict x0.
  • Diffusion States: The diffusion information proxy estimates the fraction of pixels where the signal dominates the noise, averaged over the data distribution.
  • Scale Spaces: For scale spaces, information is modeled as a function of normalized resolution, with lower resolution representing less information under the simplifying assumption.
  • Scale Spaces: Halving spatial dimensions implies one quarter the information in the model, although pixel redundancy means this may not be strictly true.
  • Information Degradation in Diffusion and Scale Spaces: Information degrades similarly as diffusion timestep increases and scale-space resolution decreases, supporting a correspondence between their hierarchies.
  • Information Degradation in Diffusion and Scale Spaces: Independent diffusion processes at varying scales accumulated inference errors, motivating a formulation that models scale changes within diffusion itself.

4. Scale Space Diffusion (SSD)

Scale Space Diffusion generalizes diffusion by replacing scalar signal attenuation with linear degradations such as resizing, allowing scale transitions to occur within the diffusion process. Flexi-UNet supports these variable-resolution states by activating only relevant network levels and handling resolution increases.

  • Generalized Linear Diffusion: The framework replaces the scalar diffusion coefficient with a generic linear operator M_t, allowing blurring or downsampling in the forward process.The resulting transition is x_t = M_t x_{t−1} + η_t with potentially non-isotropic noise covariance.
  • Relationship to Existing Diffusion Models: DDPM is recovered as a special case when M_t = √¯α_t I and σ_t = √1 − ¯α_t, while the generalized framework also supports degradations unavailable to DDPM and Blurring Diffusion.The framework is designed to cover general linear degradations, including downscaling.
  • Scale Space Diffusion: Scale Space Diffusion selects resizing as M_t, combining blurring and downsampling with timestep-dependent attenuation and defining a schedule whose resolution decreases as t increases.This choice changes the signal dimensionality, unlike conventional formulations that preserve dimensionality.

5. Experiments

Experiments evaluate Scale Space Diffusion on CelebA and ImageNet across resolutions, visual samples, resolution schedules, temporal scaling, and architecture variants. The study reports reasonable FID scores while examining computational scaling and qualitative generation.

  • Datasets: Experiments use CelebA and ImageNet, with CelebA evaluated at 64×64, 128×128, and 256×256, and both datasets evaluated at 64×64.CelebA contains around 200K training images, while ImageNet contains around 1.3 million images across 1000 classes.
  • Qualitative results: Figure 4 presents unconditional ImageNet-64 and CelebA-256 samples, including predictions at multiple intermediate scales during SSD.The shown scales are 8, 16, 32, and 64 for ImageNet-64, and 8 through 256 for CelebA-256.
  • Resolution schedules: Five four-level resolution schedules are compared through FIDs at 500k iterations, alongside discrete and continuous mappings from diffusion timesteps to resolution.The schedules use resolutions 64, 32, 16, and 8.
  • Scaling: CelebA experiments analyze temporal scaling by measuring training time for SSD with Flexi-UNet across multiple resolutions.The experiments also report average GFlops per iteration and training time in hours in the main multi-resolution results.
  • Architecture ablation: Architecture ablations compare network variants at two and four levels using CelebA-64 FID at 500K iterations and inference time.Inference time is measured in seconds per generation for 1000 steps, batch size 1, on one A4000.
  • Overall findings: The reported evaluation observes faster training and inference while achieving reasonable FID scores.The framework is analyzed on unconditional image generation using CelebA and ImageNet.

6. Clarifications

The paper clarifies the notation used in Algorithms 1–3, including signal and noise schedule coefficients and the output size of the degradation operator.

  • Notation: In Algorithms 1–3, a_t and a_t−1 denote signal coefficients, σ_t and σ_t−1 denote noise schedule values, and size_out denotes the output dimensions of M.The clarification states that these variable names have their usual meanings.

7. Future Works

The paper keeps optimization choices simple and identifies newer samplers and progressive curriculum learning as future directions for improving performance and speed. Its architectural discussion also notes a resolution-related concern for transformer attention in pixel space.

  • Scope: The study does not use advanced techniques to optimize the framework or architectures, relying instead on standard hyperparameters from the base codebase.The authors state that advanced tuning is outside the work’s scope because of the conference-length manuscript and training expense.
  • Future directions: Newer diffusion samplers and progressive curriculum learning across layers or resolutions are proposed as future directions.The authors associate newer samplers with potential performance and inference-speed improvements and curriculum learning with training optimization.
  • Architecture discussion: The paper notes that DiT was designed for latent spaces, making pixel-space attention’s quadratic complexity a consideration at higher resolutions.The passage frames this as part of the discussion of why transformer-based architectures are not used directly.
  • Evaluation details: Inference-time speeds are additionally documented for DDPM and 25-step DDIM sampling, with a separate comparison of Lanczos sampling and torch.randn calls.These measurements are listed in the inference-time tables.

8. Additional Material

The additional experiments separate the contributions of SSD’s mathematical formulation and Flexi-UNet, test alternative degradations and noise assumptions, and assess resolution-transition correction. Together, they show validity, computational trade-offs, and the importance of non-isotropic sampling.

  • Parts of our Approach: The approach comprises two independently testable parts: the SSD mathematical formulation and the Flexi-UNet architecture.Table 9 summarizes their individual validity and combined use.
  • Validity of SSD: SSD without Flexi-UNet is tested on CelebA-32 by resizing inputs and outputs to a fixed model resolution.The experiment uses five resolutions, from 2 to 32, over 300 training epochs.
  • Alternative Degradation: Gradual one-pixel downsizing from resolution 2 to 32 demonstrates that the generalized degradation theory is not limited to 2× downsampling.Static visual results are reported in Figure 9.
  • Effect of Isotropic Approximation: Isotropic noise produces flat, saturated colors despite preserved facial structure, supporting non-isotropic Gaussian sampling for resolution-dependent reverse processes.The comparison uses the gradual-downsizing setting and is shown in Figure 10.
  • Effectiveness of Flexi-UNet: Flexi-UNet supports multi-resolution diffusion by correcting the covariance mismatch created when noisy latents are upsampled.Upsampling induces covariance proportional to UU⊤ rather than the isotropic covariance assumed at the higher resolution.
  • Effectiveness of Flexi-UNet: Flexi-UNet with SSD achieves better FID values and faster inference than Flexi-UNet without SSD, while rollback around upsampling adds denoising overhead.The comparison is conducted after 500k iterations.

8.3. Resolution Schedules

The resolution schedule maps diffusion time to progressively changing image resolutions. Continuous and discrete schedules support linear, power-controlled, tanh-like, and sigmoid-like transitions.

  • Schedule Definition: The schedule uses resolutions [rmin, 2rmin, ..., 2n−1rmin] and normalized time τ = t/(T−1).The continuous schedule interpolates resolution, while the discrete schedule selects from the resolution list.
  • Linear Schedule: The linear schedule uses f(τ) = 1−τ continuously and i(τ) = n−1−⌊nτ⌋ discretely.These functions determine the resolution associated with each diffusion timestep.
  • Power Schedule: The parameter γ controls convex or concave schedules: γ > 1 gives slower initial decay, whereas γ < 1 gives faster initial decay.The continuous form is f(τ) = 1−(1−τ)^γ.
  • Tanh-like Schedule: The tanh-like schedule is designed to spend more time at middle resolutions by making the schedule steep near the endpoints and flat in the middle.A polynomial approximation is used to obtain the desired shape.
  • Sigmoid-like Schedule: The sigmoid-like schedule instead is steep in the middle and flatter at the beginning and end, using an inverted and normalized polynomial construction.Its continuous form is f(τ) = 1−sigmoid_like(1−τ, γ).

8.4. More Comparisons

Additional comparisons examine inference-step reduction, runtime, alternative diffusion systems, and super-resolution baselines. SSD retains favorable speed behavior while degrading less under fewer sampling steps.

  • More Comparisons: The comparisons include SSD, LDMs, and UDPM using FID, training time, and inference speed at 64 resolution.Table 14 specifically compares SSD and UDPM across these measures.
  • UDPM Comparison: Without adversarial and perceptual losses, UDPM generations become washed out, whereas SSD still outperforms UDPM in FID and training time.UDPM is described as a GAN-like system under this comparison, and it has not been tested above 64 resolution.
  • Number of Inference Steps: Reducing DDPM sampling from 1000 to 250 steps causes a much larger performance degradation for DDPM-ε and DDPM-x0 than for SSD.The comparison also includes DDIM with 25 steps.
  • Number of Inference Steps: SSD degrades far less at 250 sampling steps and provides substantial inference speedups, although final paper comparisons use 1000 steps.Lanczos overhead is negligible because it is applied only at resolution-changing steps.

8.6. Qualitative Results

The qualitative results visualize SSD’s denoising trajectories and generated samples across CelebA and ImageNet resolutions, network depths, and Flexi-UNet configurations.

  • CelebA Results: CelebA-256 visualizations cover Flexi-UNet depths 3L, 4L, and 6L.Each setting includes noisy-state progression, predicted clean images, and generated samples.
  • CelebA Results: CelebA-128 visualizations cover Flexi-UNet depths 3L and 5L, with corresponding noisy-state, predicted-clean-image, and sample figures.The figures document both intermediate trajectories and final outputs.
  • ImageNet Results: ImageNet-64 visualizations show three samples for Flexi-UNet depths 2L and 4L, pairing noisy-state progressions with generated samples.Each pair of rows corresponds to one sample in the noisy-state figures.

9. Mathematical Derivations

This section derives mathematical results for generalized linear diffusion processes, including forward transitions and posterior distributions. It also simplifies the posterior under isotropic marginals using matrix identities and Gaussian algebra.

  • The derivations establish the transition mean and covariance for a generalized linear diffusion process.The proof expands successive linear transitions and Gaussian noise terms to obtain the cumulative representation.
  • The posterior distribution is derived by applying Bayes’ rule and the Markov assumption, then completing the square in the previous state.Quadratic and linear terms determine the posterior normal distribution’s mean and variance.
  • Symmetry of covariance matrices justifies the final transpose-inverse simplification.For symmetric A, the derivation uses (A−1)T = A−1.
  • Under isotropic marginals, the posterior covariance and mean admit simplified closed forms.These simplifications substitute isotropic covariance values into the general posterior expressions.
  • The closed-form simplifications use the Woodbury matrix identity and its corollary to manipulate inverse covariance expressions.Additional substitutions use the forward transition covariance and isotropic marginal covariances.
Loading 2603.08709v1…