PaperlayerView exact revision on arXiv

Source-linked AI summary

Everything at Every Scale: Scale-Invariant Diffusion with Continuous Super-Resolution

Zixin Jessie Chen, Zhuo Chen, Archer Wang, Jeff Gore, William T. Freeman, Congyue Deng, Marin Soljačić

arXiv:2605.26032v1cs.CVcond-mat.stat-mechcs.AIcs.LG

TL;DR

Image generation and super-resolution can be viewed as reversing information loss across scales, but conventional approaches often separate these tasks. SKILD uses a scale-ordered, frequency-space diffusion process so one unconditional reverse trajectory performs both, achieving strong image results and matching higher-order structure in critical Ising data.

  • Problem

    Image generation and super-resolution are practically distinct despite both reversing information loss across scales, motivating a unified framework.

  • Method

    SKILD attenuates frequency content from fine to coarse scales while injecting dataset-spectrum-matched Gaussian noise, then varies only the reverse-process starting timestep.

  • Results

    SKILD reaches FID 2.65 and IS 9.63 on unconditional CIFAR-10, supports 2×–8× ImageNet super-resolution from one checkpoint, and reconstructs critical-Ising four-point correlations close to ground truth.

  • Takeaways & Limitations

    A single unconditional reverse process supports generation and continuous super-resolution while reproducing scale-dependent higher-order statistics in a critical physical system.

  • Takeaways & Limitations

    Sampling currently requires 1000 ancestral steps, and the super-resolution protocol uses exact forward marginals rather than real-world camera or compression degradations.

Abstract

from arXiv · show

Creating images from noise is image generation; reconstructing fine details from coarse inputs is super-resolution. Despite their practical differences, both can be understood as reversing information loss across scales. We introduce $\textbf{SKILD}$, a $\textbf{S}$cale-invariant $\textbf{K}$-Space $\textbf{I}$mage $\textbf{L}$earning $\textbf{D}$iffusion model that unifies generation and continuous super-resolution within a single unconditional framework. Both natural images and critical physical systems exhibit scale invariance, and we leverage it to design a forward process that attenuates image content from fine to coarse scales while injecting spectrum-matched Gaussian noise, making scale an explicit coordinate of the diffusion dynamics. The same trained reverse process performs generation and continuous super-resolution by varying only the starting timestep: $\textit{no task-specific architecture, no conditioning branch, no classifier-free guidance, no retraining per scale factor}$. Empirically, SKILD reaches FID $2.65$ and Inception Score $9.63$ on unconditional CIFAR-10, performs $2\times$--$8\times$ super-resolution on ImageNet from a single unconditional checkpoint while outperforming conditional models across perceptual metrics, and reconstructs critical Ising models whose connected four-point correlations closely track the ground truth.

1 Introduction

SKILD treats image generation and super-resolution as the same reverse process across scales. It orders frequency attenuation from fine to coarse and uses spectrum-matched noise to support both tasks with one unconditional model.

  • Generation starts from pure noise, whereas super-resolution starts from an intermediate state retaining coarser scales.
  • The forward process attenuates high-frequency content before low-frequency content while injecting noise matched to the dataset spectrum.
  • SKILD unifies unconditional generation and continuous super-resolution within a single reverse diffusion process.
  • One unconditional architecture replaces task-specific architectures, conditioning branches, classifier-free guidance, and per-scale retraining.
  • SKILD is competitive with state-of-the-art diffusion models on unconditional CIFAR-10 and achieves the strongest sample quality among frequency-informed diffusion models.
  • At 4× super-resolution on ImageNet-256, one checkpoint outperforms strong diffusion-based conditional baselines on multiple perceptual quality metrics.
  • On a critical Ising scientific dataset, SKILD reproduces explicit self-similar statistics while a strong conditional baseline fails.

2 Related Works

Prior work connects scale structure to image statistics, diffusion, and multiscale reconstruction. SKILD differs by using self-similarity explicitly in frequency space to support unconditional generation and continuous super-resolution without conditioning.

  • Scale-space theory, natural-image statistics, and renormalization-group theory motivate modeling images through smoothing, power laws, and coarse-graining.
  • Cascaded and relay diffusion models compose resolution-specific conditional stages, while other methods connect diffusion to multiscale flows or use Fourier and wavelet structures.
  • Natural-image spectra exhibit similar power-law decay across CIFAR-10, ImageNet-128, and ImageNet-256 over shared frequencies.
  • Unlike prior progressive reconstruction approaches, SKILD continuously attenuates image statistics from fine to coarse frequency modes.
  • Unlike many diffusion super-resolution methods, SKILD uses the low-resolution input as an intermediate state of its own forward process rather than adding extra conditioning.

3 Preliminaries and Motivations

Diffusion reverses a fixed noising process, while scale-invariant natural images and critical physical systems motivate a scale-ordered alternative. These observations lead to a frequency-space forward process designed to respect multiscale structure.

  • Standard diffusion gradually transforms data into isotropic Gaussian noise and generates samples by iterative reverse denoising.
  • Critical physical systems have no characteristic length scale and exhibit power-law observables under coarse-graining and rescaling.
  • Natural-image power spectra approximately follow k^-2 over wide frequency ranges, with differences mainly near finite-resolution cutoffs.
  • CIFAR-10, ImageNet-128, and ImageNet-256 recover approximate k^-2 variance-spectrum scaling across their shared frequency range.
  • These observations motivate a diffusion forward process that respects a hierarchy of structure instead of treating all image scales identically.

4 Scale-Invariant Diffusion in Frequency Space

SKILD defines diffusion as stochastic frequency-space coarse-graining: high frequencies are damped first, while spectrum-matched noise preserves the dataset covariance. Continuous effective resolution then makes generation and super-resolution different reverse-process starting points.

  • 4.1 Formulation: The forward marginal applies frequency-dependent Gaussian attenuation to DCT coefficients and adds noise based on the empirical variance spectrum.
  • 4.1 Formulation: As time increases, high-frequency modes are attenuated before low-frequency modes, while the marginal preserves per-mode covariance in expectation.
  • 4.2 Discretization: The DDPM discretization retains diagonal frequency-space covariances, yielding a Gaussian reverse posterior and ancestral sampling procedure.
  • 4.2 Discretization: The implementation floors small αn(k) values at 10^-6 and introduces a low-frequency cutoff to stabilize high-frequency updates and handle the zero mode.
  • 4.3 Schedules and effective resolution: The schedule parameters control how the damping cutoff moves through frequency space, with λi governing early high-frequency behavior and λf, kc, and θ shaping late low-frequency behavior.
  • 4.3 Schedules and effective resolution: Generation begins where effective resolution is zero, while super-resolution begins from a lower-resolution state; continuous scheduling supplies factors across a continuum.
  • 4.4 Connections: The process extends Gaussian scale-space smoothing with stochastic coarse-graining whose final covariance matches the dataset variance spectrum.
  • 4.4 Connections: The renormalization-group connection is an analogy rather than a claim that SKILD performs an exact RG transformation.

5 Experiments

SKILD is evaluated as a single frequency-space diffusion process for unconditional generation, continuous ImageNet super-resolution, and critical-Ising reconstruction. Across these settings, it achieves competitive or superior results while preserving scale-dependent structure and higher-order correlations.

  • Experimental scope: SKILD is evaluated on CIFAR-10 generation, 2×–8× ImageNet super-resolution, and critical-Ising reconstruction using four-point correlations.The experiments test whether one frequency-space diffusion process supports both image-generation and scale-reconstruction tasks.
  • Unconditional CIFAR-10 generation: CIFAR-10 generation achieves competitive state-of-the-art quality and the best FID and IS among the listed frequency- or scale-informed models.The linear schedule is used for the reported best results.
  • Unconditional CIFAR-10 generation: SKILD is robust across tested log-linear and linear schedules, with most schedules reaching FID below or near 5 and IS near or above 9 within 400K steps.Convergence speed remains schedule-dependent despite the broad quality consistency.
  • ImageNet super-resolution: At 4× ImageNet super-resolution, the unconditional 256-resolution model achieves the best LPIPS, CLIPIQA, and MUSIQ and the second-best SSIM among compared methods.Unlike the conditional baselines, SKILD starts from the corresponding forward marginal without conditioning, class labels, or classifier-free guidance.
  • ImageNet super-resolution: A single ImageNet checkpoint produces 2×–8× reconstructions by varying only the reverse starting timestep, providing a continuum of effective super-resolution factors.The evaluated effective-resolution inputs span 16 × 16 through 64 × 64 for ImageNet-128.
  • Scientific benchmark: For critical Ising fields, SKILD’s connected four-point correlator closely tracks ground truth at every patch size, whereas SR3 deviates significantly.The correlator isolates non-Gaussian dependence beyond mean, variance, and two-point correlations, making it a stronger scale-preservation test than visual texture alone.

6 Conclusions

SKILD makes scale an explicit coordinate of diffusion, allowing unconditional generation and super-resolution to share one reverse trajectory. The paper reports strong image, super-resolution, and critical-Ising results, while identifying sampling cost, low-frequency sensitivity, degradation assumptions, and benchmark scope as limitations.

  • FID 2.65 and IS 9.63 are achieved on unconditional CIFAR-10.
  • A single ImageNet checkpoint supports continuous 2×–8× super-resolution without task-specific conditioning or guidance.
  • SKILD reconstructs critical-Ising fields whose connected four-point correlations closely track the ground truth.
  • Super-resolution uses exact forward marginals as low-resolution inputs, leaving unknown camera and compression degradations for future work.
  • Sampling currently requires 1000 ancestral steps, and unconditional generation remains sensitive to low-frequency structure.
  • The Ising evaluation covers one critical system, while tailored architectures and broader physical benchmarks remain future directions.

A DDPM formulation

The appendix formulates SKILD's scale-dependent diffusion in frequency space and derives its discrete DDPM, continuous SDE, score-training, and sampling procedures. The trajectory removes high frequencies before low frequencies while preserving the dataset spectrum in expectation.

  • A DDPM formulation: High-frequency content is suppressed before low-frequency content, with resolution effectively lost below the SNR threshold.
  • A DDPM formulation: The forward process is written as a one-step Markov transition with ᾱ_n = e^-k^2λ_n, α_n = ᾱ_n/ᾱ_n−1, and β_n = 1 − α_n.
  • A DDPM formulation: When X_0 is known, Bayes' theorem yields the exact reverse posterior q(X_n−1 | X_n, X_0) from the forward transition and marginal.
  • A DDPM formulation: At sampling time, the unknown X_0 is replaced by a network estimate; reported results use ε-prediction, while X_0-, w-, and v-prediction are diagnostic alternatives.
  • B SDE formulation: The continuous formulation uses a linear mode-wise SDE whose drift and diffusion are matched to the desired forward mean and dataset-spectrum variance.
  • B SDE formulation: The forward and reverse SDEs preserve per-mode covariance S_0, with score training using a weighted anisotropic noise target.
  • B SDE formulation: A low-frequency cutoff prevents the DC mode from remaining unchanged, and the weighting function controls the prediction target and can emphasize drift-sensitive errors.
  • B SDE formulation: Experiments use ancestral DDPM sampling rather than the reference EM, ODE, or predictor-corrector samplers; the corrector reduces predictor discretization error.

C Scale invariance and power laws in nature and physics

The paper connects natural images and critical physical systems through approximate scale invariance and self-similarity. Frequency-space power-law spectra motivate the shared coarse-graining framework and provide a benchmark for its scale structure.

  • C.I Dictionary between physics and natural images: Natural images exhibit recurring textures, edges, and structures across scales, reflected statistically in approximate power-law power spectra.
  • C.I Dictionary between physics and natural images: A field-theory sample can be represented as an image, with pixels storing field values and resolution corresponding to inverse lattice spacing.
  • C.I Dictionary between physics and natural images: Critical field theories are scale-invariant under coarse-graining and rescaling, motivating analogous low-pass filtering and zooming for images.
  • C.II Power-law benchmark in frequency space: The appendix specifies DCT/IDCT transforms, frequency normalization, and per-dataset variance fits used throughout the frequency-space analysis.
  • C.II Power-law benchmark in frequency space: A 4/(HW) DCT amplitude convention keeps radial-spectrum magnitudes comparable across resolutions, while fixed spacing extends spectra to higher frequencies.
  • C.II Power-law benchmark in frequency space: All three fitted datasets recover an approximate k^-2 decay over their shared frequency range, with highest-frequency deviations shrinking as resolution grows.

D Supplementary materials for CIFAR-10 experiments

The CIFAR-10 supplements characterize frequency-space schedules, convergence, and failure modes, while testing schedule, sampler, cutoff, and prediction-target choices. They identify low-frequency instability as the main limitation but find broad schedule robustness and feasible step reduction.

  • Spectrum and schedules: Power-law fits recover the k^-2 variance-spectrum decay for CIFAR-10 and ImageNet, with finite-resolution deviations at the highest CIFAR-10 frequencies.The spectra are computed independently for RGB channels.
  • Spectrum and schedules: Higher-frequency modes decay before lower-frequency modes in the tested schedules, unlike scalar DDPM schedules that attenuate all frequency modes uniformly.The supplement compares log-linear and linear frequency schedules with linear and cosine DDPM references.
  • Convergence and failure modes: FID 2.65 and Inception Score 9.63 occur at 160K steps, while later checkpoints show rising FID despite continued IS improvement.For the linear schedule, FID rises to 3.87 and IS reaches 10.03 by 400K steps.
  • Convergence and failure modes: Low-frequency instability makes global color tone and class-level structure harder to correct, especially on object-centric CIFAR-10.The supplement attributes the divergence between improving IS and degrading FID to drift in the marginal class distribution.
  • Ablations and robustness: Most schedules reach FID near or below 5 and all reach IS at or above 9 within 400K steps, indicating limited dependence on precise schedule tuning.Convergence speed varies across schedules, but final converged scores are not strongly schedule-dependent.
  • Ablations and robustness: Reducing diffusion steps from 1000 to 500 leaves FID and IS nearly unchanged, whereas a naive 100-step reduction degrades both metrics.The supplement therefore points to solver or distillation work tailored to the mode-dependent schedule for faster sampling.

E ImageNet super-resolution experiment supplements

The ImageNet supplements validate SKILD’s effective-resolution interpretation and detail continuous super-resolution paths and training configurations. At the selected SNR threshold, forward-process inputs closely match conventional bicubic degradations.

  • Effective-resolution validation: At SNR = 0.1, MSE is within O(10^-4) for every degradation pipeline and PSNR exceeds 30 dB.These measurements compare surviving forward-diffusion signals with matching bicubic-degraded images.
  • Effective-resolution validation: The SNR-defined inputs closely match conventional 4× or 8× degradations while retaining exact forward-process marginals.This validates the low-resolution initialization used in the super-resolution experiments.
  • Schedules and training: The ImageNet-128 paths use 1000-step linear schedules for 4× and 8× super-resolution, with corresponding schedule parameters reported for each factor.The 4× paths target ImageNet-256 and ImageNet-128, while the 8× path targets ImageNet-128.

F Critical Ising super-resolution details

The Ising supplement evaluates super-resolution as reconstruction of scale-dependent critical structure rather than image quality. It specifies exact forward-marginal initialization, correlation-based evaluation, paired testing, and the underlying simulation setup.

  • Evaluation: Critical Ising super-resolution is evaluated with connected four-point correlations across scales instead of image-quality metrics.The reported side lengths are {1, 2, 4, 8, 16, 32, 64}.
  • Simulation setup: The benchmark uses 128×128 binary spin fields at the exact critical inverse temperature, generated with the Wolff cluster algorithm and periodic boundary conditions.The training set contains 90,000 samples and the held-out test set contains 1,000 samples.
  • Initialization: The reverse process starts from the exact forward marginal of a high-resolution spin field, preserving low-frequency modes while higher modes are noise-dominated.At n0 = 1000, the retained low-frequency content corresponds roughly to 32×32 effective resolution.
  • Evaluation: The connected four-point statistic removes contributions explained by pairwise correlations and tests reproduction of non-Gaussian critical structure.It is computed from four-corner, edge, and diagonal spin correlations.
  • Evaluation: SKILD and SR3 share the same 1,000 held-out spin fields, and uncertainty uses 1,000 paired bootstrap resamples with 99% percentile intervals.Pairing preserves sample-wise dependence induced by conditioning on the same low-frequency input.

G Additional image samples

The additional-samples supplement provides uncurated qualitative examples for CIFAR-10 generation and 4× and 8× ImageNet super-resolution at 128- and 256-pixel resolutions.

  • ImageNet samples: Additional figures show 4× super-resolution samples on ImageNet-256.
  • ImageNet samples: Additional figures show 4× and 8× super-resolution samples on ImageNet-128.
  • CIFAR-10 samples: An additional figure presents uncurated generated-image samples from CIFAR-10.
Loading 2605.26032v1…