PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models, Image Super-Resolution And Everything: A Survey

Brian B. Moser, Arundhati S. Shanbhag, Federico Raue, Stanislav Frolov, Sebastian Palacio, Andreas Dengel

arXiv:2401.00736v3cs.CVcs.AIcs.LGcs.MM

TL;DR

Diffusion models have advanced image super-resolution by producing realistic images that align closely with human judgments, but their growing literature and unresolved challenges make the field difficult to navigate. This survey unifies foundational principles, reviews SR-specific methods and research directions, and highlights computational demands and other open challenges.

  • Problem

    The rapid expansion of diffusion-model research in image super-resolution creates a need for a unified, SR-focused account distinct from broader surveys.

  • Method

    The survey synthesizes diffusion-model foundations and analyzes SR methods across state domains, conditioning, guidance, corruption spaces, sampling, and zero-shot learning.

  • Results

    Diffusion models generate high-quality super-resolution images that human raters perceive as more realistic than outputs from other generative models such as GANs.

  • Takeaways & Limitations

    The survey organizes current image super-resolution research with diffusion models and identifies emerging trends and directions for future investigation.

  • Takeaways & Limitations

    Diffusion models impose substantial computational demands and memory costs, motivating research into smaller domains and more efficient sampling.

Abstract

from arXiv · show

Diffusion Models (DMs) have disrupted the image Super-Resolution (SR) field and further closed the gap between image quality and human perceptual preferences. They are easy to train and can produce very high-quality samples that exceed the realism of those produced by previous generative methods. Despite their promising results, they also come with new challenges that need further research: high computational demands, comparability, lack of explainability, color shifts, and more. Unfortunately, entry into this field is overwhelming because of the abundance of publications. To address this, we provide a unified recount of the theoretical foundations underlying DMs applied to image SR and offer a detailed analysis that underscores the unique characteristics and methodologies within this domain, distinct from broader existing reviews in the field. This survey articulates a cohesive understanding of DM principles and explores current research avenues, including alternative input domains, conditioning techniques, guidance mechanisms, corruption spaces, and zero-shot learning approaches. By offering a detailed examination of the evolution and current trends in image SR through the lens of DMs, this survey sheds light on the existing challenges and charts potential future directions, aiming to inspire further innovation in this rapidly advancing area.

I. INTRODUCTION

Image super-resolution is ill-posed because one low-resolution input can correspond to multiple plausible high-resolution images. This survey organizes foundations, models, evaluation resources, and emerging diffusion-model directions for image SR.

  • SR foundations: Image SR transforms low-resolution images into high-resolution counterparts, but multiple HR images can plausibly explain one LR input.Plausible outputs may differ in brightness, color, and other visual properties.
  • Diffusion models: Diffusion models generate SR images that human raters perceive as more realistic than outputs from other generative models such as GANs.The survey presents this as a key reason for their growing importance in SR.
  • Survey scope: The survey bridges foundational diffusion concepts with current image-SR research while analyzing strengths, weaknesses, and coherent research directions.It is motivated by the rapidly expanding publication volume and the difficulty of entering the field.
  • Survey scope: Its coverage includes diffusion foundations, conditioning and zero-shot approaches, alternative domains, corruption spaces, applications, and relationships with other generative SR models.The survey distinguishes its detailed SR focus from broader reviews of SR or image restoration.
  • SR foundations: The ill-posed inverse problem requires SR models to hallucinate realistic high-frequency details beyond interpolation, especially at larger magnifications.Traditional regression methods may struggle with these details at magnifications such as s > 4.
  • SR models: Standard SR approaches include traditional, regression-based deep-learning, GAN, and flow-based methods, each with documented drawbacks such as artifacts or computational cost.These methods rely on image statistics, end-to-end mappings, adversarial training, or invertible flow architectures.

C. Image Quality Assessment (IQA)

Image quality assessment in SR combines reference-based metrics, perceptual measures, subjective ratings, consistency, and no-reference methods. The survey emphasizes that pixel-based scores do not always reflect human judgments of realism.

  • IQA overview: Image quality assessment evaluates SR outputs through properties such as sharpness, contrast, noise, and perceived realism.The section frames IQA as a non-trivial task because image quality is multifaceted.
  • Reference-based metrics: PSNR compares the maximum pixel value with mean squared error between the SR image and its HR reference.It is widely used for SISR reconstruction quality but focuses on pixel differences.
  • Reference-based metrics: PSNR does not accurately match human perception because small pixel shifts can worsen its value without changing perceived quality.Pixel-based training can produce high PSNR, whereas generative models often obtain lower PSNR values.
  • Reference-based metrics: SSIM combines luminance, contrast, and structure comparisons, with α, β, and γ controlling their relative importance.The comparison function includes a stability constant c to support numerical computation.
  • Other metrics: MOS averages human quality ratings, while consistency measures output stability and no-reference metrics assess quality without a ground-truth image.No-reference approaches compare statistical features with high-quality natural-image collections or use models such as CLIP.
  • Perceptual metrics: LPIPS projects SR and HR images into learned CNN feature spaces and compares corresponding feature patches for a more human-centric assessment.The survey describes it as better aligned with human perception than PSNR and SSIM.

III. DIFFUSION MODELS BASICS

Diffusion Models iteratively add noise to move data into a corruption space, then learn reverse denoising to recover image-space distributions. DDPMs use finite discrete Markov chains, with conditional reverse transitions and variational optimization.

  • Core principle: Forward diffusion increases t by progressively degrading data with noise, while backward diffusion decreases t through learned iterative denoising.The reverse model pθ approximates the ideal denoised transition distribution.
  • Core principle: DDPMs, SGMs, and SDEs define different implementations of the forward process and its learned reverse approximation.DDPMs and SGMs are discrete formulations, whereas SDEs provide a continuous formulation.
  • Denoising Diffusion Probabilistic Models: DDPMs use two Markov chains across a finite number of discrete time steps.The forward chain transforms data toward a manually designed prior, while the reverse chain learns the inverse process.
  • Denoising Diffusion Probabilistic Models: Conditional DDPM reverse transitions incorporate an LR image through learnable mean and covariance functions.The conditional transition uses µθ(zt, x, γt) and Σθ(zt, x, γt).
  • Denoising Diffusion Probabilistic Models: DDPM optimization minimizes a variational objective derived from the joint forward and reverse sequences.The constant c does not affect optimization, and Equation 18 without c is the VLB of the data log-likelihood.

B. Score-based Generative Models (SGMs)

Score-based Generative Models represent distributions through score functions rather than directly through probability densities. They estimate noisy intermediate states and reverse diffusion using a trained score network, commonly with Annealed Langevin Dynamics.

  • Score formulation: SGMs use the score function ∇z log p(z), which preserves density information while offering a computationally convenient representation.Their training and sampling procedures are decoupled, allowing flexibility in defining sampling methods.
  • Forward diffusion: SGM forward diffusion creates a sequence of increasingly noisy densities using noise levels σ1 < ... < σT.The intermediate state zt = z0 + σt·ϵ can be sampled in one step independently of previous time steps.
  • Backward diffusion: A Noise-Conditional Score Network estimates ∇zt log q(zt) at each time step for reverse sampling.The network sθ(zt, t) supplies the gradient approximation used to generate intermediate states.
  • Backward diffusion: Unlike DDPMs, SGMs iteratively sample zt during reverse generation rather than directly predicting zt from zt+1.Annealed Langevin Dynamics uses N estimation iterations and step size αt for each time step.
  • Sampling and conditioning: The Annealed Langevin Dynamics update converges to q(z0) as αt approaches 0 and N approaches infinity.SGMs can also be conditioned on an LR image by estimating the conditional score ∇zt log q(zt|x).
  • Optimization: Denoising score matching estimates the score by weighting the discrepancy between the true noisy-data score and the model score.The weighting function λ(t) is positive, and alternative score-matching objectives are also possible.

C. Stochastic Differential Equations (SDEs)

SDEs generalize diffusion to continuous time and provide a unified formulation for DDPMs and SGMs. Their reverse-time SDE and probability-flow ODE support sampling from the same data distribution.

  • Continuous-time formulation: SDEs describe diffusion over infinitely many continuous time steps and can be viewed as continuous-time generalizations of DDPMs and SGMs.The framework uses stochastic-process mathematics to represent diffusion processes in DMs.
  • Forward diffusion: The forward SDE uses drift and diffusion functions to represent how data is progressively corrupted.The drift function is f, the diffusion function is g, and w is a standard Wiener process.
  • Backward diffusion: The reverse-time SDE converts noise toward data through backward-time stochastic evolution.Its formulation uses a backward Wiener process and an infinitesimal negative time step.
  • Sampling: The probability-flow ODE has the same marginals as the reverse-time SDE, so both enable sampling from the same data distribution.This provides a deterministic ODE counterpart to the stochastic reverse process.
  • Optimization: Continuous-time score optimization extends the SGM objective by modeling sθ(zt, t) as an approximation to ∇z log qt(z).The continuous-time objective uses a positive weighting function λ(t).

D. Relation between Diffusion Models

DDPMs and SGMs have mathematically connected optimization objectives. The connection appears when the DDPM noise-prediction network is identified with the SGM score network through a scale-dependent substitution.

  • Optimization objectives: Reweighting the DDPM variational lower bound yields an objective that can be compared directly with the SGM optimization target.The weighting function λ(t) is positive.
  • Optimization objectives: Setting ϵθ(zt, t) = −σt sθ(zt, t) establishes the mathematical connection between DDPM noise prediction and SGM score estimation.The constant c is irrelevant to optimization, so the resulting objectives are equivalent in the stated formulation.

E. Relation to other Image SR Generative Models

Image SR generative models differ in architecture and training objectives, trading advantages such as perceptual quality against challenges including stability and computational cost. The survey emphasizes diffusion-model improvements particularly relevant to SR, especially efficient sampling and likelihood estimation.

  • Generative SR models differ primarily in how their architectures and training objectives generate high-resolution images from low-resolution inputs.
  • GANs use adversarial generator–discriminator training, while diffusion models iteratively add and remove noise without a discriminator.
  • VAEs use a variational latent space for ill-posed SR, whereas diffusion models can be viewed as using a fixed perturbation encoder and decoder-like reverse process.
  • Autoregressive models generate pixels sequentially, making high-resolution image generation computationally expensive compared with diffusion’s gradual forward and reverse processes.
  • Efficient sampling reduces diffusion inference steps, while likelihood improvement targets applications including compression, semi-supervised learning, and image SR.
  • A DDPM requires about 20 hours to sample 50,000 32x32 images and nearly 1,000 hours for 256x256 images on an Nvidia 2080 Ti GPU.

V. DIFFUSION MODELS FOR IMAGE SR

The survey organizes diffusion-based image SR around conditioning, guidance, state representation, and corruption space. It presents DDPMs as the dominant practical formulation and reviews representative realizations, guidance mechanisms, and their trade-offs.

  • The survey’s image-SR diffusion framework covers DDPM realizations, guidance, alternative state domains, conditioning methods, corruption spaces, color shifting, and architectural designs.
  • DDPMs are favored over SGMs in image SR because their straightforward implementation lowers the entry barrier and supports quicker development and replication.
  • The framework distinguishes pixel, latent, frequency, and residual state representations from the target or starting point defined by the corruption space.
  • SR3 adds Gaussian noise to the LR-conditioned target and iteratively denoises it, with a model predicting noise from the LR image, noisy target, and noise variance.
  • SRDiff differs from SR3 by predicting residual information between LR and HR images rather than the HR image directly.
  • Guidance strengthens conditioning during backward diffusion but trades sample diversity for greater conditioning influence.
  • Classifier guidance requires a classifier that handles arbitrarily noisy inputs, a capability most existing pretrained image-classification models lack.
  • Classifier-free guidance can prioritize conditioning when λ > 1, but standard training may require two separate diffusion models and therefore higher computational cost.

C. State Domains

Alternative state domains reshape where diffusion operates and can target computational efficiency or detail modeling. The survey covers latent, frequency, residual, and conditioning-oriented approaches, including two-stage VAE–DM designs and text-enhanced conditioning.

  • Alternative-domain methods map pixel inputs into latent, frequency, or residual spaces, adding a domain-conversion step to the diffusion pipeline.
  • Latent Space: Latent diffusion moves computation into a low-dimensional autoencoder space, reducing resource requirements while maintaining performance through decoupled pretrained components.
  • Frequency Space: Wavelet conversion is lossless, reduces spatial size by a factor of four, and separates high-frequency details into distinct channels.
  • Residual Space: Residual-space diffusion focuses generation on differences between upsampled LR and HR images, which can speed convergence and stabilize training.
  • Conditioning: DiffuseVAE uses a two-stage generator–refiner framework in which a VAE produces varying, often blurred reconstructions that condition a diffusion model.
  • Conditioning: Low-resolution references, super-resolved references, and pretrained feature representations provide alternative conditioning signals for backward diffusion.
  • Evaluation: The survey reports that generative approaches generally produce lower PSNR and SSIM than regression-based methods, while consistency can be assessed separately.
  • Conditioning: Text-to-image conditioning enables pretrained models and textual descriptions to contribute guidance, as illustrated by StableSR and PASD.

E. Corruption Space

Diffusion models can modify corruption beyond Gaussian noise by using alternative state domains and degradation processes. These choices affect computational efficiency, interpretability, training stability, and color fidelity.

  • Alternative corruption processes: I2SB uses the clean image as the initial state and the degraded image as the final state, creating a traceable path for restoration.This design enhances interpretability and can require fewer backward-diffusion steps than standard corruption.
  • Alternative corruption processes: Cold Diffusion shows that effective generative diffusion can use degradations other than Gaussian noise, including iterative face-to-animal animorphosis.I2SB similarly modifies the corruption process without imposing assumptions on underlying prior distributions.
  • Color shifting: Color shifting can occur when computational limits force smaller batch sizes or shorter training, as illustrated by vanilla SR3.StableSR addresses this with color normalization, while YODA uses time-dependent masks to diffuse important image areas more frequently.
  • Denoising-network design: Denoising networks commonly use U-Nets, but designs also include residual blocks, wavelet-domain models, latent autoencoders, and Vision Transformers.The survey presents no single optimal denoising-network design and notes that architectures often draw from broader computer-vision advances.

VI. DIFFUSION-BASED ZERO-SHOT SR

Diffusion-based zero-shot SR uses pretrained diffusion models without paired training examples, enforcing consistency through projection or decomposition-based constructions. These methods extend beyond SR, but decomposition approaches introduce computational and modeling constraints.

  • Zero-shot SR: Zero-shot SR avoids prior image examples or task-specific training by exploiting image redundancy and conditioning pretrained diffusion models during sampling.This distinguishes zero-shot approaches from methods that train directly on paired examples or use conditioning only through standard model inputs.
  • Projection-based methods: Projection-based methods incorporate LR structures or textures at each step to complement generated images and maintain data consistency.ILVR projects LR low-frequency information onto the HR image, while CCDF modifies unified projection for SR.
  • Decomposition-based methods: Decomposition-based methods formulate SR as a linear reverse problem and use range-null space decomposition to construct general solutions.DDNM applies this framework across SR, colorization, inpainting, and deblurring.
  • Decomposition-based methods: DDNM rectifies intermediate clean estimates by preserving null-space content while replacing range-space content with a pseudo-inverse reconstruction.The resulting estimate is used to sample the preceding diffusion state.
  • Limitations: DDNM adds computational expense, requires manually specifying the degradation operator, and assumes that operator is linear.These constraints can make the approach difficult to apply to some restoration tasks and may limit effectiveness when degradation is not linear.

C. Posterior Estimation

Posterior-estimation methods address weaknesses of projection-based inverse-problem solvers by modeling the conditional distribution of clean images given noisy diffusion states. They use score-based identities and approximations to enforce data consistency probabilistically.

  • Motivation: Projection can weaken data consistency by moving the sampling path away from the data manifold, motivating posterior estimation with Bayes’ theorem.The Bayesian formulation seeks a more robust and probabilistic inverse-problem framework.
  • Posterior score: The posterior score combines the likelihood gradient ∇zt log pt(x|zt) with the pretrained diffusion score sθ(x,t).Because the likelihood term is intractable, posterior-estimation methods approximate it using available clean-image estimates or degradation models.
  • Posterior approximations: MCG and DPS approximate pt(x|zt) using the clean estimate ˆz0(zt), while DPS removes the projection step to improve consistency.IIGDM extends this formulation to linear, nonlinear, and differentiable inverse problems with a Moore–Penrose pseudoinverse.
  • Heuristic guidance: GDP heuristically links higher conditional probability to smaller degradation distance, using distance and quality metrics to guide sampling.It replaces noisy zt with its clean estimate ˆz0 when calculating distance to address mismatched noise levels.

VII. DOMAIN-SPECIFIC APPLICATIONS

Domain-specific SR applications adapt diffusion models to medical imaging, unknown face degradations, atmospheric turbulence, and satellite imagery. These methods modify conditioning, initialization, architecture, or knowledge transfer to address domain-specific distortions and reconstruction needs.

  • Overview: SR3 produces strong perceptual results for faces and natural images but may be unsuitable for domains such as remote sensing.The survey therefore examines diffusion-based SR methods designed for specialized applications.
  • Medical imaging: R2D2+ combines MRI denoising with super-resolution and begins backward diffusion from an initially noisy image rather than pure Gaussian noise.Its design is inspired by the zero-shot CCDF approach.
  • Blind face restoration: Blind face restoration addresses the mismatch between fixed training degradations and unknown real-world distortions, compounded by limited paired clean and degraded datasets.DifFace uses pretrained diffusion priors, while DiffBFR separates identity restoration from texture enhancement.
  • Blind face restoration: DR2E uses degradation removal followed by enhancement, diffusing severely degraded faces and guiding reverse diffusion with their low-frequency information.The method targets severe and unknown degradations in face images.
  • Atmospheric turbulence: AT-VarDiff corrects atmospheric-turbulence distortions, while knowledge distillation transfers clean-face class priors to realistic face restoration under substantial degradation.The resulting process starts from noise- and turbulence-degraded images so outputs remain close to the observed distortions.
  • Remote sensing: DMDC addresses remote-sensing SR by supplementing missing high-frequency detail and texture needed for satellite-image reconstruction.The application is motivated by the scarcity of small, complex targets in available HR imagery.

VIII. DISCUSSION AND FUTURE WORK

Future work for diffusion-based image super-resolution centers on computational efficiency, principled sampling, corruption-space design, and fairer evaluation. The survey also highlights unresolved color-shift mechanisms and the need to preserve high-frequency information while reducing computational burden.

  • Architectural and theoretical directions: Color shifts can emerge under hardware-constrained training, while cascaded generation and wavelet-based designs offer distinct routes toward efficiency and information preservation.The survey notes that theoretical understanding or normalization may mitigate color shifts, and that cascaded models use more refinement at lower resolutions and fewer steps at higher resolutions.
  • Computational costs: Diffusion models impose substantial computational and memory demands, creating practical barriers and widening the gap between academic and industrial research.The survey identifies smaller spatial domains, latent diffusion, and wavelet-based models as possible responses, while noting unresolved precision concerns for latent methods.
  • Efficient sampling: Reducing inference steps is promising, but systematic methods for selecting inference schedules remain underdeveloped.The survey discusses training-based sampling, efficient diffusion models, alternative noise schedulers, knowledge distillation, and truncated diffusion as research directions.
  • Corruption spaces: Alternative corruption spaces can map low-resolution images toward high-resolution outputs more directly while addressing assumptions such as diversity and blurriness in existing diffusion frameworks.The survey identifies methods including InDI and I2SB as examples motivating further study of diverse and complex corruption patterns.
  • Evaluation: Comparisons across diffusion super-resolution methods are difficult because datasets differ in resolution, content diversity, color distribution, and noise levels.The survey calls for representative benchmarks with uniform evaluation metrics to improve comparability.
Loading 2401.00736v3…