Source-linked AI summary
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models
Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, Yueting Chen
TL;DR
SISR must recover an HR image from an LR input even though multiple HR images may correspond to the same observation, and prior methods face smoothing, collapse, or footprint limitations. SRDiff introduces conditional diffusion with residual prediction, transforming Gaussian noise into SR images through a Markov chain. On CelebA and DIV2K, it generates diverse results from one LR input, uses a small footprint, and supports latent-space manipulation.
Problem
SISR is ill-posed because one LR image can correspond to multiple HR images, while prior PSNR-oriented, GAN-driven, and flow-based methods face over-smoothing, mode collapse, or large-footprint issues.
Method
SRDiff uses an LR-conditioned diffusion model that iteratively denoises Gaussian noise into an SR image and applies residual prediction to accelerate convergence and stabilize training.
Results
Experiments on CelebA and DIV2K show diverse SR results from one LR input, state-of-the-art performance, a small training footprint, and latent-space manipulation.
Takeaways & Limitations
SRDiff provides diverse and realistic super-resolution outputs while remaining stable and efficient to train and enabling flexible image manipulation.
Takeaways & Limitations
The authors identify improving diffusion-based SISR performance and speeding up inference as future work.
Abstract
from arXiv · showhide
Single image super-resolution (SISR) aims to reconstruct high-resolution (HR) images from the given low-resolution (LR) ones, which is an ill-posed problem because one LR image corresponds to multiple HR images. Recently, learning-based SISR methods have greatly outperformed traditional ones, while suffering from over-smoothing, mode collapse or large model footprint issues for PSNR-oriented, GAN-driven and flow-based methods respectively. To solve these problems, we propose a novel single image super-resolution diffusion probabilistic model (SRDiff), which is the first diffusion-based model for SISR. SRDiff is optimized with a variant of the variational bound on the data likelihood and can provide diverse and realistic SR predictions by gradually transforming the Gaussian noise into a super-resolution (SR) image conditioned on an LR input through a Markov chain. In addition, we introduce residual prediction to the whole framework to speed up convergence. Our extensive experiments on facial and general benchmarks (CelebA and DIV2K datasets) show that 1) SRDiff can generate diverse SR results in rich details with state-of-the-art performance, given only one LR input; 2) SRDiff is easy to train with a small footprint; and 3) SRDiff can perform flexible image manipulation including latent space interpolation and content fusion.
1 Introduction
SISR is ill-posed because one LR image can correspond to multiple HR images, while existing approaches trade realism, diversity, or efficiency. SRDiff addresses these issues with conditional diffusion and residual prediction, producing diverse outputs with efficient training and flexible manipulation.
- Problem: SISR recovers HR images from LR inputs, but the inverse problem is ill-posed because multiple HR images can yield one LR image.This ambiguity motivates methods that can represent multiple plausible reconstructions.
- Prior limitations: PSNR-oriented methods can average plausible reconstructions into over-smoothed images, while flow-based methods require large footprints and costly training.The supplied passages also identify mode collapse as a problem associated with GAN-driven methods.
- SRDiff: SRDiff conditions a Markov-chain reverse process on LR features to transform Gaussian noise into SR predictions, using residual prediction to focus on high-frequency details.The residual is the difference between the HR image and the upsampled LR image.
- SRDiff: SRDiff generates diverse and high-quality SR results without mode collapse by converting Gaussian white noise into predictions through a Markov chain.The method is presented as the first diffusion-based model for super-resolution.
- Efficiency: SRDiff uses a single loss and no discriminator or bijective architectural constraint, supporting stable training with a small footprint and fast training.These design choices are contrasted with GAN-driven and flow-based methods.
- Results: Experiments on CelebA and DIV2K report multiple SR results from one LR input, one-quarter the parameters of SRFlow, and latent-space manipulation for more diverse outputs.The reported training time is about 30 hours on one GPU until convergence.
2 Related Works
Related work organizes SISR methods into PSNR-oriented, GAN-driven, and flow-based approaches, while diffusion models address one-to-many generation but had not yet been used for super-resolution.
- SISR methods: Deep learning SISR methods are categorized as PSNR-oriented, GAN-driven, and flow-based approaches.This taxonomy frames the comparison space for SRDiff.
- Diffusion models: Diffusion probabilistic models use Markov chains to transform simple-distribution latent variables into complex data distributions.The passages describe their use in speech and image synthesis.
- Diffusion models: Diffusion models are useful for one-to-many problems and high-quality synthesis, but had not yet been applied to image reconstruction tasks such as super-resolution.SRDiff is introduced to fill this application gap.
3 Diffusion Model
Diffusion models use a fixed forward Markov chain to add Gaussian noise and a learned reverse Markov chain to recover data from a simple latent distribution. Training uses variational inference with Gaussian transitions and an ELBO-based objective.
- Diffusion process: A diffusion model gradually transforms data x0 into a simple latent variable xT by adding Gaussian noise through a fixed Markov chain.The total number of diffusion steps is T, and each xt has the same dimension as x0.
- Diffusion process: The forward process permits sampling xt directly at any timestep using a closed-form expression parameterized by the cumulative noise schedule.The schedule uses αt := 1 − βt and ᾱt as the cumulative product of αs.
- Reverse process: The reverse process learns Gaussian transitions that transform xT back toward the data distribution, beginning from p(xT) = N(xT; 0, I).Its transitions are parameterized by θ through learned means and variances.
- Training objective: Training maximizes a variational lower bound on negative log likelihood, with KL-divergence terms comparing the reverse transitions against corresponding diffusion posteriors.The formulation is arranged so the KL divergences compare Gaussian distributions.
- Training objective: For simplicity, the training procedure minimizes a variational-bound variant using x0 and timestep t as inputs, with εθ serving as a noise predictor.The model predicts the noise added during diffusion.
4 SRDiff
SRDiff conditions a T-step diffusion model on encoded LR information and predicts the residual between HR images and upsampled LR inputs. Iterative denoising reconstructs SR images, while the architecture and inference procedure are designed for efficient, stable generation.
- SRDiff framework: SRDiff predicts the residual between the HR image and the upsampled LR image instead of predicting the HR image directly.The generated residual is added to the upsampled LR image to reconstruct the SR image.
- SRDiff framework: A conditional noise predictor iteratively denoises a Gaussian latent into the residual image while conditioning on hidden LR features from an RRDB-based encoder.The LR encoder output guides every reverse step toward the corresponding HR space.
- Architecture: The conditional noise predictor uses a U-Net body that combines the noisy residual, diffusion timestep, and LR-encoder output.Its contracting and expansive paths use residual blocks with downsampling and upsampling layers.
- Training: Training uses LR-HR pairs, residual targets, encoded LR images, sampled Gaussian noise, and randomly selected diffusion timesteps to optimize noise prediction.The implementation uses an L1 objective for better training stability instead of the original L2 objective.
- Inference: During inference, SRDiff samples a Gaussian latent, encodes the LR image once, and performs reverse iterations from t = T down to 1.The final SR prediction is obtained by adding the generated residual to the upsampled LR input.
5 Experiments
Experiments evaluate SRDiff on face and general super-resolution benchmarks, compare it with existing methods, analyze design choices, and demonstrate image-manipulation applications.
- Experimental Settings: SRDiff is evaluated on face SR at 8× using CelebA and general SR at 4× using DIV2K.The face benchmark uses 162,770 training images and 5,000 test images; general SR uses DIV2K and Flickr2K for training and 100 DIV2K validation images.
- Performance: On CelebA face SR, SRDiff outperforms previous methods on most evaluation metrics while producing diverse, detailed, LR-consistent portraits.Compared with PSNR-oriented methods it reconstructs clearer textures, while compared with GAN-driven methods it avoids artifacts and produces more natural results.
- Performance: SRDiff uses 12M parameters and takes about 30 hours to converge, compared with SRFlow's 40M parameters and five-day training time.The paper attributes the smaller footprint to the absence of architectural constraints for guaranteeing bijection and notes that no discriminator is needed during training.
- Performance: On DIV2K general SR, SRDiff achieves better quantitative results on most PSNR, SSIM, and LR-PSNR metrics, with comparable LPIPS.The reported visual comparison describes a balance between sharpness, naturalness, and consistency with the LR image.
- Ablation Study: Increasing diffusion steps or model width improves image quality but slows inference, while residual prediction improves quality and accelerates training.The default setting is T = 100 and c = 64 after trading off quality and inference speed.
- Extensions: SRDiff supports content fusion and latent-space interpolation for manipulating generated SR images.Content fusion combines eye and face contents, while latent interpolation produces new predictions by decoding linearly interpolated latents.
6 Conclusion
SRDiff uses a conditioned diffusion process with residual prediction to generate diverse, realistic super-resolution images while avoiding over-smoothing and mode collapse. Experiments show stable training with a small footprint and support for flexible image manipulation, while future work targets better performance and faster inference.
- SRDiff generates diverse and realistic super-resolution images by iteratively denoising latent variables with a noise predictor conditioned on encoded low-resolution information.Residual prediction is introduced to speed convergence and stabilize training.
- Experiments on face and general datasets show that SRDiff avoids over-smoothing and mode collapse issues associated with PSNR-oriented and GAN-driven methods.
- SRDiff is stable to train with a small footprint and without an extra discriminator.
- SRDiff supports flexible image manipulation, including latent space interpolation and content fusion.
- Future work will improve diffusion-based SISR performance, speed up inference, and extend the approach to other image restoration tasks.