Source-linked AI summary
Image Super-Resolution via Iterative Refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, Mohammad Norouzi
TL;DR
Large-magnification super-resolution must produce detailed images consistent with ambiguous low-resolution inputs, a setting where simple regression can fall short. SR3 adapts denoising diffusion models to conditional generation through iterative U-Net refinement and cascading. It works across faces and natural images, achieves a human fool rate close to 50% on 8× face super-resolution, and supports efficient cascaded high-resolution synthesis.
Problem
Large-magnification super-resolution is difficult because multiple high-resolution images can match one low-resolution input, while simple regression often lacks high-fidelity detail.
Method
SR3 adapts denoising diffusion models to conditional generation, using a U-Net denoising objective for iterative refinement and cascaded models for high-resolution synthesis.
Results
SR3 achieves a human fool rate close to 50% on 8× face super-resolution and is effective across face and natural-image tasks at different magnification factors.
Takeaways & Limitations
Cascaded SR3 models enable high-resolution synthesis more efficiently than direct high-resolution generation while maintaining image quality.
Takeaways & Limitations
SR3 exhibits bias issues including possible mode dropping and overly continuous skin textures that remove reference details, so the authors advise against real-world use until these biases are understood and mitigated.
Abstract
from arXiv · showhide
We present SR3, an approach to image Super-Resolution via Repeated Refinement. SR3 adapts denoising diffusion probabilistic models to conditional image generation and performs super-resolution through a stochastic denoising process. Inference starts with pure Gaussian noise and iteratively refines the noisy output using a U-Net model trained on denoising at various noise levels. SR3 exhibits strong performance on super-resolution tasks at different magnification factors, on faces and natural images. We conduct human evaluation on a standard 8X face super-resolution task on CelebA-HQ, comparing with SOTA GAN methods. SR3 achieves a fool rate close to 50%, suggesting photo-realistic outputs, while GANs do not exceed a fool rate of 34%. We further show the effectiveness of SR3 in cascaded image generation, where generative models are chained with super-resolution models, yielding a competitive FID score of 11.3 on ImageNet.
1. Introduction
SR3 adapts denoising diffusion models to conditional image generation for super-resolution, addressing the difficulty of producing high-fidelity details at large magnification factors. It performs well across faces and natural images, with human evaluation favoring its outputs over GAN baselines.
- Motivation: Single-image super-resolution must generate a high-resolution image consistent with a low-resolution input despite multiple plausible outputs and complex conditional distributions.These ambiguities make simple regression methods inadequate for preserving high-fidelity details at large magnification ratios.
- Approach: SR3 adapts denoising diffusion models to conditional generation using a U-Net trained to iteratively remove noise at different levels.The model transforms a standard normal distribution into an empirical data distribution through repeated refinement.
- Approach: SR3 uses a constant number of inference steps regardless of output resolution, unlike autoregressive models.
- Results: SR3 works across different magnification factors and input resolutions, including both face and natural-image super-resolution.The models can also be cascaded for progressively higher-resolution generation.
- Results: A human fool rate close to 50% on standard 8× face super-resolution exceeds the at-most-34% rates reported for FSRGAN and PULSE.The evaluation uses human judgments because automated scores such as PSNR and SSIM do not reliably reflect preference in this setting.
2. Conditional Denoising Diffusion Model
SR3 adapts conditional DDPMs to learn a stochastic, iterative refinement process for super-resolution. It starts from Gaussian noise and uses a noise-conditioned denoising model to reconstruct targets consistent with source images.
- The task models super-resolution as learning a stochastic approximation to the one-to-many conditional distribution p(y | x).Multiple target images can be consistent with one source image.
- SR3 reverses a forward Gaussian diffusion process through learned conditional transitions pθ(y_t−1 | y_t, x).Inference begins with y_T sampled from N(0, I) and iteratively produces y_0.
- The forward process adds Gaussian noise with q(y_t | y_t−1) = N(y_t | √α_t y_t−1, (1 − α_t)I), while α_t controls the per-step noise variance.The attenuation by √α_t keeps the variance bounded across iterations.
- The denoising model fθ receives a source image, a noisy target, and a noise-level statistic γ, and is trained to predict the noise vector ϵ.The model can instead regress toward y_0; changing the target changes the loss scale and requires adjusting p(γ).
- Inference repeatedly estimates the clean target, parameterizes reverse transitions, and samples through a reverse Markov chain resembling Langevin dynamics.The model uses Gaussian conditional transitions and a variance tied to the forward process.
- SR3 conditions a DDPM-like U-Net by concatenating a bicubically upsampled source image with the noisy target and supports flexible inference schedules through direct γ conditioning.Training uses a piecewise γ distribution with T = 2000, while direct γ conditioning permits changing the number of inference steps.
3. Related Work
Prior super-resolution methods trade off distributional fidelity, expressiveness, sampling speed, or optimization difficulty. SR3 is positioned as a diffusion-based alternative that generates high-resolution images through repeated reverse-diffusion steps.
- Autoregressive models capture rich distributions but sequential pixel generation is expensive, limiting their use to low-resolution images.
- Normalizing flows improve sampling speed and model exact likelihoods, but invertible transformations with tractable Jacobians limit expressiveness.
- VAEs offer fast sampling but tend to underperform GANs and autoregressive models in image quality.
- Regression-based super-resolution methods trained with MSE estimate posterior means, producing blurry outputs when the posterior is multi-modal.
- SR3 generates images such as 1024×1024 outputs with a constant number of refinement steps, often no more than 100.
- GAN-based methods can produce high-quality face super-resolution but are often difficult to optimize and may require auxiliary consistency objectives.
4. Experiments
Experiments evaluate SR3 on face and natural-image super-resolution using qualitative comparisons, automated metrics, human 2AFC tests, and cascaded generation. Across these settings, SR3 produces detailed outputs, strong consistency, competitive perceptual results, and effective cascaded synthesis.
- Qualitative Results: SR3 produces sharper, more detailed natural-image outputs than the regression baseline, which remains faithful but blurry.The comparison is shown for 64×64→256×256 ImageNet images, including enlarged patches for finer inspection.
- Automated metrics: On low-resolution, high-magnification tasks, PSNR and SSIM favor conservative regression outputs even when SR3 achieves higher sample-quality scores.For ImageNet 64×64→256×256 super-resolution, SR3 has higher FID and IS but worse PSNR and SSIM than regression.
- Automated metrics: SR3 achieves the best consistency error among the compared face super-resolution methods, slightly outperforming the regression baseline and substantially beating PULSE and FSRGAN.Consistency is measured by MSE between down-sampled super-resolution outputs and the low-resolution inputs.
- Human Evaluation (2AFC): In face human evaluation, SR3’s fool rate is close to 50%, while FSRGAN and PULSE reach at most 34%.The fool rate is based on subjects selecting a model output over ground truth in 2AFC experiments.
5. Discussion and Conclusion
SR3 achieves photo-realistic super-resolution while supporting cascaded generation, but the authors report unresolved bias and mode-dropping concerns that limit real-world use.
- Discussion and Conclusion: SR3 suffers from bias issues, including possible mode dropping and overly continuous skin texture that omits reference moles, pimples, and piercings.
- Discussion and Conclusion: The authors state that SR3 should not be used for real-world super-resolution until these biases are thoroughly understood and mitigated.
- Discussion and Conclusion: SR3 can generate high-resolution super-resolution images and unconditional samples when cascaded with an unconditional model.The appendix describes additional cascaded examples and the conclusion reports both uses.
- Discussion and Conclusion: SR3 demonstrates face and natural-image super-resolution at high resolution and high magnification ratios, including 64×64→256×256 and 256×256→1024×1024.
- Discussion and Conclusion: SR3 achieves a human fool rate close to 50%, suggesting photo-realistic outputs.
A. Task Specific Architectural Details
The SR3 super-resolution architecture conditions a diffusion U-Net by concatenating an upsampled low-resolution image with the noisy high-resolution input, using task-specific hyperparameters.
- A. Task Specific Architectural Details: For each super-resolution task, SR3 and Regression models use the same U-Net architecture.
- A. Task Specific Architectural Details: The low-resolution input is interpolated to the target resolution and concatenated with the noisy high-resolution image along the channel dimension.Figure A.1 illustrates this conditioning for 16×16→128×128 super-resolution.
- A. Task Specific Architectural Details: Table A.1 summarizes task-specific U-Net hyperparameters, including the first-layer channel dimension and depth multipliers for subsequent resolutions.
B. Justification of the Training Objective
The training objective is justified through a variational lower bound and connected to denoising score matching, with the simplified loss corresponding to an L2 regression objective.
- B. Justification of the Training Objective: The training objective is justified as a variational lower bound on the marginal log-likelihood of the probabilistic diffusion model.
- B. Justification of the Training Objective: The negative variational lower bound becomes a simplified loss up to constant weighting of terms at each diffusion time step.
- B. Justification of the Training Objective: The simplified objective corresponds to the L2 norm in Equation (6), with p(γ) characterized by a uniform distribution over {γ1, . . . , γT}.
- B. Justification of the Training Objective: SR3 is presented as a denoising score-matching variant whose target density mixes Gaussian perturbations q(ỹ|y0, γ) across data points and noise levels.
- B. Justification of the Training Objective: The resulting gradient of the data log-density is used as the model’s regression target.
C. Additional Experimental Results
The appendix provides additional SR3 results for faces, natural images, comparisons, and cascaded generation across multiple resolutions and datasets.
- C. Additional Experimental Results: The appendix presents further examples, cascaded 1024×1024 face generation, unconditional face samples, and class-conditional 256×256 ImageNet samples.
- C. Additional Experimental Results: Additional FFHQ results apply 64×64→512×512 SR3 models to faces outside the training set after cropping and alignment.
- C. Additional Experimental Results: Additional ImageNet results evaluate SR3 models trained on ImageNet for 64×64→256×256 natural-image super-resolution.
- C. Additional Experimental Results: The appendix includes comparisons among Bicubic, FSRGAN, PULSE, Regression, and SR3 on the 16×16→128×128 face super-resolution task.
- C. Additional Experimental Results: Cascaded face generation chains an unconditional model with two SR3 models to produce 1024×1024 images through 64×64→256×256→1024×1024 refinement.
- C. Additional Experimental Results: Cascaded ImageNet generation samples a random label, generates a 64×64 class-conditional image, and applies 4× SR3 to obtain 256×256 images.
C.1. Failure Cases of SR3
SR3 performs well on natural and aligned face images but has identifiable failure cases involving complex fine structure, text, and ambiguous details.
- Complex facial detail: SR3 struggles with complex, regular hair patterns such as finely braided hair.The authors suggest limited exposure to such structures in the relatively small FFHQ training dataset as one possible reason.
- Complex facial detail: SR3 can struggle with long-range correlations in fine details, including consistent eye highlights.
- Complex facial detail: Frameless eyeglasses are difficult to generate when their structure is hard to discern from the low-resolution input.
- Text generation: SR3 fails to generate natural-looking text in certain ImageNet images because it has not learned common alphabets.Its generated characters may be sharper than a regression baseline but lack meaningful structure, making them easier to distinguish from real images.
- Ambiguous fine detail: When low-resolution inputs omit fine details, SR3 may infer incorrect or overly uniform structure, including building-side textures.The output can appear sharper while still differing from the high-resolution original.
D. Images with the Lowest and Highest Fool Rates
Inspecting images with extreme fool rates reveals both visible distortions in weak outputs and cases where SR3 frequently fools human evaluators.
- Examining images that maximize or minimize fool rates provides insight into model problems and outputs that regularly fool people.
- PULSE images with the lowest fool rates show obvious distortions, with fool rates below 10% in both evaluation tasks.
- SR3’s best fool rates reach 84% and 88% in the examples that most successfully fool human subjects.The corresponding original images are somewhat noisy, which leads many subjects to prefer the SR3 outputs.
Task-1: Human Evaluation given low-resolution inputs
Task-1 compares model outputs with reference high-resolution images while showing the low-resolution input, and reports a per-image Mean Fool Rate.
- Task-1 compares each algorithm’s output with reference high-resolution images in the presence of low-resolution inputs.
- The figure omits reference images for privacy reasons and instead displays the corresponding PULSE and SR3 outputs for each input image.
- Mean Fool Rate is reported directly below each input image’s corresponding outputs.
Task-2: Human Evaluation without low-resolution inputs
Task-2 compares model outputs with reference high-resolution images without providing low-resolution inputs, and reports a per-image Mean Fool Rate.
- Task-2 compares each algorithm’s output with reference high-resolution images in the absence of low-resolution inputs.
- For privacy reasons, the figure omits reference images and displays the corresponding PULSE and SR3 outputs for each input image.
- Mean Fool Rate is reported below each image’s corresponding outputs.