Image Super-Resolution via Iterative Refinement

Motivation

• Single-image super-resolution must generate a high-resolution image consistent with a low-resolution input despite multiple plausible outputs and complex conditional distributions.These ambiguities make simple regression methods inadequate for preserving high-fidelity details at large magnification ratios.Source: §1 · p. 1

Approach

• SR3 adapts denoising diffusion models to conditional generation using a U-Net trained to iteratively remove noise at different levels.The model transforms a standard normal distribution into an empirical data distribution through repeated refinement.Source: §1 · p. 1
• SR3 uses a constant number of inference steps regardless of output resolution, unlike autoregressive models.Source: §1 · p. 2

Results

• SR3 works across different magnification factors and input resolutions, including both face and natural-image super-resolution.The models can also be cascaded for progressively higher-resolution generation.Source: §1 · p. 2
• A human fool rate close to 50% on standard 8× face super-resolution exceeds the at-most-34% rates reported for FSRGAN and PULSE.The evaluation uses human judgments because automated scores such as PSNR and SSIM do not reliably reflect preference in this setting.Source: §1 · p. 2

• The task models super-resolution as learning a stochastic approximation to the one-to-many conditional distribution p(y | x).Multiple target images can be consistent with one source image.Source: §2 · p. 2
• SR3 reverses a forward Gaussian diffusion process through learned conditional transitions pθ(y_t−1 | y_t, x).Inference begins with y_T sampled from N(0, I) and iteratively produces y_0.Source: §2 · p. 2, §2.3 · p. 3
• The forward process adds Gaussian noise with q(y_t | y_t−1) = N(y_t | √α_t y_t−1, (1 − α_t)I), while α_t controls the per-step noise variance.The attenuation by √α_t keeps the variance bounded across iterations.Source: §2.1 · p. 2
• The denoising model fθ receives a source image, a noisy target, and a noise-level statistic γ, and is trained to predict the noise vector ϵ.The model can instead regress toward y_0; changing the target changes the loss scale and requires adjusting p(γ).Source: §2.2 · p. 3
• Inference repeatedly estimates the clean target, parameterizes reverse transitions, and samples through a reverse Markov chain resembling Langevin dynamics.The model uses Gaussian conditional transitions and a variance tied to the forward process.Source: §2.3 · pp. 3–4
• SR3 conditions a DDPM-like U-Net by concatenating a bicubically upsampled source image with the noisy target and supports flexible inference schedules through direct γ conditioning.Training uses a piecewise γ distribution with T = 2000, while direct γ conditioning permits changing the number of inference steps.Source: §2.4 · p. 4

• Autoregressive models capture rich distributions but sequential pixel generation is expensive, limiting their use to low-resolution images.Source: §3 · p. 4
• Normalizing flows improve sampling speed and model exact likelihoods, but invertible transformations with tractable Jacobians limit expressiveness.Source: §3 · pp. 4–5
• VAEs offer fast sampling but tend to underperform GANs and autoregressive models in image quality.Source: §3 · p. 4
• Regression-based super-resolution methods trained with MSE estimate posterior means, producing blurry outputs when the posterior is multi-modal.Source: §3 · p. 4
• SR3 generates images such as 1024×1024 outputs with a constant number of refinement steps, often no more than 100.Source: §3 · p. 4
• GAN-based methods can produce high-quality face super-resolution but are often difficult to optimize and may require auxiliary consistency objectives.Source: §3 · p. 5

Qualitative Results

• SR3 produces sharper, more detailed natural-image outputs than the regression baseline, which remains faithful but blurry.The comparison is shown for 64×64→256×256 ImageNet images, including enlarged patches for finer inspection.Source: §4.1 · p. 6

Automated metrics

• On low-resolution, high-magnification tasks, PSNR and SSIM favor conservative regression outputs even when SR3 achieves higher sample-quality scores.For ImageNet 64×64→256×256 super-resolution, SR3 has higher FID and IS but worse PSNR and SSIM than regression.Source: §4.2.1 · p. 7
• SR3 achieves the best consistency error among the compared face super-resolution methods, slightly outperforming the regression baseline and substantially beating PULSE and FSRGAN.Consistency is measured by MSE between down-sampled super-resolution outputs and the low-resolution inputs.Source: §4.2.1 · p. 7

Human Evaluation (2AFC)

• In face human evaluation, SR3’s fool rate is close to 50%, while FSRGAN and PULSE reach at most 34%.The fool rate is based on subjects selecting a model output over ground truth in 2AFC experiments.Source: §4.2.2 · p. 8

Discussion and Conclusion

• SR3 suffers from bias issues, including possible mode dropping and overly continuous skin texture that omits reference moles, pimples, and piercings.Source: §5 · p. 10
• The authors state that SR3 should not be used for real-world super-resolution until these biases are thoroughly understood and mitigated.Source: §5 · p. 10
• SR3 can generate high-resolution super-resolution images and unconditional samples when cascaded with an unconditional model.The appendix describes additional cascaded examples and the conclusion reports both uses.Source: §5 · p. 10, Appendix · p. 13
• SR3 demonstrates face and natural-image super-resolution at high resolution and high magnification ratios, including 64×64→256×256 and 256×256→1024×1024.Source: §5 · p. 10
• SR3 achieves a human fool rate close to 50%, suggesting photo-realistic outputs.Source: §5 · p. 10

A. Task Specific Architectural Details

• For each super-resolution task, SR3 and Regression models use the same U-Net architecture.Source: A. Task Specific Architectural Details · p. 13
• The low-resolution input is interpolated to the target resolution and concatenated with the noisy high-resolution image along the channel dimension.Figure A.1 illustrates this conditioning for 16×16→128×128 super-resolution.Source: A. Task Specific Architectural Details · p. 13, Figure A.1 · p. 13
• Table A.1 summarizes task-specific U-Net hyperparameters, including the first-layer channel dimension and depth multipliers for subsequent resolutions.Source: Table A.1 · p. 13

B. Justification of the Training Objective

• The training objective is justified as a variational lower bound on the marginal log-likelihood of the probabilistic diffusion model.Source: §B.1 · p. 13
• The negative variational lower bound becomes a simplified loss up to constant weighting of terms at each diffusion time step.Source: §B.1 · p. 14
• The simplified objective corresponds to the L2 norm in Equation (6), with p(γ) characterized by a uniform distribution over {γ1, . . . , γT}.Source: §B.1 · p. 14
• SR3 is presented as a denoising score-matching variant whose target density mixes Gaussian perturbations q(ỹ|y0, γ) across data points and noise levels.Source: §B.2 · p. 14
• The resulting gradient of the data log-density is used as the model’s regression target.Source: §B.2 · p. 14

C. Additional Experimental Results

• The appendix presents further examples, cascaded 1024×1024 face generation, unconditional face samples, and class-conditional 256×256 ImageNet samples.Source: C. Additional Experimental Results · p. 14
• Additional FFHQ results apply 64×64→512×512 SR3 models to faces outside the training set after cropping and alignment.Source: Figure C.1 · p. 15, Figure C.2 · p. 16
• Additional ImageNet results evaluate SR3 models trained on ImageNet for 64×64→256×256 natural-image super-resolution.Source: Figure C.3 · p. 17, Figure C.4 · p. 18
• The appendix includes comparisons among Bicubic, FSRGAN, PULSE, Regression, and SR3 on the 16×16→128×128 face super-resolution task.Source: Figure C.5 · p. 19, C. Additional Experimental Results · p. 19
• Cascaded face generation chains an unconditional model with two SR3 models to produce 1024×1024 images through 64×64→256×256→1024×1024 refinement.Source: Figure C.6 · p. 20, Figure C.7 · p. 21, Figure C.8 · p. 22
• Cascaded ImageNet generation samples a random label, generates a 64×64 class-conditional image, and applies 4× SR3 to obtain 256×256 images.Source: Figure C.10 · p. 24, Figure C.11 · p. 25

Complex facial detail

• SR3 struggles with complex, regular hair patterns such as finely braided hair.The authors suggest limited exposure to such structures in the relatively small FFHQ training dataset as one possible reason.Source: §C.1 · p. 26
• SR3 can struggle with long-range correlations in fine details, including consistent eye highlights.Source: §C.1 · p. 26
• Frameless eyeglasses are difficult to generate when their structure is hard to discern from the low-resolution input.Source: §C.1 · p. 26

Text generation

• SR3 fails to generate natural-looking text in certain ImageNet images because it has not learned common alphabets.Its generated characters may be sharper than a regression baseline but lack meaningful structure, making them easier to distinguish from real images.Source: §C.1 · p. 26

Ambiguous fine detail

• When low-resolution inputs omit fine details, SR3 may infer incorrect or overly uniform structure, including building-side textures.The output can appear sharper while still differing from the high-resolution original.Source: §C.1 · p. 26

• Examining images that maximize or minimize fool rates provides insight into model problems and outputs that regularly fool people.Source: D. Images with the Lowest and Highest… · p. 26
• PULSE images with the lowest fool rates show obvious distortions, with fool rates below 10% in both evaluation tasks.Source: D. Images with the Lowest and Highest… · p. 26
• SR3’s best fool rates reach 84% and 88% in the examples that most successfully fool human subjects.The corresponding original images are somewhat noisy, which leads many subjects to prefer the SR3 outputs.Source: D. Images with the Lowest and Highest… · p. 26

• Task-1 compares each algorithm’s output with reference high-resolution images in the presence of low-resolution inputs.Source: Figure D.1 · p. 27
• The figure omits reference images for privacy reasons and instead displays the corresponding PULSE and SR3 outputs for each input image.Source: Figure D.1 · p. 27
• Mean Fool Rate is reported directly below each input image’s corresponding outputs.Source: Figure D.1 · p. 27

• Task-2 compares each algorithm’s output with reference high-resolution images in the absence of low-resolution inputs.Source: Figure D.2 · p. 28
• For privacy reasons, the figure omits reference images and displays the corresponding PULSE and SR3 outputs for each input image.Source: Figure D.2 · p. 28
• Mean Fool Rate is reported below each image’s corresponding outputs.Source: Figure D.2 · p. 28