PaperlayerView exact revision on arXiv

Source-linked AI summary

Arbitrary-steps Image Super-resolution via Diffusion Inversion

Zongsheng Yue, Kang Liao, Chen Change Loy

arXiv:2412.09013v2cs.CV

TL;DR

Super-resolution is ill-posed under complex, often unknown real-world degradation, and diffusion inversion poses additional challenges for high-fidelity restoration. InvSR predicts noise to initialize a pretrained diffusion model partway through its trajectory; it supports one to five sampling steps and is superior or comparable to recent dedicated one-step diffusion methods even at one step.

  • Problem

    Real-world super-resolution is ill-posed because its degradation model is complex and often unknown, while diffusion inversion is challenging for high-fidelity restoration.

    Source: §1 · p. 2

  • Method

    InvSR uses a trained noise predictor and Partial noise Prediction to construct an intermediate state for sampling, without modifying the diffusion network.

    Source: §1 · p. 2

  • Results

    InvSR supports one to five sampling steps and is superior or comparable to recent dedicated one-step diffusion methods even with one step.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sampling steps can be adjusted to degradation: extra steps can reduce blur, while for severe noise one step may suffice and additional steps may amplify noise or introduce artifacts.

    Source: §1 · p. 2

    For fidelity, the starting noise is constrained to relatively high SNR; in practice, N≤250 corresponds to an SNR threshold of 1.44.

    Source: §3.2.1 · p. 4

Abstract

from arXiv · show

This study presents a new image super-resolution (SR) technique based on diffusion inversion, aiming at harnessing the rich image priors encapsulated in large pre-trained diffusion models to improve SR performance. We design a Partial noise Prediction strategy to construct an intermediate state of the diffusion model, which serves as the starting sampling point. Central to our approach is a deep noise predictor to estimate the optimal noise maps for the forward diffusion process. Once trained, this noise predictor can be used to initialize the sampling process partially along the diffusion trajectory, generating the desirable high-resolution result. Compared to existing approaches, our method offers a flexible and efficient sampling mechanism that supports an arbitrary number of sampling steps, ranging from one to five. Even with a single sampling step, our method demonstrates superior or comparable performance to recent state-of-the-art approaches. The code and model are publicly available at https://github.com/zsyOAOA/InvSR.

1. Introduction

InvSR uses diffusion inversion to restore high-resolution images from low-resolution observations, estimating noise maps while keeping the diffusion network fixed. Its Partial noise Prediction strategy supports flexible sampling, with one-step performance superior or comparable to recent dedicated one-step diffusion methods.

1. Introduction

  • Diffusion inversion for super-resolution is challenging because per-step noise optimization is expensive and complex, while iterative errors and randomness can compromise fidelity.The passage notes that existing diffusion inversion methods have mainly focused on lower-fidelity tasks such as image editing.Source: §1 · p. 2
  • A noise predictor estimates a low-resolution-dependent noise map, and Partial noise Prediction uses it to construct the intermediate state from which reverse sampling begins.Instead of predicting noise maps at every diffusion step, the strategy predicts for the starting step.Source: §1 · p. 2
  • Sampling can be adjusted to degradation: multiple steps reduce blur and restore detail in one example, while one step is satisfactory for severe noise and extra steps may amplify noise or artifacts.The method supports selecting a sampling step from multiple pretrained starting steps and using an arbitrary number of sampling steps.Source: §1 · p. 2
  • One to five sampling steps yield performance superior or comparable to recent dedicated one-step diffusion methods, even with a single step.The method also keeps the diffusion backbone fixed while integrating an auxiliary noise predictor.Source: §1 · p. 2

2. Related Work

Prior diffusion-prior super-resolution methods re-optimize intermediate results or fine-tune pretrained text-to-image models, while diffusion inversion methods have primarily targeted image editing. InvSR adapts inversion to super-resolution using a trained noise predictor rather than iterative optimization at inference.

2. Related Work

  • Diffusion-prior super-resolution methods either re-optimize intermediate results, as in DDRM [23], CCDF [7], and DDNM [56], or fine-tune pretrained models, including StableSR [53] and successors; the optimization-based class is computationally complex and often assumes a degradation model.The cited review also notes that these assumed degradation models limit handling of blind real-world super-resolution.Source: §2 · pp. 2–3
  • Diffusion inversion methods optimize text or visual prompts and intermediate noise maps, but have mainly focused on image editing rather than super-resolution's high-fidelity requirements [13] [20] [33] [51].The passage also identifies DDIM [46] as establishing a deterministic generation process through non-Markovian processes.Source: §2 · p. 3
  • Unlike Chihaoui et al. [5], which solves an optimization problem at each inversion step, InvSR uses a trained noise prediction module to avoid iterative optimization during inference.The paper describes this as improving efficiency and practicality for super-resolution.Source: §2 · p. 3

3. Methodology

The methodology formulates super-resolution as finding noise maps that let a pre-trained diffusion model reconstruct an HR image from an LR observation. It simplifies inversion by predicting noise for a selected starting state, then follows the reverse trajectory with a flexible number of sampling steps.

3.1. Motivation

  • The method seeks noise maps that reconstruct the target HR image from an LR observation through a pre-trained diffusion model’s reverse process.It uses the DDPM formulation for presentation while stating that the technique applies to probabilistic and SDE-based diffusion formulations.Source: §3.1 · p. 3

3.2. Diffusion Inversion

  • A trained network takes the LR image and timestep as input to predict noise maps, replacing per-image optimization and improving inference efficiency.The network predicts Gaussian mean and variance parameters using a reparameterization trick.Source: §3.2 · p. 3

3.2.1. Problem Simplification

  • The Partial Noise Prediction strategy starts from an intermediate timestep N<T, reducing the inversion noise-map set from T maps to N.The approach exploits that LR and HR images often differ mainly in high-frequency components.Source: §3.2.1 · p. 4
  • For high-fidelity SR, the starting state is constrained to mild noise: N≤250 in Stable Diffusion, corresponding to an SNR threshold of 1.44.Skipping selected diffusion steps further reduces the predicted set to at most five noise maps.Source: §3.2.1 · p. 4

3.2.2. Inversion Trajectory

  • Because the HR image is unavailable at test time, the predictor perturbs the LR image to construct the starting state, then the method restores the HR image along the inversion trajectory.The design relies on LR and HR images becoming increasingly indistinguishable under Gaussian noise of appropriate magnitude.Source: §3.2.2 · p. 4

3.2.3. Model Training

  • Training uses an alternative marginal-distribution formulation for intermediate states to avoid recurrent calls to the diffusion model, and combines L2, LPIPS, and GAN losses.The initial predicted noise is LR-correlated and non-zero-mean, so the Gaussian constraint is not applied to the initial state.Source: §3.2.3 · pp. 4–5
  • The predictor estimates noise only for the starting state and supports multiple preselected starting timesteps, allowing inference to choose a timestep and sampling-step count.The selected starting timestep yields a fidelity–realism trade-off; this study considers one to five sampling steps.Source: §3.2.3 · p. 5

4. Experiments

Experiments analyze InvSR and assess ×4 super-resolution on one synthetic and two real-world datasets. They examine sampling choices and compare the method with recent approaches.

4.1. Experimental Setup

  • The predictor was trained on LSDIR and FFHQ and sampled starting timesteps, then evaluated on one synthetic and two real-world datasets against nine methods using seven metrics.RealSet80 used non-reference metrics because ground-truth images were unavailable.Source: §4.1 · pp. 5–6

4.2. Model Analysis

  • InvSR supports arbitrary sampling-step counts; starting-timestep choice trades fidelity for realism, while extra steps can hurt noisy images but restore detail in substantial blur.The analysis compares one-, three-, and five-step settings.Source: §4.2 · pp. 6–7
  • The predicted initial noise map correlates strongly with image content and provides an LR-dependent noise map to facilitate super-resolution.Figure 3 shows the zoomed LR image, predicted initial noise map, and single-step result.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3. Comparison to State of the Arts

  • InvSR beats OSEDiff on all seven ImageNet-Test metrics; against 50-step StableSR and DiffBIR, it is comparable on PSNR/SSIM and better on LPIPS, NIQE, PI, and MUSIQ.The paper describes PSNR and SSIM as distortion-oriented metrics and LPIPS, NIQE, PI, and MUSIQ as perception-oriented metrics.Source: §4.3 · pp. 7–8
  • Across RealSR and RealSet80, InvSR surpasses recent one-step methods on most non-reference metrics and ranks second to multi-step methods; examples show artifact removal and sharper structures.The examples contrast a compression-noisy image with a noticeably blurred image.Source: §4.3 · p. 8, Figure 4 · p. 8

5. Conclusion

InvSR uses a noise prediction network to initialize a pre-trained diffusion model from an intermediate state. It supports flexible sampling, and even one-step sampling outperforms recent one-step diffusion-based methods.

5. Conclusion

  • A noise prediction network estimates an optimal noise map to construct an intermediate state of a pre-trained diffusion model as the sampling start.The diffusion prior remains pre-trained while the predicted noise map supplies the initialization.Source: §5 · p. 8
  • A time-dependent noise predictor supports starting from various intermediate states, letting users adjust sampling steps to degradation type or specific requirements.Source: §5 · p. 8
  • With one sampling step, InvSR still exhibits significant superiority over recent one-step diffusion-based methods.Source: §5 · p. 8

Supplementary Material

The supplement surveys the paper’s sampling-step choice and supporting analyses. It recommends one step for most real-world SR applications, while allowing the step count to vary with degradation type.

Supplementary Material

  • The supplementary material covers sampling-step selection, base diffusion models, noise-prediction and loss ablations, efficiency and limitations, and additional visual comparisons.It lists ImageNet-Test and real-world examples among the visual comparisons.Source: Supplementary Material · p. 12

A. Discussion on Sampling Steps

  • InvSR’s single-step results outperform recent state-of-the-art one-step methods, supporting a one-step recommendation for most real-world applications.The authors frame this as a balance between efficiency and performance.Source: A. Discussion on Sampling Steps · p. 12
  • For noisy images, multi-step sampling can amplify noise and create artifacts, whereas for blurry images it can restore more detailed, realistic structures.The authors suggest estimating noise level to choose between one-step and multi-step sampling; the multi-step count remains adjustable.Source: A. Discussion on Sampling Steps · p. 12

B.1. Base Diffusion Model

The authors compare InvSR using SD-2.0 and SD-Turbo. Their results favor SD-Turbo for multi-step sampling, so they use it as the base model throughout the study.

B.1. Base Diffusion Model

  • With one sampling step, InvSR shows similar performance with SD-2.0 and SD-Turbo; in multi-step settings, SD-Turbo is more stable, particularly on reference metrics.The comparison is on ImageNet-Test.Source: §B.1 · p. 12
  • At five sampling steps, the SD-2.0-based model produces noticeable artifacts, consistent with the quantitative comparison.Figure 5 compares the two base diffusion models visually.Source: §B.1 · p. 12, Figure 5 · p. 12
  • The study uses SD-Turbo as InvSR’s base model throughout.The compared variants are SD-2.0 and SD-Turbo.Source: §B.1 · p. 12

B.2. Intermediate Noise Prediction

InvSR samples intermediate noise maps randomly rather than predicting them. The authors report negligible performance differences from adding an intermediate-timestep predictor and omit it to simplify the framework.

B.2. Intermediate Noise Prediction

  • Adding an extra predictor trained for intermediate timesteps yields negligible performance differences from InvSR.The comparison is between InvSR and InvSR-Int on ImageNet-Test.Source: §B.2 · p. 12
  • InvSR samples intermediate noise maps randomly rather than predicting them, motivated by the high-SNR constraint on inversion timesteps.Source: §B.2 · p. 12
  • The authors omit intermediate noise prediction, further simplifying the framework.Source: §B.2 · p. 12

B.3. Loss Functions

The noise predictor is trained with L2, LPIPS, and GAN losses. The ablation describes a trade-off between smoothness, detail, and artifacts, motivating the use of both perceptual and adversarial losses.

B.3. Loss Functions

  • Combining LPIPS and GAN losses is the chosen configuration for balancing perceptual quality and artifact suppression.The authors state that their combination achieves optimal performance.Source: §B.3 · pp. 12–13, Table 6 · p. 13
  • L2-only training produces over-smooth outputs, consistent with its superior PSNR scores.Source: §B.3 · pp. 12–13
  • Adding GAN loss enhances finer image details but may introduce undesirable artifacts; adding LPIPS can mitigate those artifacts to some extent.Source: §B.3 · pp. 12–13

B.4. Efficiency and Limitation

InvSR is faster than OSEDiff among one-step diffusion methods, but remains less efficient than GAN-based methods because it relies on Stable Diffusion. The accompanying comparisons illustrate how sampling-step choices relate to degradation type.

B.4. Efficiency and Limitation

  • Despite having more parameters than OSEDiff, InvSR achieves a 50% reduction in inference time, mainly because OSEDiff relies on an additional image captioning model that InvSR does not use.Source: §B.4 · p. 13
  • InvSR remains less efficient than GAN-based methods because it relies on the large-scale Stable Diffusion model.The authors identify model quantization as future work to accelerate inference.Source: §B.4 · p. 13
  • For blurry examples, multi-step sampling recovers finer details, while one step can suffice for severe noise and additional steps may amplify it.Figure 6 compares InvSR with different numbers of sampling steps.Source: Figure 6 · p. 14
  • The visual comparisons cover alternative loss configurations and method outputs on ImageNet-Test and RealSet80.The captions identify the compared settings and datasets but do not state which methods perform best.Source: Figure 7 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16

Related papers

Loading 2412.09013v2…