PaperlayerView exact revision on arXiv

Source-linked AI summary

ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting

Zongsheng Yue, Jianyi Wang, Chen Change Loy

arXiv:2307.12348v3cs.CV

TL;DR

Diffusion-based SR methods are slowed by hundreds or thousands of sampling steps, and acceleration techniques can cause over-blurry results. ResShift instead shifts residuals between HR and LR images in a shorter Markov chain with a flexible noise schedule, achieving superior or comparable performance on synthetic and real-world datasets with 15 sampling steps.

  • Problem

    Diffusion-based SR methods often require hundreds or thousands of sampling steps, while acceleration techniques can sacrifice performance and produce blurry outputs.

  • Method

    ResShift constructs a Markov chain between HR and LR images by shifting their residual and uses a flexible noise schedule to control shifting speed and noise strength.

  • Results

    15 sampling steps achieve superior or at least comparable performance to current diffusion-based SR methods on synthetic and real-world datasets.

  • Takeaways & Limitations

    Starting diffusion from an LR-based prior rather than Gaussian white noise can shorten the diffusion process and improve inference efficiency for SR.

  • Takeaways & Limitations

    Further optimizations of the proposed method remain necessary, and the authors leave them for future work.

Abstract

from arXiv · show

Diffusion-based image super-resolution (SR) methods are mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose a novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post-acceleration during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, even only with 15 sampling steps. Our code and model are available at https://github.com/zsyOAOA/ResShift.

1 Introduction

Image super-resolution is ill-posed in real-world settings, while diffusion-based approaches face an efficiency–performance trade-off. ResShift addresses this by shifting residuals between HR and LR images through a shorter, flexibly scheduled Markov chain.

  • Real-world image super-resolution is severely ill-posed because degradation models are complex and unknown.
  • Existing diffusion-based SR methods often require hundreds or thousands of sampling steps, while acceleration can produce over-smooth results.The introduction identifies the need to improve efficiency without sacrificing performance.
  • ResShift constructs a shorter Markov chain that transitions between HR and LR images by shifting their residual step by step.Its initial and final states approximate the HR and LR image distributions, respectively.
  • 15 sampling steps yield appealing SR results while outperforming or matching diffusion-based methods that require longer sampling processes.The contribution compares the efficiency of the proposed approach with existing diffusion-based SR methods.
  • The proposed noise schedule controls both residual-shifting speed and noise strength, enabling a fidelity–realism trade-off through its hyperparameters.

2 Methodology

ResShift builds a short HR-to-LR Markov chain by progressively shifting the LR–HR residual, with a flexible schedule controlling transition speed and noise. Reverse sampling uses a learnable inverse kernel to estimate the HR image from the LR image.

  • ResShift assumes equal-resolution HR and LR images, using nearest-neighbor pre-upsampling when necessary.
  • 2.1 Model Design: The forward process shifts the residual e0 = y0 − x0 from the HR image toward the LR image over T Markov steps.
  • 2.1 Model Design: The transition kernel adds a residual-dependent mean shift αte0 and Gaussian noise with variance κ2αtI.
  • 2.1 Model Design: Its marginal at timestep t is analytically tractable as a Gaussian centered at x0 + ηte0 with noise scale κ2ηtI.
  • 2.1 Model Design: The chain approaches the HR image distribution at its initial state and a noisy LR-centered distribution at its final state, enabling reverse sampling conditioned on y0.
  • 2.1 Model Design: The reverse process learns an inverse transition kernel whose neural network fθ predicts x0 from xt, y0, and t.
  • ResShift can operate in VQGAN latent space, compressing spatial dimensions by a factor of four without changing the model formulation.
  • 2.2 Noise Schedule: The noise schedule controls initial and final noise levels, intermediate residual-shifting speed, and the fidelity–realism trade-off through κ and p.

3 Related Work

Diffusion-based SR methods commonly either retrain conditioned diffusion models or modify unconditional reverse paths, but both inherit long Markov chains. ResShift instead uses an iterative HR–LR restoration formulation with a discrete Markov chain.

  • Image Super-Resolution: Diffusion models have been applied to SR through conditional retraining or guided reverse processes using unconditional diffusion priors.
  • Image Super-Resolution: These diffusion-based SR strategies often require hundreds or thousands of sampling steps to generate realistic HR images.
  • Image Super-Resolution: ResShift formulates iterative SR as a discrete Markov-chain transition between HR and LR images, differing mathematically from parallel InDI and SDE-based approaches.
  • Table 1 evaluates ResShift under different configurations on ImageNet-Test, supporting comparisons of its diffusion steps and hyper-parameters.

4 Experiments

Experiments evaluate ResShift on synthetic and real-world SR datasets, analyze its hyperparameters and perception-distortion behavior, and compare its efficiency and quality with competing methods. ResShift achieves strong quality with 15 sampling steps, while further optimization remains necessary for speed relative to GAN-based methods.

  • Experimental Setup: Experiments focus on the challenging ×4 SR task using one synthetic dataset and three real-world datasets.The synthetic evaluation uses ImageNet-Test, while real-world evaluations include RealSR and RealSet65.
  • Experimental Setup: ResShift is compared with seven recent SR methods, including LDM and its accelerated variants matched to ResShift’s inference-step count.LDM uses 1,000 diffusion steps, while LDM-A denotes DDIM-accelerated sampling to A steps.
  • Model Analysis: Increasing p improves PSNR and SSIM but degrades non-reference metrics, creating a fidelity-realism trade-off and potentially blurrier outputs.The analyzed metrics include PSNR, SSIM, LPIPS, CLIPIQA, and MUSIQ.
  • Model Analysis: κ values between 1.0 and 2.0 yield the most realistic quality according to CLIPIQA and MUSIQ, so κ is set to 2.0.Excessively large or small κ values smooth recovered results despite favorable PSNR and SSIM.
  • Efficiency Comparison: ResShift uses T = 15 and p = 0.3, surpasses LDM in PSNR and LPIPS, and achieves a fourfold computational-efficiency improvement over LDM-100.It still runs slower than current GAN-based methods because of iterative sampling.
  • Perception-Distortion Trade-off: ResShift’s perception-distortion curve remains beneath LDM’s across 10, 15, 20, 30, 40, and 50 diffusion steps.The comparison uses LPIPS for perception and MSE for distortion.
  • Evaluation on Synthetic Data: On ImageNet-Test, ResShift shows superior or comparable performance across all five metrics, with advantages in fidelity and perceptual quality.It also achieves comparable MUSIQ performance with recent state-of-the-art methods.
  • Evaluation on Real-World Data: On two real-world datasets, ResShift surpasses existing methods in CLIPIQA and achieves competitive MUSIQ performance against BSRGAN, SwinIR, and RealESRGAN.Qualitative examples show more naturalistic structures in several real-world scenarios, while LDM outputs are excessively smooth.

5 Conclusion

The conclusion presents ResShift as an efficient diffusion model for SR that uses 15 sampling steps and shifts residual information from HR toward LR rather than corrupting HR to Gaussian noise. Experiments on synthetic and real-world datasets support its reported effectiveness and inference-efficiency gains.

  • Conclusion: ResShift is introduced as an efficient diffusion model for image super-resolution.The method is designed to address the large iteration counts used by existing diffusion-based SR methods.
  • Conclusion: ResShift uses only 15 sampling steps, significantly improving inference efficiency.Its diffusion process corrupts the HR image toward the LR image instead of Gaussian white noise.
  • Conclusion: Experiments on synthetic and real-world datasets demonstrate the superiority of the proposed method.The conclusion frames these results as evidence of the method’s effectiveness for SR.

A Mathematical Details

The appendix derives the reparameterized transition and posterior distributions used in ResShift’s diffusion process. It applies Bayes’ theorem and Gaussian quadratic-form algebra to obtain the reverse transition.

  • Transition Derivation: The transition distribution is sampled through a reparameterization trick and recursively expanded to relate x_t to x_0.The derivation first gives the sampling form, then applies it recursively before merging the independent noise terms.
  • Posterior Derivation: Bayes’ theorem is used to derive the conditional distribution q(x_{t−1}|x_t, x_0, y_0).The derivation proceeds by analyzing the conditional distribution implied by the forward transitions.
  • Posterior Derivation: The quadratic form in the exponent induces the Gaussian distribution used for the reverse transition.Terms independent of x_{t−1} are treated as constants while completing the Gaussian form.

B.1 Degradation Settings of the Synthetic Dataset

The synthetic testing data use a RealESRGAN-based degradation pipeline, with the second-order operation removed to better match authentic real-world low-resolution images. The settings vary blur, downsampling, noise, and JPEG compression.

  • Dataset Construction: The synthetic dataset follows RealESRGAN’s degradation model but removes second-order degradation because it produced excessively corrupted low-resolution images.The adjustment is intended to align the synthetic degradation more closely with most real-world low-resolution images.
  • Blurring Kernel: Blurring uses 13-sized isotropic or anisotropic Gaussian kernels sampled with probabilities 0.6 and 0.4, respectively.Kernel widths are uniformly sampled from [0.2, 0.8].
  • Downsampling: Downsampling randomly selects PyTorch interpolation modes: area, bilinear, or bicubic.The degradation pipeline uses the “interpolate” function for this operation.
  • Noise and Compression: Noise randomly uses Gaussian or Poisson corruption with probabilities 0.5 and 0.5, followed by JPEG compression with quality 70–95.Gaussian noise levels range from 1 to 15, while Poisson scale ranges from 0.05 to 0.3.

B.2 Evaluation on Bicubic Degradation

On x4 bicubic super-resolution, ResShift is reported as superior to the compared diffusion-based methods across quantitative measures, parameter counts, and inference throughput. Visual examples likewise show richer and more realistic recovered details.

  • Quantitative Comparison: ResShift outperforms DDRM, IRSDE, and I2SB across assessment metrics, parameter counts, and inference throughput.The comparison retrains a model for bicubic degradation and evaluates 3,000 randomly selected validation images.
  • Qualitative Comparison: Visual comparisons show ResShift recovering richer and more realistic image details than the alternative methods.The qualitative result is reported as consistent with the quantitative comparison.

B.3 Limitation

ResShift can fail on severely degraded real-world inputs, including a comic image in RealSet65. The paper attributes this boundary to synthetic degradation models that do not cover the full range of complicated real degradation types.

  • Observed Failure: ResShift occasionally fails to produce satisfactory results for a severely degraded comic image in RealSet65.The compared methods also struggle with this example.
  • Scope Boundary: Synthetic degradation models based on manual assumptions cannot cover the full range of complicated real degradation types.The paper identifies developing a more practical degradation model as future work.
  • Real-World Comparisons: Figures 10 and 11 provide qualitative comparisons of methods on real-world datasets.The captions instruct readers to zoom in for a better view.
Loading 2307.12348v3…