Efficient Diffusion Model for Image Restoration by Residual Shifting

Zongsheng Yue; Jianyi Wang; Chen Change Loy

Source-linked AI summary

Efficient Diffusion Model for Image Restoration by Residual Shifting

Zongsheng Yue, Jianyi Wang, Chen Change Loy

arXiv:2403.07319v3Mar 12, 2024cs.CV

TL;DR

Diffusion-based image restoration requires many iterations for satisfactory results, and complex degradations make restoration challenging. ResShift shifts residual information along a short Markov chain between high- and low-quality images; it reports appealing results in four sampling steps, outperforming or matching current state-of-the-art methods.

Problem
Diffusion-based image restoration methods require many iterations for satisfactory results, while complex practical degradations make restoration severely ill-posed and challenging.
Source: §I · p. 1, §VI · p. 12
Method
ResShift uses a short Markov chain that shifts residual information between high- and low-quality images, corrupting the high-quality image toward its low-quality counterpart.
Source: §I · p. 2, §VI · p. 12
Results
With four sampling steps, the method achieves appealing results and outperforms or is at least comparable to current state-of-the-art methods.
Source: §I · p. 2
Takeaways & Limitations
Corrupting the high-quality image toward its low-quality counterpart truncates the diffusion chain and improves inference efficiency.
Source: §VI · p. 12
The method can fail on severely degraded comic images, and synthetic degradation models do not cover the full range of complicated real degradations.
Source: 3) Limitation · pp. 19–20

Abstract

from arXiv · show

While diffusion-based image restoration (IR) methods have achieved remarkable success, they are still limited by the low inference speed attributed to the necessity of executing hundreds or even thousands of sampling steps. Existing acceleration sampling techniques, though seeking to expedite the process, inevitably sacrifice performance to some extent, resulting in over-blurry restored outcomes. To address this issue, this study proposes a novel and efficient diffusion model for IR that significantly reduces the required number of diffusion steps. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Specifically, our proposed method establishes a Markov chain that facilitates the transitions between the high-quality and low-quality images by shifting their residuals, substantially improving the transition efficiency. A carefully formulated noise schedule is devised to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experimental evaluations demonstrate that the proposed method achieves superior or comparable performance to current state-of-the-art methods on three classical IR tasks, namely image super-resolution, image inpainting, and blind face restoration, \textit{\textbf{even only with four sampling steps}}. Our code and model are publicly available at \url{https://github.com/zsyOAOA/ResShift}.

I. INTRODUCTION

Image restoration seeks to recover high-quality images from low-quality inputs, but diffusion-based approaches face a trade-off between slow sampling and quality loss from acceleration. This introduction motivates a shorter HQ-to-LQ residual-shifting chain and a flexible noise schedule for restoration.

I. INTRODUCTION

Diffusion-based restoration methods can require hundreds or thousands of sampling steps, while acceleration techniques may produce overly smooth results and lower performance.The introduction frames the need as improving efficiency without sacrificing performance.Source: §I · p. 1
The proposed model shifts residual information between HQ and LQ images along a shorter Markov chain instead of starting from Gaussian noise.Its transition kernel is designed to transfer residual information across a limited number of steps.Source: §I · pp. 1–2
A flexible noise schedule controls residual-transfer speed and noise strength, with hyperparameters used to balance fidelity and realism.The schedule is designed to control transition properties more precisely, including shifting speed and noise level.Source: §I · p. 2
With four sampling steps, the method is reported to outperform or match current state-of-the-art methods; the study evaluates four image-restoration tasks.The tasks are super-resolution, inpainting, blind face restoration, and deblurring.Source: §I · p. 2
The expanded model adds perceptual loss and replaces self-attention with shifted-window Swin Transformer blocks; the authors report reducing steps from 15 to 4.The architecture change is intended to support arbitrary inference resolutions.Source: §I · p. 2

II. RELATED WORK

The related-work overview traces restoration from conventional methods to deep-learning and diffusion-based approaches.The cited review describes conventional methods as relying on manually designed noise and image-prior assumptions, then notes deep networks' larger model capacity.Source: §I · p. 1, §II · p. 2, A. Conventional Image Restoration… · p. 3

A. Conventional Image Restoration Approaches

Conventional restoration methods use likelihood and prior terms, while manually designed assumptions about noise and image priors limit their model capacity.Deep-learning methods subsequently extended restoration performance across tasks including denoising, super-resolution, and deblurring.Source: A. Conventional Image Restoration… · p. 3

B. Diffusion-based Image Restoration Approaches

The section motivates faster diffusion-based restoration and introduces a short HQ-to-LQ residual-shifting Markov chain whose reverse trajectory restores an LQ input.

B. Diffusion-based Image Restoration Approaches

Prior diffusion-based restoration methods condition on the LQ image or use a pre-trained prior, while the section identifies their many sampling steps as an inference-efficiency constraint.The pre-trained-prior methods described here include zero-shot approaches based on degradation models and trainable modules for blind restoration.Source: B. Diffusion-based Image Restoration… · p. 3

A. Model Design

The method shifts the HQ–LQ residual through a short Markov chain, then restores by reverse sampling from the given LQ image.The chain’s endpoint distributions approximate the HQ and LQ image distributions, respectively.Source: A. Model Design · pp. 3–4
The transition distribution shifts the residual incrementally, with a noise schedule controlling the transition and its variance.The marginal distribution is given analytically, and its terminal state approximates a noisy LQ image.Source: A. Model Design · p. 3
The reverse-process mean uses a neural network to predict the HQ image, and the trained model generates a restoration by sampling the reverse trajectory.The authors report that their chosen mean parameterization has superior stability and performance.Source: A. Model Design · pp. 4–5
Perceptual regularization constrains the solution space to support four-step sampling while maintaining superior performance.The authors add a pre-trained LPIPS constraint because L2-based loss tends to produce over-smooth results with fewer than five steps.Source: A. Model Design · p. 5
The model can optionally operate in VQGAN latent space, compressing the image by a factor of four in spatial dimensions.This extension substitutes latent codes for the HQ and LQ images without otherwise modifying the model.Source: Figure 2 · p. 4, A. Model Design · p. 5

B. Noise Schedule

The noise schedule uses κ to set overall noise intensity and p to control shifting speed, allowing the transition profile to vary across configurations.

B. Noise Schedule

κ regulates overall noise intensity, while p controls the growth rate of the shifting sequence √ηt.The schedule sets η1 to a small initial value and ηT to 0.999, with a non-uniform geometric schedule between them.Source: B. Noise Schedule · pp. 5–6
The schedule can end near a perturbed LQ image for small κ, shortening the chain relative to diffusion ending in Gaussian noise.The authors also state that p controls the shifting-speed trade-off in super-resolution results.Source: B. Noise Schedule · p. 6
With κ = 40 and p = 0.8, the proposed method’s diffusion process degenerates into LDM, as demonstrated by the visual diffusion-process results in Fig. 3(e)-(f).Figure 3(g) shows comparisons of relative noise strength.Source: B. Noise Schedule · p. 6

C. Relation to Flow Matching

The paper presents its diffusion method as a conditional flow between LQ and HQ image distributions. The flow linearly interpolates between images with added noise, providing a rapid transport path.

C. Relation to Flow Matching

The method corresponds to a conditional flow from the LQ image distribution to the HQ image distribution, tailored to image restoration.For a given LQ/HQ pair, the conditional path is described as a linear interpolation followed by noise disturbance.Source: C. Relation to Flow Matching · p. 6
The authors describe the LQ-to-HQ path as a rapid transport map that improves sampling inference.They frame flow matching as an alternative formulation and theoretical perspective on their diffusion method.Source: C. Relation to Flow Matching · p. 6

D. Discussion on Arbitrary Resolution

The discussion links self-attention to blur on images whose resolution differs from training and presents Swin Transformers as a resolution-independent alternative.

D. Discussion on Arbitrary Resolution

A self-attention model trained at 64 × 64 produces blurred results on 128 × 128 inputs, where its attention map becomes more uniform.The authors associate uniformly distributed attention maps with over-smooth outcomes.Source: D. Discussion on Arbitrary Resolution · p. 7
Replacing self-attention layers with Swin Transformers alleviates the blurring problem and maintains promised performance; Swin computes attention in a local window, independent of image resolution.Source: D. Discussion on Arbitrary Resolution · p. 7

IV. EXPERIMENTS ON IMAGE SUPER-RESOLUTION

The ×4 super-resolution evaluation ablates the proposed model before comparing ResShift variants with recent state-of-the-art methods on synthetic and real-world data.

The ×4 SR evaluation tests model ablations and compares ResShift and ResShiftL with recent state-of-the-art methods on synthetic and two real-world datasets.ResShift uses the primary loss with 15 diffusion steps; ResShiftL adds perceptual regularization and uses 4 steps.Source: §IV · p. 7

A. Experimental Setup

The super-resolution setup uses synthetic and real-world test data, compares ResShift variants with recent methods, and evaluates both fidelity and perceptual quality.

Test Datasets

ImageNet-Test uses 3,000 validation images with varied blur kernels, downsampling operators, and noise; larger source-image variety is intended to strengthen evaluation across degradations.The authors avoid Set5, Set14, and Urban100 because they contain few source images.Source: A. Experimental Setup · p. 7
RealSR-V3 contains 100 camera-captured images, while RealSet80 combines 50 images used in prior work with 30 collected online.The RealSR-V3 images were captured using Canon 5D3 and Nikon D810 cameras.Source: A. Experimental Setup · p. 7

Compared Methods and Evaluation Metrics

ResShift and ResShiftL are compared with nine recent SR methods, with diffusion-based competitors accelerated to 15 or 4 inference steps.Evaluation uses PSNR, SSIM, LPIPS, CLIPIQA, and MUSIQ; CLIPIQA and MUSIQ are non-reference realism metrics.Source: A. Experimental Setup · pp. 7–8

B. Ablation Studies

The experiments report competitive super-resolution results, examine loss and attention choices, and assess sampling consistency and real-world image quality.

B. Ablation Studies

The study notes that more complete analyses of the noise schedule, perception-distortion tradeoff, and advanced samplers appear in Appendix B2.The current section presents selected component ablations.Source: B. Ablation Studies · p. 8
ResShiftL with L2 fidelity loss performs better on reference metrics, while its L1 variant performs better on non-reference metrics; the study adopts L2 for IR's high-fidelity requirement.The comparison is summarized in Table II.Source: B. Ablation Studies · p. 8
Adding perceptual loss reduces sampling from 15 to 4 steps, slightly lowers PSNR and SSIM, and improves LPIPS, CLIPIQA, and MUSIQ.The authors treat the perceptual regularizer as a key performance-and-efficiency enhancement.Source: B. Ablation Studies · p. 8
Across random seeds, the method produces consistent outputs, while its remaining sampling randomness is described as manageable and useful for ill-posed restoration.The discussion contrasts starting from the HQ–LQ diffusion process with starting from random Gaussian noise.Source: B. Ablation Studies · p. 9

C. Evaluation on Synthetic Data

On ImageNet-Test, ResShift performs better than other diffusion methods across reference and non-reference metrics at equal sampling steps, while ResShiftL balances performance and speed against SwinIR.The authors attribute ResShiftL's efficiency to its shorter transition trajectory.Source: C. Evaluation on Synthetic Data · p. 9, Table V · p. 9

D. Evaluation on Real-World Data

On RealSR-V3 and RealSet80, ResShift and ResShiftL outperform existing methods on CLIPIQA, while ResShift is competitive on MUSIQ against BSRGAN, SwinIR, and RealESRGAN.Visual examples show more naturalistic structures than compressed-step LDM and StableSR outputs, which are described as excessively smooth; some GAN-based outputs show artifacts.Source: Table VI · p. 9, D. Evaluation on Real-World Data · p. 10, Figure 7 · p. 10

V. EXPERIMENTS ON IMAGE INPAINTING

The inpainting experiments evaluate natural and facial images across several mask types, reporting competitive results with four sampling steps and identifying a boundary on large masked regions.

V. EXPERIMENTS ON IMAGE INPAINTING

The study also evaluates the general IR framework on blind face restoration and image deblurring in the appendix.The inpainting models are trained separately for natural and facial images.Source: §V · p. 10, A. Experimental Setup · p. 11

A. Experimental Setup

The evaluation covers ImageNet and CelebA-HQ test sets with Box, Irregular, Half, and Expand masks, using LPIPS and CLIPIQA as principal metrics.Each test set contains 2,000 images divided across four mask types; diffusion competitors are accelerated to 250 DDIM steps.Source: A. Experimental Setup · p. 11

B. Comparison with SotA Methods

ResShiftL achieves best or comparable inpainting performance across most cases, particularly on challenging masks, while using 4 sampling steps versus 250 for other diffusion methods.Visual comparisons show plausible, realistic results on large occlusions where most comparison methods struggle, especially in preserving coherence with unmasked regions.Source: B. Comparison with SotA Methods · pp. 11–12, Table VIII · p. 11, Figure 8 · p. 11
Very large masked areas remain a failure case because the image provides too little information; the authors suggest supplementary guidance such as text prompts for future work.The failure is illustrated by a natural-image inpainting example.Source: B. Comparison with SotA Methods · p. 12, Figure 10 · p. 12

VI. CONCLUSION

The conclusion attributes the four-step efficiency to corrupting the HQ image toward its LQ counterpart rather than toward Gaussian white noise.The reported experiments demonstrate the method on super-resolution and inpainting.Source: §VI · p. 12

APPENDIX

The appendix derives the transition distributions, specifies the synthetic super-resolution degradation pipeline, and analyzes how diffusion-step and residual-shifting settings affect fidelity and perception.

A. Mathematical Details •

The derivation recursively combines Gaussian transition noise to obtain the marginal distribution of x_t from x_0.It begins with a reparameterized sampling step and merges the accumulated noise terms.Source: A. Mathematical Details • · p. 17
The quadratic form in the posterior derivation yields the Gaussian distribution q(x_{t−1}|x_t, x_0, y_0).The derivation identifies the terms dependent on x_{t−1} and completes the square.Source: A. Mathematical Details • · p. 17

B. Experimental Results on Image Super-resolution

The synthetic ImageNet-Test degradation pipeline uses randomized blur, downsampling, Gaussian or Poisson noise, and JPEG compression.Second-order degradation is omitted because it produced more pronounced corruption than most real-world LQ images.Source: 1) Degradation Settings of the… · p. 17

1) Degradation Settings of the Synthetic Dataset:

Increasing p is associated with better PSNR and SSIM but worse LPIPS, and larger p suppresses detail hallucination, producing blurrier results.The comparison evaluates ResShift configurations with different T and p on ImageNet-Test.Source: 2) Model Analysis · p. 18, Table IX · p. 18, Figure 11 · p. 18

2) Model Analysis:

The model analysis examines perception–distortion balance and accelerated-sampler comparisons, while the limitation and blind-face-restoration passages describe a real-degradation failure and evaluation results.

3) Limitation:

ResShift occasionally fails on severely degraded comic images, and the comparison methods also struggle with that example.The passage relates the difficulty to synthetic SR degradation models not covering the full range of complicated real degradations.Source: Figure 12 · p. 19, 3) Limitation · pp. 19–20

2) Model Analysis:

ResShiftL outperforms LDM with DPM, while PLMS improves non-reference metrics but compromises fidelity and introduces visible artifacts.The comparison covers ImageNet-Test super-resolution and uses four diffusion steps for LDM in the qualitative comparison.Source: Table X · p. 19, 2) Model Analysis · p. 19, Figure 14 · p. 20
Across diffusion steps from 4 to 50, ResShift’s perception–distortion curve lies below LDM’s, indicating a better balance of perception and distortion.Perception is measured by LPIPS and distortion by MSE.Source: Figure 13 · p. 19, 2) Model Analysis · p. 19
Values of κ that are too large or too small smooth results; κ in [1.0, 2.0] gives the most realistic quality by LPIPS, so the paper sets κ to 2.0.The passage notes that extreme κ values can have favorable PSNR and SSIM despite smoother outputs.Source: 2) Model Analysis · p. 19

1) Experimental Setup:

On CelebA-Test, ResShift leads in LPIPS and FID-G, ranks second on identity metrics, and is at least comparable on the other evaluated metrics.The result supports its reported performance in blind face restoration.Source: Table XI · p. 20, 2) Evaluation on Synthetic Dataset · p. 21
BFR training uses FFHQ-derived 512 × 512 pairs; evaluation spans CelebA-Test, LFW, WebPhoto, and WIDER against seven methods using six metrics and real-world no-reference FID-F and MUSIQ.The real-world datasets represent different degradation levels, including very heavy degradation in WIDER.Source: 1) Experimental Setup · pp. 20–21

2) Evaluation on Synthetic Dataset:

The evaluation reports blind-face-restoration and deblurring results, describes experimental settings, and provides qualitative comparisons across synthetic and real-world restoration examples.

1) Experimental Setup:

The deblurring evaluation trains ResShiftL on GoPro and reports PSNR, SSIM, LPIPS, FID, and NIQE, computing FID from non-overlapping 256 × 240 patches.The patch protocol is specified for 1280 × 720 source images.Source: 1) Experimental Setup · p. 21

2) Experimental Results:

On GoPro deblurring, ResShiftL performs better on LPIPS and FID but less favorably on PSNR and SSIM than existing methods.The passage attributes the distortion-metric limitation to information loss from implementation in VQGAN’s 8×-compressed latent space.Source: Figure 15 · p. 22, 2) Experimental Results · p. 22

2) Evaluation on Synthetic Dataset:

On CelebA-Test, ResShift handles severe degradation in four sampling steps; CodeFormer has slight artifacts and DifFace requires 100 steps.The described comparison notes that only CodeFormer, DifFace, and ResShift produce satisfactory faces in the two most severely degraded examples.Source: 2) Evaluation on Synthetic Dataset · p. 21, Figure 16 · p. 23

3) Evaluation on Real-world Dataset:

On the three real-world face datasets, ResShift surpasses counterparts on FID-F and is comparable on MUSIQ, with stronger visual results on the two more severely degraded datasets.All compared methods perform well on mildly degraded LFW images.Source: 3) Evaluation on Real-world Dataset · p. 21, Figure 17 · p. 24

2) Experimental Results:

The qualitative SR figures cover ImageNet-Test and RealSet80 examples, including heavy- and slight-degradation cases and synthesized SDXL-Turbo images.The RealSet80 examples include x4 upscaling at the resolutions stated in the figure captions.Source: Figure 18 · p. 25, Figure 19 · p. 26, Figure 22 · p. 28
The visual sampling comparison presents restored outputs from recent methods and ResShift under different random seeds, while another comparison shows diffusion results from 4 to 200 steps.The captions specify these as visual analyses rather than reporting a quantitative outcome.Source: Figure 20 · p. 26, Figure 21 · p. 27
The network architecture modifies a diffusion U-Net with Swin Transformer blocks containing LayerNorm, multi-head self-attention, and MLP components.The blocks are introduced to better handle images with various resolutions.Source: Figure 25 · p. 31