PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Step Effective Diffusion Network for Real-World Image Super-Resolution

Rongyuan Wu, Lingchen Sun, Zhiyuan Ma, Lei Zhang

arXiv:2406.08177v3eess.IVcs.CV

TL;DR

Real-ISR diffusion methods offer strong natural-image priors but commonly require multiple noisy diffusion steps, creating computational cost and output uncertainty. OSEDiff instead starts from the LQ image, adapts a pretrained diffusion network with trainable layers, and applies latent-space variational score distillation. It generates HQ images in one step while achieving comparable or better results than prior diffusion-based Real-ISR methods.

  • Problem

    Existing diffusion-based Real-ISR methods require multiple diffusion steps and random-noise initialization, increasing computation and introducing output uncertainty.

  • Method

    OSEDiff directly uses the LQ image as the diffusion starting point, fine-tunes trainable layers for complex degradations, and applies latent-space variational score distillation for KL regularization.

  • Results

    OSEDiff achieves comparable or superior Real-ISR performance to prior SD-based methods while reducing inference from N steps to 1 and delivering more than ×100 speedup over StableSR.

  • Takeaways & Limitations

    A pretrained text-to-image diffusion model can serve as both generator and regularizer for efficient one-step Real-ISR.

  • Takeaways & Limitations

    OSEDiff’s detail generation can improve, and like other SD-based methods it is limited on fine-scale structures such as small scene texts.

Abstract

from arXiv · show

The pre-trained text-to-image diffusion models have been increasingly employed to tackle the real-world image super-resolution (Real-ISR) problem due to their powerful generative image priors. Most of the existing methods start from random noise to reconstruct the high-quality (HQ) image under the guidance of the given low-quality (LQ) image. While promising results have been achieved, such Real-ISR methods require multiple diffusion steps to reproduce the HQ image, increasing the computational cost. Meanwhile, the random noise introduces uncertainty in the output, which is unfriendly to image restoration tasks. To address these issues, we propose a one-step effective diffusion network, namely OSEDiff, for the Real-ISR problem. We argue that the LQ image contains rich information to restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling. We finetune the pre-trained diffusion network with trainable layers to adapt it to complex image degradations. To ensure that the one-step diffusion model could yield HQ Real-ISR output, we apply variational score distillation in the latent space to conduct KL-divergence regularization. As a result, our OSEDiff model can efficiently and effectively generate HQ images in just one diffusion step. Our experiments demonstrate that OSEDiff achieves comparable or even better Real-ISR results, in terms of both objective metrics and subjective evaluations, than previous diffusion model-based Real-ISR methods that require dozens or hundreds of steps. The source codes are released at https://github.com/cswry/OSEDiff.

1 Introduction

Real-ISR seeks perceptually realistic HQ images from LQ observations with complex, unknown degradations, but diffusion approaches remain computationally costly and noise-dependent. OSEDiff addresses these issues by using the LQ image directly in a one-step diffusion model while adapting pretrained SD priors and preserving natural-image distributions.

  • Motivation: Real-ISR aims to reproduce perceptually realistic HQ images from real-world LQ images whose degradations are complex and unknown.Conventional ISR models trained with simplified degradation assumptions generalize poorly to real-world inputs.
  • Motivation: Real-ISR training must address both construction of LQ-HQ pairs and the naturalness of restored images.Real-world paired data are costly and cover limited degradation types, while pixel-wise learning can produce over-smoothed outputs.
  • Motivation: Pretrained text-to-image diffusion models provide strong natural-image priors that can improve the perceptual quality and detail of Real-ISR outputs.These models were trained on billions of image-text pairs and have been applied to Real-ISR because of their generative priors.
  • OSEDiff: OSEDiff directly feeds the LQ image into a pretrained SD model, avoiding random-noise initialization and reducing diffusion to one step.The method treats the LQ image as information-rich enough to restore its HQ counterpart.
  • OSEDiff: OSEDiff combines trainable LoRA layers with latent-space variational score distillation to adapt SD to complex degradations and align outputs with natural-image priors.The authors report comparable or superior performance to SD-based Real-ISR methods while reducing inference steps from N to 1 and achieving more than ×100 speedup over StableSR.

2 Related Work

Related Real-ISR work progresses from models built for simple degradations toward GAN and diffusion approaches for complex real-world inputs. Diffusion methods can improve realism but typically require many inference steps, motivating efficient alternatives.

  • ISR and GAN-based Real-ISR: Most conventional ISR methods assume simple, known degradations such as bicubic downsampling, limiting their use on real-world images.Real-world degradations are more complex and unknown than the assumptions used by these models.
  • ISR and GAN-based Real-ISR: BSRGAN and Real-ESRGAN use randomized or high-order degradation processes to generate more realistic LQ-HQ training pairs for Real-ISR.These methods demonstrated promising results with complex degradation modeling.
  • Diffusion-based Real-ISR: Early diffusion-based ISR methods modify pretrained DDPM reverse transitions for simple degradations but cannot handle complex unknown degradations.These approaches are training-free and use gradient descent during the reverse process.
  • Diffusion-based Real-ISR: Diffusion-based Real-ISR methods generally require tens or hundreds of steps, creating unfavorable inference latency.SinSR reduces ResShift to one step, but its non-distribution-based distillation tends to produce smooth results and uses smaller model capacity than SD-based methods.

3 Methodology

OSEDiff formulates Real-ISR as a learned LQ-to-HQ mapping that combines a one-step diffusion transformation with natural-image-prior regularization. It starts from the LQ image without random noise, adapts a pretrained Stable Diffusion model with trainable components, and uses latent-space VSD during training.

  • Problem Modeling: Real-ISR estimates an HQ image from an LQ observation, while unknown degradations and complex natural-image priors make conventional optimization difficult.The model learns an LQ-HQ mapping and uses data and regularization terms to balance fidelity and naturalness.
  • One-Step Effective Diffusion Network: OSEDiff directly feeds the LQ image into a pretrained Stable Diffusion model, avoiding random noise and performing only one denoising step.The LQ latent is transformed at the T-th diffusion timestep, with text prompts extracted from the LQ input.
  • Network Architecture Design: The generator consists of a trainable encoder, a LoRA-finetuned diffusion network, and a frozen decoder that synthesizes the HQ output.LoRA layers are introduced into the encoder and diffusion network, while the decoder remains fixed to preserve compatibility with the regularizers.
  • One-Step Effective Diffusion Network: Text prompts extracted from the LQ image condition noise prediction and stimulate the diffusion network's image-generation capacity.The denoising process uses text embeddings derived from the LQ input through a prompt extractor.
  • Training Loss: Training combines a weighted MSE-plus-LPIPS data loss with a VSD regularization loss that aligns generated outputs with the HQ natural-image distribution.VSD uses a pretrained regularizer and a LoRA-finetuned regularizer, with regularization gradients back-propagated to the generator.

4 Experiments

OSEDiff is evaluated against diffusion-based Real-ISR methods on synthetic and real-world benchmarks using quality, perceptual, distributional, semantic, and complexity measures. It achieves strong perceptual and efficiency results with one-step inference, while prompt extraction and VAE finetuning reveal explicit quality–cost and stability trade-offs.

  • Experimental settings: OSEDiff is trained on LSDIR and 10K FFHQ face images with Real-ESRGAN degradations and evaluated on StableSR’s synthetic and real-world test data.The comparison includes synthetic data and real-world benchmarks, alongside competing diffusion-based methods.
  • Quantitative comparisons: OSEDiff shows clear advantages in LPIPS, DISTS, FID, and CLIPIQA, especially on the real-world DrealSR and RealSR datasets.SeeSR and PASD perform better on some no-reference metrics, whereas ResShift and SinSR obtain stronger PSNR but poorer perceptual quality.
  • Qualitative comparisons: OSEDiff reproduces realistic facial details and natural leaf veins in qualitative comparisons despite using only one forward propagation.Other methods respectively show blurred details, excessive or incorrect details, or unnatural structures in the illustrated examples.
  • Complexity comparisons: OSEDiff is approximately 105 times faster than StableSR, 39 times faster than SeeSR, and 6 times faster than ResShift.It uses one diffusion step, 2265G MACs, and 8.5M trainable LoRA parameters; StableSR uses 200 steps and 79940G MACs.
  • Ablation study: Removing VSD loss significantly degrades perceptual quality, while image-domain VSD and GAN loss improve results less than latent-space VSD.The ablation compares removing VSD, replacing it with GAN loss, and applying VSD in the image domain.
  • Ablation study: DAPE is selected over LLaVA because both produce similar visual detail, while LLaVA increases prompt-extraction inference time by 170 times.Without prompts, full-reference metrics improve but no-reference metrics worsen; prompt conditioning produces richer synthesized details.
  • Ablation study: Finetuning the VAE encoder improves perceptual quality, while fixing the decoder preserves the original latent space and supports more effective VSD minimization.MUSIQ improves from 58.99 to 69.09 with encoder finetuning, and CLIPIQ improves from 0.5778 to 0.6693 when the decoder remains fixed.

5 Conclusion and Limitation

OSEDiff uses a pretrained text-to-image model as generator and regularizer, directly starts diffusion from the LQ image, and produces HQ Real-ISR results in one step. Its remaining limitations are detail generation and reconstructing fine-scale structures such as small scene text.

  • OSEDiff uses a pretrained text-to-image model as both generator and regularizer for one-step Real-ISR.
  • Directly using the LQ image as the diffusion starting point avoids uncertainty from random noise sampling.
  • Trainable LoRA layers adapt the pretrained diffusion network to complex real-world image degradations.
  • Variational score distillation in latent space aligns predicted scores with pretrained multi-step models, enabling one-step HQ image generation.
  • OSEDiff achieves comparable or superior outcomes to previous multi-step diffusion-based methods in objective metrics and subjective assessments.
  • OSEDiff’s details generation can improve, and it remains limited on fine-scale structures such as small scene texts.

A Appendix

The appendix provides supplementary comparisons, user-study results, additional visual comparisons at 4× scaling, and OSEDiff’s training algorithm.

  • The appendix includes comparisons with GAN-based methods, referring to Section 4.1 of the main paper.
  • The appendix includes user-study results, referring to Section 4.1 of the main paper.
  • The appendix provides more real-world visual comparisons under a 4× scaling factor, referring to Section 4.2.
  • The appendix includes OSEDiff’s training algorithm, referring to Section 3.2 of the main paper.

A.1 Comparison with GAN-based Methods

Compared with four representative GAN-based Real-ISR methods, OSEDiff trades lower fidelity measures for stronger perceptual quality and realistic detail generation.

  • OSEDiff is compared with BSRGAN, Real-ESRGAN, LDL, and FeMaSR.
  • GAN-based methods achieve better fidelity measures such as PSNR and SSIM than OSEDiff.
  • OSEDiff achieves much better perceptual quality metrics than the compared GAN-based methods.
  • Visual comparisons show OSEDiff generating realistic details including squirrel hair, petal textures, buildings, and leaves.

A.2 User Study

A user study evaluated Real-ISR outputs on 20 real-world LQ images with 15 volunteers. OSEDiff ranked second, slightly behind SeeSR, while running over 10 times faster through one-step diffusion.

  • Volunteers selected the best HQ result based equally on perceptual quality and consistency of structure and texture with the LQ input.
  • The user study presented outputs from seven diffusion-based Real-ISR methods for 20 real-world LQ images.
  • Fifteen volunteers participated in the evaluation.
  • OSEDiff ranked second in the user study, slightly behind SeeSR.
  • OSEDiff runs over 10 times faster than SeeSR by performing only one-step diffusion.

A.3 More Visual Comparisons

OSEDiff is visually compared with GAN-based and other diffusion-model-based Real-ISR methods. The reported comparisons show comparable or better results across several image-content scenarios.

  • Visual comparisons: OSEDiff achieves comparable or better results than multi-step diffusion methods in visual comparisons.The comparisons include portraits, flower patterns, buildings, animal fur, and letters.
  • Quantitative comparisons: Table 8 compares OSEDiff with GAN-based methods on synthetic and real-world benchmarks.The best result for each metric is highlighted in red.

A.4 Algorithm of OSEDiff

Algorithm 1 trains OSEDiff using paired low- and high-quality images, trainable LoRA layers, diffusion-time sampling, and reconstruction plus regularization objectives. The resulting generator includes the trained encoder, diffusion network, and decoder components.

  • Initialization: The training scheme initializes OSEDiff from a pretrained diffusion model and adds trainable LoRA parameters.The initialization includes trainable LoRA components for the encoder, diffusion network, and decoder.
  • Network forward: Training samples low-quality and high-quality images from the dataset for the network forward pass.Algorithm 1 denotes these samples as xL and xH.
  • Diffusion-time sampling: The algorithm samples diffusion times from {20, ..., 980} and {1, ..., T} during training.These two time-sampling operations support the regularization and diffusion objectives shown in the pseudocode.
  • Optimization: The diffusion loss uses mean squared error between predicted and target noise, followed by a network parameter update.The pseudocode labels this objective Ldiff.
  • Output and guidance: The trained generator outputs the encoder, latent diffusion network, and decoder components of OSEDiff.Classifier-free guidance is used when calculating zϕ with cfg set to 7.5.
Loading 2406.08177v3…