Source-linked AI summary
Exploiting Diffusion Prior for Real-World Image Super-Resolution
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C. K. Chan, Chen Change Loy
TL;DR
Blind real-world super-resolution must handle unknown degradations while retaining fidelity, but training diffusion SR models from scratch is costly. StableSR fine-tunes a time-aware encoder around a frozen diffusion prior and adds controllable feature wrapping and progressive aggregation sampling. Experiments report improved SR performance, while the method remains limited on small texts, faces, and patterns.
Problem
Blind real-world SR involves unknown degradations, while training diffusion SR models from scratch is costly and can jeopardize pretrained generative priors.
Method
StableSR fine-tunes a time-aware encoder and feature modulation layers around a frozen Stable Diffusion model, with controllable feature wrapping and progressive aggregation sampling.
Results
StableSR improves SR performance, with experiments identifying both time awareness and the diffusion prior as crucial to the reported gains.
Takeaways & Limitations
StableSR provides a strong baseline for exploiting diffusion priors in restoration while reducing source-intensive training and supporting fixed-resolution challenges.
Takeaways & Limitations
StableSR may struggle with small texts, faces, and patterns, although stronger diffusion priors and more high-quality data may help.
Abstract
from arXiv · showhide
We present a novel approach to leverage prior knowledge encapsulated in pre-trained text-to-image diffusion models for blind super-resolution (SR). Specifically, by employing our time-aware encoder, we can achieve promising restoration results without altering the pre-trained synthesis model, thereby preserving the generative prior and minimizing training cost. To remedy the loss of fidelity caused by the inherent stochasticity of diffusion models, we employ a controllable feature wrapping module that allows users to balance quality and fidelity by simply adjusting a scalar value during the inference process. Moreover, we develop a progressive aggregation sampling strategy to overcome the fixed-size constraints of pre-trained diffusion models, enabling adaptation to resolutions of any size. A comprehensive evaluation of our method using both synthetic and real-world benchmarks demonstrates its superiority over current state-of-the-art approaches. Code and models are available at https://github.com/IceClear/StableSR.
1 Introduction
StableSR adapts a frozen pretrained diffusion prior to blind real-world super-resolution through lightweight time-aware conditioning, while addressing fidelity, stochasticity, and arbitrary-resolution challenges.
- Training diffusion SR models from scratch is computationally expensive and may jeopardize the generative priors captured by synthesis models.
- Blind real-world SR involves unknown, complex degradations that make fixed degradation assumptions unreliable and limit generalizability.
- StableSR fine-tunes a lightweight time-aware encoder and feature modulation layers while keeping the pretrained diffusion model fixed.
- Time-aware conditioning provides stronger guidance early in diffusion sampling for fidelity and weaker guidance later to reduce degradation introduction.
- StableSR uses controllable feature wrapping to reduce diffusion randomness and allow a user-adjustable quality–fidelity trade-off.
- Progressive aggregation sampling fuses overlapping patches with Gaussian weighting at each diffusion iteration to smooth boundaries for arbitrary-resolution SR.
2 Related Work
Prior work covers predefined, learned, and synthesized degradation models for blind SR, while StableSR explores fine-tuning a frozen diffusion prior to reduce training cost and preserve generative capabilities.
- Early SR methods assume predefined degradations such as bicubic downsampling and blurring with known parameters.
- Blind SR methods address unknown real-world degradation by learning degradation models implicitly or synthesizing LR–HR pairs explicitly.
- Diffusion-based SR methods have shown competitive real-world performance, but training diffusion models for SR remains computationally costly.
- Prior-based SR methods use additional information, including reference HR images, to improve faithful texture generation in complex scenarios.
- StableSR targets the extensive generative prior of pretrained diffusion models rather than relying on conventional external image priors.
- Unlike LDM, StableSR fine-tunes directly on a frozen pretrained diffusion model instead of training the diffusion model from scratch.
3 Methodology
StableSR adapts a frozen Stable Diffusion prior for blind super-resolution using time-aware conditioning, controllable feature wrapping, and progressive aggregation sampling. These components provide adaptive guidance, balance realism with fidelity, and reduce artifacts when processing large images.
- Guided Finetuning with Time Awareness: StableSR uses a time-aware encoder and feature modulation layers trained with frozen Stable Diffusion to preserve its generative prior.The encoder extracts multi-scale low-resolution features and injects them into Stable Diffusion residual blocks through spatial feature transformations.
- Guided Finetuning with Time Awareness: Time embedding strengthens encoder guidance when the signal-to-noise ratio reaches approximately 5e−2, the stage where generated image content strongly influences super-resolution.Feature similarity before and after spatial feature transformation is used to measure the condition strength supplied by the encoder.
- Fidelity-Realism Trade-off: Color correction replaces the generated image’s low-frequency component with the low-resolution input’s corresponding component to improve color fidelity.The method supports pixel-domain correction by default and also proposes wavelet-based correction for some visual cases.
- Fidelity-Realism Trade-off: Controllable Feature Wrapping introduces an adjustable coefficient w ∈[0, 1] that trades realism against fidelity during autoencoder decoding.Smaller w favors Stable Diffusion’s generation capability, larger w provides stronger structural guidance, and w = 0.5 is observed to balance quality and fidelity.
- Aggregation Sampling: Progressive aggregation sampling processes overlapping 64×64 patches at every diffusion timestep and fuses them with Gaussian weight maps.The strategy mitigates discrepancies in overlapping regions and supports images larger than the diffusion model’s fixed training resolution.
4 Experiments
StableSR is evaluated on synthetic and real-world super-resolution benchmarks through quantitative, qualitative, user-study, and ablation comparisons. It generally improves perceptual quality and detail restoration while retaining efficiency advantages, though performance varies across metrics, resolutions, and content types.
- Quantitative Comparisons: StableSR achieves a 24.44 FID score on DIV2K Valid, 7.7% lower than LDM and at least 32.9% lower than GAN-based methods.It also achieves the highest CLIP-IQA scores on two real-world benchmarks, while performing worse than non-diffusion methods on PSNR, SSIM, and LPIPS.
- Qualitative Comparisons: StableSR outperforms competing methods on real-world images beyond 512×512, producing more vivid details and fewer artifacts.The comparisons demonstrate adaptation beyond the training-sized resolution.
- Qualitative Comparisons: StableSR restores sharper edges, faithful details, and fewer artifacts than competing methods on 128 →512 real-world comparisons.Other methods are described as producing blurry or unnatural details, whereas StableSR generates realistic details and sharp edges.
- User Study: StableSR receives over 80% of votes from 35 subjects evaluating 40 real-world images against Real-ESRGAN+, SwinIR-GAN, and LDM.The study totals 1,400 votes and indicates potential for real-world super-resolution applications.
- Comparison with Concurrent Diffusion Applications: StableSR uses a 105M-parameter time-aware encoder and runs 50 sampling steps in 10.37s, compared with ControlNet-tile’s 364M and 14.47s.The comparison attributes the difference to StableSR avoiding copied layers from the fixed diffusion prior.
- Comparison with Follow-up Approaches: StableSR is comparable with DiffBIR on real-world benchmarks, avoids DiffBIR’s tendency to generate overly repeated patterns, and can outperform SUPIR on smaller images.SUPIR is reported to perform poorly below 512 after upsampling, while it outperforms StableSR on some larger-resolution cases.
- Ablation Study: Training from scratch requires 2.06 times more GPU memory than StableSR on NVIDIA Tesla 32G-V100 GPUs.StableSR also achieves better LPIPS scores than the matched training-from-scratch comparison during training.
- Ablation Study: SFT layers slightly improve validation LPIPS scores during training, while the time-aware encoder and diffusion prior are reported as crucial for performance improvements.The SFT ablation sets α_n = 0 and β_n = 0 in Eq.(1).
Importance of Time-aware Guidance and Color
Time-aware guidance and color correction improve StableSR’s restoration quality, while CFW exposes a controllable realism–fidelity trade-off. These components address complementary visual-quality and fidelity concerns.
- Ablations: Removing time-aware guidance or color correction worsens SSIM and LPIPS on RealSR and DRealSR benchmarks.Visual comparisons likewise show blurrier textures without time-aware guidance and color shifts without color correction.
- Fidelity–realism trade-off: CFW uses coefficient w to trade realism against fidelity: smaller values favor realistic outputs, whereas larger values improve fidelity.The supported range is w ∈ [0, 1].
- Color correction: Wavelet color correction further improves visual quality beyond pixel-domain channel matching in cases with color shifts.Channel matching alleviates shifts, while the wavelet formulation provides an additional visual-quality improvement.
- Fidelity–realism trade-off: Larger w values such as 0.75 achieve higher PSNR and SSIM, while w = 0.0 achieves higher CLIP-IQA and MUSIQ scores.The reported metrics indicate a fidelity–perceptual-quality trade-off across three paired benchmarks.
- Practical consideration: CFW requires extra GPU memory and is therefore designated as an optional feature for different applications.Its optional status follows from the resource cost of enabling the realism–fidelity control.
5 Inference Strategies
StableSR’s inference strategies improve visual quality or reduce sampling cost without additional finetuning. Negative-prompt guidance sharpens results, while advanced samplers and SD-turbo substantially accelerate generation.
- Overview: StableSR already achieves superior quantitative and qualitative performance on synthetic and real-world benchmarks.The section presents additional sampling strategies intended to further improve inference performance without finetuning.
- Classifier-free guidance with negative prompts: Higher classifier-free guidance scales with negative prompts increase CLIP-IQA and MUSIQ scores, although s = 7.5 can cause oversharpening.The prompt experiments compare multiple positive and negative prompts and guidance strengths.
- Sampling efficiency: StableSR’s default DDPM sampler uses 200 steps, making inference time-consuming compared with non-diffusion approaches.The complexity comparison evaluates 4x SR from 128 × 128 inputs on an NVIDIA Tesla 32G-V100 GPU.
- Sampling efficiency: DDIM produces faithful details in 20 steps, and SD-turbo generates high-quality results in 4 steps without further finetuning.These samplers reduce the number of diffusion iterations while retaining reported image quality.
- Sampling efficiency: SD-turbo reduces inference time to 0.83s, 6.3 times faster than LDM with 200 steps, while outperforming cited GAN-based methods and LDM.Directly accelerating LDM with DDIM instead causes a severe performance drop in the visual comparison.
- Limitations: StableSR may fail on texts, very small patterns, and small faces because it shares limitations with the Stable Diffusion prior.The paper suggests stronger diffusion priors and larger training datasets as possible future remedies.
6 Limitations
StableSR inherits diffusion-prior limitations and may struggle with small texts, faces, and patterns. The paper identifies stronger priors and more high-quality training data as future directions.
- Limitations: StableSR may struggle with small texts, faces, and patterns, despite these cases also challenging existing generic super-resolution methods.The limitation is attributed to its shared diffusion prior.
- Future work: The paper leaves stronger diffusion priors and training on more high-quality data as future work for these difficult cases.This is presented as a possible way to improve handling of the identified limitations.
7 Conclusion
The paper presents StableSR as a way to adapt pretrained diffusion priors for real-world super-resolution without degradation assumptions or source-intensive training from scratch. Its modules target computational cost, fidelity, and fixed-resolution constraints, and extensive experiments support the approach.
- Conclusion: StableSR exploits pretrained diffusion models for real-world super-resolution while avoiding explicit degradation assumptions and training from scratch.The conclusion frames diffusion-prior adaptation for super-resolution as an underexplored problem.
- Conclusion: The time-aware encoder, controllable feature wrapping module, and progressive aggregation sampling scheme address high computational cost and fixed resolution.The paper describes these as respective solutions to key real-world SR challenges.
- Conclusion: Extensive experiments evaluate StableSR, and additional inference strategies are provided to facilitate practical applications.The conclusion presents the method as a foundation for future work on diffusion-prior super-resolution.
A Details of Time-aware Encoder
The time-aware encoder follows the contracting path of Stable Diffusion’s denoising U-Net while using substantially fewer parameters. Its detailed configuration is provided in Table 7.
- The time-aware encoder resembles the contracting path of Stable Diffusion’s denoising U-Net.
- Approximately 105M parameters, including SFT layers, are used by reducing the number of channels.
- Table 7 lists the detailed settings of the time-aware encoder.
B Aggregation Sampling
StableSR’s aggregation sampling strategy enables arbitrary-size image generation by combining overlapping latent patches during diffusion. Boundary overlap is dynamically adjusted to support flexible target resolutions.
- Aggregation sampling enables arbitrary-size image generation without a perceptible performance drop.
- The overlapping size at the right and bottom boundaries is dynamically adjusted to fit the target resolution.
- StableSR may produce suboptimal results under severe input degradations, motivating an auxiliary pre-cleaning step during sampling.
C Pre-cleaning for Severe Degradations
StableSR can yield suboptimal results for severely blurred or noisy low-resolution inputs. An auxiliary pre-cleaning phase uses existing super-resolution or face-restoration models before StableSR in these cases.
- Severe blur or noise in low-resolution images can lead StableSR to produce suboptimal results.
- The auxiliary pre-cleaning phase precedes StableSR to address severe degradations.
- Real-ESRGAN+ is used for general super-resolution, while CodeFormer is used for face super-resolution.
D Additional Visual Results
Additional real-world qualitative comparisons evaluate StableSR at fixed 512×512 output resolution and at original arbitrary resolutions. The reported results show finer details, sharper edges, high-quality images beyond 4× resolution, and realistic textures across complex scenes.
- D.1 Visual Results on Fixed Resolution: At 512×512 output resolution from 128×128 low-resolution inputs, StableSR produces finer details and sharper edges than state-of-the-art methods.
- D.2 Visual Results on Arbitrary Resolution: At original image resolutions, StableSR generates high-quality super-resolution images beyond 4× resolution.
- D.2 Visual Results on Arbitrary Resolution: StableSR generates realistic textures for diverse real-world scenes such as buildings and text.