Source-linked AI summary
ResDiff: Combining CNN and Diffusion Model for Image Super-Resolution
Shuyao Shang, Zhengyang Shan, Guangxing Liu, LunQian Wang, XingHua Wang, Zekai Zhang, Jinglin Zhang
TL;DR
SISR must recover HR images from ambiguous LR inputs while reconstructing lost high-frequency detail. ResDiff combines a CNN for primary restoration with residual diffusion and frequency-domain guidance, achieving faster convergence, higher generation quality, and more diverse samples than prior diffusion-based methods. Its reported limitations include computationally costly wavelet operations, under-trained color discrepancies, and lower performance than current SOTA methods.
Problem
SISR is ill-posed because degradation removes high-frequency components and multiple HR images can correspond to the same LR image, while GAN-based approaches can lack diversity through pattern collapse.
Method
ResDiff uses a pre-trained CNN to restore primary image components, predicts the residual to ground truth with diffusion, and adds frequency-domain losses and guided diffusion for high-frequency details.
Results
Experiments on multiple datasets show that ResDiff accelerates convergence, improves image generation quality, and produces more diverse samples than prior diffusion-based methods.
Takeaways & Limitations
Residual and frequency-domain guidance let the diffusion model focus on high-frequency reconstruction while leveraging CNN prediction information beyond LR-space conditioning.
Takeaways & Limitations
ResDiff remains computationally burdened by time-consuming DWT operations and does not outperform current SOTA SISR methods under its reported model-parameter setting.
Abstract
from arXiv · showhide
Adapting the Diffusion Probabilistic Model (DPM) for direct image super-resolution is wasteful, given that a simple Convolutional Neural Network (CNN) can recover the main low-frequency content. Therefore, we present ResDiff, a novel Diffusion Probabilistic Model based on Residual structure for Single Image Super-Resolution (SISR). ResDiff utilizes a combination of a CNN, which restores primary low-frequency components, and a DPM, which predicts the residual between the ground-truth image and the CNN predicted image. In contrast to the common diffusion-based methods that directly use LR images to guide the noise towards HR space, ResDiff utilizes the CNN's initial prediction to direct the noise towards the residual space between HR space and CNN-predicted space, which not only accelerates the generation process but also acquires superior sample quality. Additionally, a frequency-domain-based loss function for CNN is introduced to facilitate its restoration, and a frequency-domain guided diffusion is designed for DPM on behalf of predicting high-frequency details. The extensive experiments on multiple benchmark datasets demonstrate that ResDiff outperforms previous diffusion based methods in terms of shorter model convergence time, superior generation quality, and more diverse samples.
Introduction
SISR is ill-posed because LR images lose high-frequency content and can correspond to multiple HR images. ResDiff combines CNN prediction with residual diffusion and frequency-domain guidance to improve convergence, image quality, and sample diversity.
- SISR is ill-posed because degradation removes high-frequency components and multiple HR images can produce the same LR image.
- Existing diffusion methods make the model recover both low- and high-frequency content, prolonging convergence and potentially missing texture details.
- ResDiff uses a pre-trained CNN to restore major low-frequency components and guides diffusion toward the residual space between the CNN prediction and ground truth.
- Frequency-Domain-guided Diffusion additionally uses high-frequency information to guide generation toward richer fine-grained details.
- Experiments on FFHQ, CelebA, Div2k, and Urban100 show faster convergence and more fine-grained images.
- ResDiff produces more diverse samples, reflected by a lower perceptual-based evaluation value.
Related Works
Prior SISR methods use GANs, flow-based models, or diffusion, each offering different quality, diversity, stability, or computational trade-offs. ResDiff’s inference combines a CNN initialization with iterative diffusion residual prediction.
- Generative SISR methods include GAN-based, flow-based, and diffusion-based approaches.
- GAN-based methods can generate sharp textures but are prone to mode collapse and challenging training.
- Flow-based methods model the conditional HR distribution with invertible transformations, avoiding training instability but requiring higher training costs and providing lower perceptual quality.
- ResDiff inference first computes a CNN prediction, iteratively denoises a residual initialized from Gaussian noise, and returns the residual added to the CNN output.
- Diffusion methods produce sharper and more detailed images but require high computational cost because of multiple forward and backward steps.
The Proposed ResDiff
ResDiff combines a pre-trained CNN for low-frequency restoration with frequency-domain-guided diffusion for residual and high-frequency reconstruction. Its CNN uses spatial and frequency losses, while the diffusion model separates frequency information and uses high-frequency-guided cross-attention.
- CNN pre-training: The pre-trained CNN generates an initial prediction that restores primary low-frequency and partial high-frequency content before diffusion predicts residual details.This reduces the diffusion model’s restoration burden and uses residual prediction between the CNN output and the ground truth.
- CNN pre-training: DWT decomposes images into LL, LH, HL, and HH sub-bands, with the latter three representing horizontal, vertical, and diagonal high-frequency components.The proposed DWT loss computes mean squared error between corresponding high-frequency sub-bands of ground-truth and predicted images.
- CNN pre-training: CNN pre-training combines spatial GT loss with FFT- and DWT-based frequency losses, weighted as LCNN = LGT + αLFFT + βLDWT.The DWT loss compares high-frequency wavelet sub-bands, while the spatial loss measures prediction error against the ground truth.
- FD-guided diffusion: The FD-Info-Splitter separates low- and high-frequency information from the CNN prediction and noisy image before U-net processing.The separated feature maps are concatenated so the network can use low-frequency information globally and high-frequency information for localized detail restoration.
- FD-guided diffusion: Adaptive Gaussian high-pass filtering and inverse FFT produce a high-frequency feature map, while ResSE processing supplies low-frequency information from the CNN prediction.The frequency-domain processing uses FFT features, adaptive filtering, inverse FFT, and learned attention weights to construct complementary feature maps.
- FD-guided diffusion: HF-guided cross-attention injects DWT-derived high-frequency information into U-net processing to improve fine-grained detail recovery.High-frequency coefficients from the CNN prediction form query features, while projected input features provide keys and values for cross-attention.
Experiments
ResDiff is evaluated against diffusion- and GAN-based methods on face and general-image datasets using distortion and perceptual metrics. It shows strong performance across these settings, with ablations supporting the residual structure and frequency-guided components.
- Evaluation setup: ResDiff is evaluated on FFHQ, CelebA, DIV2K, and Urban100 using PSNR, SSIM, and FID.The experiments cover both face and general datasets and use distortion-based and perceptual-based metrics.
- Face datasets: ResDiff outperforms all diffusion-based methods on FFHQ and CelebA and achieves about 50% lower FID than the GAN-based model.The comparison includes 4× and 8× super-resolution settings across the two face datasets.
- Qualitative results: On DIV2K 4× examples, ResDiff recovers richer details and more natural textures in small objects and difficult scenes.The examples include a clock, bridge structure, and building.
- General datasets: ResDiff significantly outperforms other diffusion-based methods on distortion metrics for the more difficult DIV2K and Urban100 datasets.The reported general-dataset results are evaluated at 4× super-resolution.
- Ablation study: Ablations show that residual structure improves performance even with bilinear initial prediction, while SimpleSR, FD Info Splitter, and HF-guided CA provide further gains.The study is conducted on FFHQ at 4× super-resolution.
Conclusion and Future Work
ResDiff combines a residual diffusion structure with frequency-domain guidance to improve image generation and convergence in SISR. The authors identify computational cost, under-training color discrepancies, and a gap to current SOTA methods as directions for future work.
- Conclusion: ResDiff combines feature-richer CNN guidance, a frequency-domain CNN loss, and frequency-domain guided diffusion for SISR.The method uses the CNN prediction to guide diffusion toward residual information.
- Conclusion: Experiments across different datasets show that ResDiff accelerates training convergence and provides superior image generation quality.This is the paper's reported overall experimental conclusion.
- Future work: DWT operations remain time-consuming, motivating future optimization despite ResDiff's faster convergence.The limitation concerns computational cost within the proposed pipeline.
- Future work: ResDiff does not outperform current SOTA SISR methods because of a disparity between model parameters.The authors leave adopting a larger U-net model for future work due to equipment limitations.