Source-linked AI summary
Designing a Practical Degradation Model for Deep Blind Image Super-Resolution
Kai Zhang, Jingyun Liang, Luc Van Gool, Radu Timofte
TL;DR
SISR methods can underperform when assumed degradations differ from those in real images. This paper proposes a randomly shuffled, practical degradation model, trains a deep blind super-resolver with its synthetic data, and reports favorable performance on diverse synthetic and real images.
Problem
Real-image SISR lacks a sufficiently general degradation model because existing deep blind methods remain mismatched with diverse real degradations.
Method
The paper designs a degradation model that combines expanded blur, downsampling, and noise choices with randomly shuffled degradation order.
Results
Experiments on synthetic and real image datasets show favorable performance on images corrupted by diverse degradations.
Takeaways & Limitations
The degradation model provides a way to train deep blind super-resolution models for real-image applications.
Takeaways & Limitations
The degradation model is mainly intended for synthesizing training data and is unsuited to modeling degraded LR images because it uses many parameters and random shuffling.
Abstract
from arXiv · showhide
It is widely acknowledged that single image super-resolution (SISR) methods would not perform well if the assumed degradation model deviates from those in real images. Although several degradation models take additional factors into consideration, such as blur, they are still not effective enough to cover the diverse degradations of real images. To address this issue, this paper proposes to design a more complex but practical degradation model that consists of randomly shuffled blur, downsampling and noise degradations. Specifically, the blur is approximated by two convolutions with isotropic and anisotropic Gaussian kernels; the downsampling is randomly chosen from nearest, bilinear and bicubic interpolations; the noise is synthesized by adding Gaussian noise with different noise levels, adopting JPEG compression with different quality factors, and generating processed camera sensor noise via reverse-forward camera image signal processing (ISP) pipeline model and RAW image noise model. To verify the effectiveness of the new degradation model, we have trained a deep blind ESRGAN super-resolver and then applied it to super-resolve both synthetic and real images with diverse degradations. The experimental results demonstrate that the new degradation model can help to significantly improve the practicability of deep super-resolvers, thus providing a powerful alternative solution for real SISR applications.
1. Introduction
Real-image SISR remains difficult because common degradation assumptions do not cover diverse blur, downsampling, and noise. The paper therefore designs a practical shuffled degradation model and trains a blind SISR model on its synthetic data.
- Motivation: Real images often defeat SISR methods because bicubic degradation and simple degradation models poorly represent their diverse degradations.Existing methods can also fail on JPEG-compressed images and other real-image conditions.
- Proposed degradation model: The proposed model expands blur, downsampling, and noise into more practical variants and randomly shuffles their application order.It uses isotropic and anisotropic Gaussian blur, common interpolation operators, multiple noise types, and adjustable parameters.
- Training strategy: The model synthesizes varied realistic LR images from HR images, enabling end-to-end supervised training of a deep blind SISR model.This process can generate unlimited paired data without the misalignment issue of unpaired training data.
- Outcome: The trained blind model performs well on real images under diverse degradations.The paper presents this result as evidence that the degradation model improves the practicality of deep SISR.
- Contribution: The work claims to be the first hand-designed degradation model for general blind image super-resolution.Its stated contribution is a practical alternative focused on real-image applications.
2. Related Work
Prior blind SISR methods estimate degradations, jointly restore and estimate them, learn from captured pairs, or use unpaired data, but their assumed degradation spaces remain mismatched with real images. This motivates a practical degradation model for general real-image super-resolution.
- Existing degradation models: Existing deep SISR methods commonly rely on bicubic, traditional, or simple degradation models, while real-image noise and degradation orders are more varied.Traditional models often assume AWGN and a fixed blur-downsampling-noise sequence.
- Blind SISR directions: One blind-SISR direction estimates degradation parameters before applying a non-blind method, but errors in blur estimation can produce over-sharp or over-smooth results.The cited approach estimates the blur kernel before super-resolution.
- Blind SISR directions: A second direction jointly estimates the blur kernel and HR image through iterative, optimization-based, or alternating network procedures.These methods combine kernel estimation with HR restoration rather than separating the tasks completely.
- Blind SISR directions: Supervised learning from captured real LR/HR pairs is constrained by the difficulty of collecting abundant well-aligned data and by the captured LR domain.The resulting models are limited to the degradation conditions represented in those pairs.
- Remaining gap: Unpaired-data methods learn degradation mappings or extract degradation information to generate training pairs, but existing blind models remain tied to limited or ideal degradation spaces.The paper identifies a remaining mismatch between assumed and real degradation models and calls for a practical model for real applications.
3. A Practical Degradation Model
The model expands blur, downsampling, and noise into more realistic degradation spaces, then randomly shuffles these operations to synthesize diverse LR images.
- Design rationale: The model enlarges the degradation space by varying blur, downsampling, and noise, while allowing their operations to occur in different orders.It treats blur, downsampling, and noise as key factors and uses random shuffling to cover more realistic combinations.
- Blur: Blur is modeled through isotropic and anisotropic Gaussian convolutions applied in HR or LR space, expanding the available blur configurations.The two blur operations can both be applied, and their placement reflects blur in either image space.
- Downsampling: Downsampling includes shifted nearest-neighbor, bicubic, bilinear, and down-up sampling, with the four methods uniformly sampled.The shifted Gaussian prefilter addresses nearest-neighbor misalignment, while down-up sampling can introduce LR-space blur.
- Noise: Noise modeling combines Gaussian noise, JPEG compression, and processed camera sensor noise to represent multiple real-image noise sources.JPEG quality factors are uniformly sampled from [30, 95], and Gaussian noise uses several covariance structures and levels.
- Random shuffle: Randomly arranging degradations allows bicubic and traditional models to appear as special cases while creating noise behaviors associated with real images.Downsampling can reduce signal dependence in some noise types, whereas down-up sampling can make Gaussian noise signal-dependent.
4. Discussion
The degradation model is primarily intended to synthesize paired training data, not to identify the degradation process of a given LR image.
- Discussion: The model can generate unlimited perfectly aligned LR/HR training pairs, avoiding the limited-data and misalignment issues of collected or unpaired data.Its broad random parameterization is useful for training a deep blind super-resolver, although it is not well suited to modeling one observed LR image.
- Discussion: The model may generate degradation cases that rarely occur in real-world scenarios, but the paper expects this to improve the trained model’s generalization ability.The discussion also notes that a large-capacity DNN can handle different degradations with one model.
5. Deep Blind SISR Model Training
The paper trains blind super-resolution models by pairing the proposed degradation model with ESRGAN-based architectures and tailored training data and losses.
- Model training: The method borrows ESRGAN network structures to train deep blind models on synthetic LR/HR pairs generated by the proposed degradation model.The paper trains a PSNR-oriented BSRNet followed by a perceptual-quality-oriented BSRGAN.
- Training data: BSRGAN uses DIV2K, Flick2K, WED, and 2,000 FFHQ face images to provide a broader image prior for general-purpose blind super-resolution.Blurry training images are removed using the variance of the image Laplacian.
- Optimization: BSRGAN uses 72 × 72 LR patches and minimizes weighted L1, VGG perceptual, and spectral norm-based least-square PatchGAN losses.The loss weights are 1, 1, and 0.1, respectively.
6. Experimental Results
The evaluation uses synthetic DIV2K4D and real RealSRSet images with diverse degradations to compare blind super-resolution methods. BSRNet and BSRGAN achieve the strongest overall synthetic results, while real-image evaluation exposes limitations of no-reference metrics and GAN artifacts.
- Testing datasets: DIV2K4D contains 400 synthetic images from 100 DIV2K validation images across four degradation types, while RealSRSet contains 20 real images.The datasets are designed to evaluate blind SISR under diverse synthetic and real degradations.
- Compared methods: The comparison includes bicubic-trained, kernel-estimation, JPEG-focused, DPED-focused, and proposed BSRNet and BSRGAN models.Most comparison methods use the same network architecture as the proposed models, isolating the degradation-model contribution.
- DIV2K4D results: BSRNet achieves the best overall PSNR results, while BSRGAN achieves the best overall LPIPS results on DIV2K4D.RRDB and ESRGAN perform well on bicubic degradation but degrade on non-bicubic settings; IKC also drops sharply on degradation types III and IV.
- DIV2K4D results: On a degradation type IV example, BSRNet and BSRGAN produce better visual results than competing methods.IKC and RealSR-JPEG fail to remove noise and recover sharp edges, whereas FSSR-JPEG introduces artifacts despite producing sharp images.
- RealSRSet results: On RealSRSet, BSRNet produces better visual results despite BSRGAN failing to show promising no-reference IQA results.BSRNet removes processed camera sensor noise and complex noise while recovering sharp edges and fine details, but BSRGAN tends to produce bubble artifacts in texture regions.
7. Conclusions
The paper designs a practical degradation model with more intricate blur, downsampling, and noise factors plus random shuffling, then uses synthetic data from it to train a deep blind super-resolution model. Experiments on synthetic and real image datasets show favorable performance.
- The paper designs a new degradation model for training a deep blind super-resolution model.
- More intricate blur, downsampling, and noise factors, combined with random shuffling, enable coverage of diverse real-world degradations.
- Synthetic data generated by the new degradation model supports training a deep blind model for general image super-resolution.
- Experiments on synthetic and real image datasets show that the deep blind model performs favorably.