PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Image Super-Resolution: A Survey and Beyond

Anran Liu, Yihao Liu, Jinjin Gu, Yu Qiao, Chao Dong

arXiv:2107.03055v1cs.CV

TL;DR

Blind image super-resolution remains challenging because real-world low-resolution images often violate the fixed degradation assumptions used by existing methods. This paper surveys blind SR, introduces a taxonomy based on degradation modelling and available data, and finds that methods perform well within modeled degradations but generalize poorly beyond them.

  • Problem

    Blind SR must recover high-resolution images despite real-world degradations that differ from predefined degradation assumptions, creating a mismatch between models and inputs.

  • Method

    The paper systematically surveys blind SR and classifies methods by explicit or implicit degradation modelling and whether they use external data or a single LR image.

  • Results

    Methods generally perform well on degradations covered by their models, while single-image methods may fail without recurring image statistics and GAN-based methods face difficult domain adaptation and generalization.

  • Takeaways & Limitations

    The taxonomy organizes existing blind SR methods by application scope and supports more reasonable comparisons and method selection across scenarios.

  • Takeaways & Limitations

    Explicitly modeled methods cannot produce satisfactory results for degradations outside their modeling scope, limiting their suitability for complex real-world images.

Abstract

from arXiv · show

Blind image super-resolution (SR), aiming to super-resolve low-resolution images with unknown degradation, has attracted increasing attention due to its significance in promoting real-world applications. Many novel and effective solutions have been proposed recently, especially with the powerful deep learning techniques. Despite years of efforts, it still remains as a challenging research problem. This paper serves as a systematic review on recent progress in blind image SR, and proposes a taxonomy to categorize existing methods into three different classes according to their ways of degradation modelling and the data used for solving the SR model. This taxonomy helps summarize and distinguish among existing methods. We hope to provide insights into current research states, as well as to reveal novel research directions worth exploring. In addition, we make a summary on commonly used datasets and previous competitions related to blind image SR. Last but not least, a comparison among different methods is provided with detailed analysis on their merits and demerits using both synthetic and real testing images.

1 INTRODUCTION

The paper surveys blind image super-resolution to address the difficulty of choosing effective methods when real-world images violate assumed degradations. It proposes a taxonomy that organizes existing approaches, clarifies research gaps, and guides fair comparisons and future directions.

  • Motivation: Real-world images often violate SISR input assumptions, making it difficult to choose an appropriate method or obtain high-quality results.Although deep learning has advanced SISR, existing methods may not provide a satisfactory viewing experience on real-world images.
  • Contributions: The paper provides a systematic survey of recent blind SR methods, including their improvements and limitations.The survey is intended to clarify the current research state and remaining gaps.
  • Contributions: Its taxonomy categorizes existing approaches, assigns methods positions within a broader picture, and reveals research gaps.This organization is intended to support reasonable and fair comparisons between different method types in future work.
  • Scope and organization: The paper offers insights into the current research state and promising future directions for blind SR.The paper’s structure introduces SR formulations and real-world challenges before presenting the taxonomy and reviewing non-blind and blind SR methods.

2 PROBLEM FORMULATION

Single-image super-resolution reconstructs high-resolution images from low-resolution inputs by modeling and inverting an unknown degradation process. Blind SR addresses degradation mismatch through explicit degradation modeling or implicit modeling from data distributions, but real-world variability remains a major challenge.

  • 2 PROBLEM FORMULATION: SISR reconstructs an HR image, particularly its high-frequency contents, from an LR input by modeling and solving the inverse degradation function f^-1.The degradation maps HR image x to LR image y with scale factor s.
  • 2 PROBLEM FORMULATION: Non-blind SR typically assumes bicubic downsampling or fixed Gaussian blur, so degradation mismatch can cause severe artifacts for other LR inputs,.Applying a model trained for a predefined degradation to inputs with different intrinsic degradation creates a domain gap from the desired HR image.
  • 2 PROBLEM FORMULATION: Blind SR models unknown degradation either explicitly, using blur kernels, additive noise, and possibly JPEG compression, or implicitly through external data distributions.Explicit approaches extend the classical degradation model, while implicit approaches avoid specifying the degradation function directly.
  • 2 PROBLEM FORMULATION: Real-world degradations are more complex than explicit combinations of modeled factors, and methods tuned to a particular scene or camera can fail on other degradation types,.Existing approaches have addressed only a limited set of real-world images despite often claiming real-world applicability.

3 CHALLENGES FROM REAL-WORLD IMAGES

Real-world images exhibit diverse degradations because imaging devices and image histories differ, while compression and aging introduce additional artifacts. Existing work often targets only one image type, limiting performance across diverse scenes and motivating type-specific or general solutions.

  • Different imaging devices: Real-world images vary substantially across imaging devices, whose differing characteristics create distinct degradation patterns.The passage contrasts DSLR and smartphone cameras as examples of devices with different image quality and visual characteristics.
  • Compression and aging: Compression causes blurring and blocky artifacts, while aging produces film grain, sepia effects, and color fading.These degradations can result from poor imaging equipment, erosion in the air, and the deterioration of old photos or films.
  • Research limitations and goals: Because prior work often focuses on a single real-image type, its performance is limited in diverse scenes; future research should address each type or all types jointly.The stated goal is either effective solutions for distinct image types or a general solution covering real-world images broadly.

4 TAXONOMY

The paper classifies blind super-resolution methods by degradation modelling—explicit or implicit—and by the data used—external datasets or a single input image. This taxonomy guides the review and clarifies method assumptions and remaining research directions.

  • Explicit modelling: Explicit modelling follows the classical degradation model or variants, using external data to learn an SR model across parameterized degradations such as k and n.Representative methods include SRMD, IKC, and KMSR.
  • Single-input methods: A separate class exploits internal patch-recurrence statistics and can operate directly on a single input image, as in KernelGAN and ZSSR.These approaches are primarily based on the classical degradation model.
  • Taxonomy: The proposed taxonomy separates blind SR methods by explicit versus implicit degradation modelling and by external training data versus a single input image.The categorization is presented as the organizing guideline for reviewing existing approaches.
  • Research directions: Distinguishing modelling assumptions helps explain what degradations each method assumes and motivates feasible solutions for general real-world images with diverse contents.The review uses the taxonomy to analyze limitations and inspire future work.

5 OVERVIEW OF NON-BLIND SINGLE-IMAGE SUPER-RESOLUTION

Non-blind single-image super-resolution assumes a fixed, known degradation and has progressed from example-based methods to deep CNN frameworks with improved accuracy and efficiency. However, these models often fail on blurry or noisy inputs that deviate from the assumed degradation, motivating blind SR.

  • Traditional non-blind SR: Non-blind SR assumes a fixed known degradation process, with traditional methods learning LR-to-HR mappings from external exemplar pairs or exploiting internal self-similarity.External-pair methods typically use compact dictionaries or manifold spaces.
  • Deep CNN-based SISR: The common CNN framework comprises shallow feature extraction, deep feature extraction or mapping, and SR output reconstruction, often aided by residual learning.These modules are depicted in the common non-blind SISR framework.
  • Deep CNN-based SISR: Residual blocks, recursive or recurrent structures, attention, sub-pixel convolution, and diverse loss functions have improved non-blind SISR reconstruction accuracy and efficiency.The cited developments target deep feature extraction, reconstruction, and perceptual quality.
  • Limitations: Non-blind models generalize poorly to complex degradations: they handle bicubicly downsampled clean inputs but struggle with blurry or noisy images.This limitation creates demand for blind SR methods, the survey’s main focus.

6 EXPLICIT DEGRADATION MODELLING

Explicit degradation-modelling blind SR methods use the classical degradation model and divide into external-dataset approaches or methods relying on a single input image. External-dataset methods condition SR reconstruction on estimated degradation, while kernel mismatch and uncovered degradations remain major limitations.

  • Overview: Explicit degradation-modelling methods follow the classical degradation model and split into external-dataset approaches versus single-input approaches.The classification depends on the data used to solve the SR problem.
  • Image-Specific Adaptation with External Dataset: External-dataset methods train CNN-based SR models on varied blur kernels and noises, using estimated degradation as conditional input for feature adaptation.They perform satisfactorily when the input degradation types are covered by the training dataset.
  • Image-Specific Adaptation without Kernel Estimation: SRMD concatenates the LR image with a degradation map, while UDVD adds per-pixel dynamic convolution to handle spatially varying degradations.SRMD encodes blur kernels through PCA-based kernel coding and noise-level estimation; UDVD iteratively refines the SR output with dynamic blocks.
  • Image-Specific Adaptation without Kernel Estimation: MAP-based methods such as DPSR and USRNet avoid kernel coding by decoupling degradation-related subproblems, enabling handling of more complex kernels.DPSR uses half-quadratic splitting, while USRNet unfolds the corresponding iterative optimization into an end-to-end trainable network.
  • Image-Specific Adaptation with Kernel Estimation: Kernel-estimation approaches incorporate estimation into SR, but inaccurate kernels cause mismatch and greatly undermine performance, while explicit models fail on degradations outside their training coverage.These methods can avoid manually searching for kernel estimates during inference, yet they remain unable to produce satisfactory results for uncovered degradations.

7 IMPLICIT DEGRADATION MODELLING

Implicit degradation modelling learns real-world degradation distributions from external HR/LR data, using supervised, unpaired, or GAN-based domain-adaptation strategies. Despite its flexibility, reliance on large datasets and GAN artifacts limits practical use and motivates single-image approaches with human-provided priors.

  • Approach: Implicit modelling learns degradation through external datasets, using paired supervision or unpaired HR and LR domains for domain adaptation.Paired data can achieve satisfactory results with carefully designed SR networks, as shown by top solutions in NTIRE 2018 and AIM 2020.
  • Approach: Existing methods transform LR inputs toward clean bicubic domains, learn HR-to-LR degradation generators, or jointly train with generated and real LR samples.CinCGAN uses bicubic-domain adaptation before non-blind SR, Degradation GAN learns degradation and SR jointly, and DASR addresses domain gaps with mixed training and weighted supervision.
  • Limitations: GAN-based implicit modelling can produce severe artifacts and fake textures, while generated LR images may remain mismatched with real LR distributions.These issues can undermine SR training and harm real-world applications; Fig.17 illustrates the artifact limitation for FSSR.
  • Limitations: The approach is data-hungry and therefore unsuitable for tasks such as old-photo restoration, where large external datasets may be unavailable or insufficient.Complex cases including surveillance video, old photos, and films may not be covered by a few external datasets because their degradations are unpredictable.
  • Future directions: A proposed research direction is single-image implicit modelling for complex outlier degradations, supplemented by human intervention to provide effective SR priors.Possible interventions include modulation-based control of restoration outputs or manually selecting a clear, similarly contented reference image.

8 DATASETS AND COMPETITIONS

Blind SR datasets are dominated by synthetic LR generation because real paired HR-LR data are difficult to obtain, although several carefully captured real-world datasets now exist. Competitions have progressively introduced more complex unknown and camera-based degradations to evaluate blind SR methods.

  • 8.1 Datasets: Most blind SR methods synthesize LR inputs from HR images because real paired HR-LR data remain difficult to obtain.Explicit degradation-modelling methods commonly require HR-LR pairs for training and evaluation.
  • 8.1.1 Synthetic datasets: DIV2K and Flickr2K commonly support training, while Set5, Set14, BSD100, and Urban100 are typically used for testing.Blind SR also uses the standard HR datasets from non-blind SR; DIV2KRK further simulates complex degradation with random anisotropic Gaussian blur and multiplicative noise.
  • 8.1.2 Real-world datasets: City100, DRealSR, and RealSR provide paired real-world HR-LR data captured with imaging devices and carefully aligned observations.DRealSR is the largest among these datasets, with around 800 image pairs for each scale factor.
  • 8.2 Competitions: Blind SR competitions progressed from unknown blur and decimation in NTIRE 2017 to more complex degradations, including those produced by digital cameras.These competitions are used to assess and promote state-of-the-art solutions, with paired training data shaping the focus of leading methods.

9 QUANTITATIVE COMPARISON

The survey compares representative blind SR methods using published quantitative results and released pretrained models, while emphasizing that inaccessible code and differing training data prevent fully fair comparison. Results show that degradation coverage affects synthetic performance, whereas real-world images remain challenging for all methods.

  • Comparison challenges: Fair comparison is hindered by unavailable code, missing implementation details, and differences in training datasets and degradation types that affect performance and generalization.The survey therefore calls for a uniform, comprehensive benchmark, while presenting only published quantitative results and examples from released pretrained models.
  • Quantitative results: Published tables provide within-paper comparisons for explicit kernel modelling methods, including SRMD, ZSSR, NPBSR, KernelGAN, IKC, and DAN under synthetic degradations.Table 2 uses DIV2KRK with randomly generated anisotropic Gaussian kernels, while Table 3 uses Set14 and BSD100 with selected isotropic Gaussian kernels.
  • Quantitative results: For implicit modelling, Table 4 compares CinCGAN, FSSR, and DASR using PSNR, SSIM, and LPIPS on AIM and RealSR real-world datasets.The table’s comparison is reported from DASR, while ESRGAN is noted as trained with paired data.
  • Qualitative observations: External-data methods generalize according to degradation coverage: explicit models handle noise only when noise is included in their degradation modelling, otherwise producing artifacts.SRMD and USRNet can address noisy inputs when noise is modelled, whereas SRResNet and IKC cannot remove uncovered noise effectively.
  • Qualitative observations: Real-world images contain degradations that diverge from synthetic distributions, and none of the compared methods produces satisfactory results on the tested real image.SRResNet and IKC retain noise and create artifacts, while SRMD and USRNet reduce noise artifacts at the cost of high-frequency detail.
  • Future comparison guidance: The taxonomy is recommended as a framework for placing new methods and selecting comparable prior approaches within the same degradation-modelling category.Methods outside a category require special attention because they use different degradation modelling or data.

10 CONCLUSION

The paper surveys blind image super-resolution and organizes existing methods by degradation modelling and the data used to solve the SR model. It concludes that each established category has important limitations and identifies implicit modelling from a single image as a promising future direction.

  • 10 CONCLUSION: The proposed taxonomy combines explicit or implicit degradation modelling with either external datasets or a single LR image, yielding three represented categories and one unexplored combination.Implicit modelling with a single image has no existing representative approach in the survey.
  • 10 CONCLUSION: Explicit modelling with external datasets performs well on modelled degradations but deteriorates severely on degradations outside that coverage.SRMD and IKC use classical degradation models or variants for image-specific adaptation based on degradation information.
  • 10 CONCLUSION: Explicit modelling with a single LR image exploits cross-scale patch recurrence but may fail on diverse or monotonous scenes lacking sufficient recurring clues.Representative methods include NPBSR and KernelGAN for blind kernel estimation, and ZSSR and DGDML-SR for super-resolution.
  • 10 CONCLUSION: Implicit modelling with external datasets learns complex real-world degradations through data-distribution learning under GANs, but domain adaptation limits performance and generalization.CinCGAN, FSSR, and DASR represent this category, whose training is difficult because the natural-image domain is large.
  • 10 CONCLUSION: Implicit modelling with a single image is proposed as a future direction for complex degradations in general images without strong internal statistics.Possible solutions include human-provided SR priors, modulation in the restoration network, or manually selecting an HR reference image.
Loading 2107.03055v1…