PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Review of Deep Learning-based Single Image Super-resolution

Syed Muhammad Arsalan Bashir, Yi Wang, Mahrukh Khan, Yilong Niu

arXiv:2102.09351v3cs.CVcs.LGeess.IV

TL;DR

Image super-resolution seeks to recover high-resolution images from low-resolution inputs despite an inherently ill-posed degradation process. This survey synthesizes classical and deep-learning methods, evaluation concepts, datasets, applications, and challenges. It highlights deep-learning progress while identifying unresolved issues in realistic degradation and perceptual evaluation.

  • Problem

    Super-resolution must recover high-resolution images from low-resolution inputs even though degradation is often unknown and multiple high-resolution images may explain the same observation.

  • Method

    The survey reviews classical, supervised, unsupervised, and application-oriented SR research, covering methods, datasets, metrics, structural components, and future directions.

  • Results

    The survey reports substantial progress in deep-learning-based SR and reviews recent methods and their reported achievements relative to classical approaches.

  • Takeaways & Limitations

    The review provides a consolidated reference for understanding SR methods, evaluation practices, applications, and open research problems.

  • Takeaways & Limitations

    PSNR and SSIM remain widely used but do not completely capture perceptual image quality, while opinion scoring is impractical for large datasets.

Abstract

from arXiv · show

Image super-resolution (SR) is one of the vital image processing methods that improve the resolution of an image in the field of computer vision. In the last two decades, significant progress has been made in the field of super-resolution, especially by utilizing deep learning methods. This survey is an effort to provide a detailed survey of recent progress in single-image super-resolution in the perspective of deep learning while also informing about the initial classical methods used for image super-resolution. The survey classifies the image SR methods into four categories, i.e., classical methods, supervised learning-based methods, unsupervised learning-based methods, and domain-specific SR methods. We also introduce the problem of SR to provide intuition about image quality metrics, available reference datasets, and SR challenges. Deep learning-based approaches of SR are evaluated using a reference dataset. Some of the reviewed state-of-the-art image SR methods include the enhanced deep SR network (EDSR), cycle-in-cycle GAN (CinCGAN), multiscale residual network (MSRN), meta residual dense network (Meta-RDN), recurrent back-projection network (RBPN), second-order attention network (SAN), SR feedback network (SRFBN) and the wavelet-based residual attention network (WRAN). Finally, this survey is concluded with future directions and trends in SR and open problems in SR to be addressed by the researchers.

1 INTRODUCTION

Single-image super-resolution reconstructs high-resolution images from low-resolution inputs, but the task is ill-posed because multiple high-resolution images can produce similar observations. This survey reviews classical and deep-learning approaches, evaluation practices, applications, and open challenges.

  • 1 INTRODUCTION: Super-resolution is fundamentally ill-posed because downsampling is many-to-one, leaving multiple high-resolution images consistent with one low-resolution input.Variations may involve camera angle, color, brightness, and other image factors.
  • 1 INTRODUCTION: Classical SR methods include statistical, prediction-based, patch-based, edge-based, and sparse-representation approaches, while deep learning has become increasingly used as computational power and data expanded.The survey states that deep-learning SR studies have reported superior performance to classical methods over the past decade.
  • 1 INTRODUCTION: The survey emphasizes recent deep-learning-based SR while also reviewing classical methods, problem definitions, datasets, performance evaluation, applications, and future directions.Its key features include a detailed deep-learning survey, structural comparisons of methods, and discussion of open problems.
  • 1 INTRODUCTION: SR models commonly represent low-resolution images as degraded high-resolution images and learn an approximate inverse degradation function, although real degradation factors are often unknown.Degradation may involve scaling, blur, noise, sensor artifacts, compression, motion blur, and misfocus; bicubic downsampling is frequently used in studies.
  • 1 INTRODUCTION: More realistic degradation models incorporate blur kernels and additive white Gaussian noise rather than relying only on simple downsampling.The survey presents this richer formulation as closer to actual degradation because it considers more parameters.
  • 1 INTRODUCTION: SR quality is commonly assessed with objective metrics such as PSNR and SSIM, but neither completely captures perceptual quality.PSNR can encourage smooth, non-realistic surfaces, while SSIM considers textures, structures, brightness, and contrast; opinion scoring is resource-intensive for large datasets.

2 CONVENTIONAL METHODS OF SUPER-RESOLUTION

Conventional super-resolution methods span prediction-, patch-, edge-, statistical-, and sparse-representation approaches. These methods use filtering, local textures, image priors, learned mappings, and robust or multi-image information to reconstruct higher-resolution images.

  • Classical SR methods comprise prediction-based, edge-based, statistical, patch-based, and sparse-representation approaches.
  • Prediction-based methods: Prediction-based methods used Lanczos and frequency-domain filtering, while cubic convolution outperformed nearest-neighbor prediction and linear interpolation in reported resampling results.
  • Patch-based methods: Patch-based methods extracted training patches or exploited local patch features and recurring geometric similarities to generate high-frequency details.
  • Edge-based methods: Edge-based methods applied smoothness or gradient priors and local self-similarity filters to reconstruct realistic edges, but performance was weaker in clustered regions.
  • Statistical and other methods: Statistical and robust methods learned image mappings or used outlier information, while other approaches used a single reference image or multiple LR images for reconstruction.One post-processing model reportedly supported resolution enhancement up to 100x using a single reference image.

3 SUPERVISED SUPER-RESOLUTION

Supervised deep-learning SR combines specialized upsampling, loss functions, normalization, and iterative or progressive frameworks to reconstruct detailed HR images. These designs balance perceptual quality, scalability, computational efficiency, and reconstruction stability.

  • 3.1 UPSAMPLING METHODS: Meta upscaling supports arbitrary scaling factors within one model and reportedly surpasses fixed-factor models, while weight prediction takes 100 times less time than feature extraction.At larger magnifications, independently predicting convolution weights for each pixel may become unstable.
  • 3.3 LEARNING STRATEGIES: Loss design affects the trade-off between fidelity and realism: pixel loss can yield overly variable or smooth outputs, whereas adversarial and texture losses improve perceptual realism but may reduce PSNR or introduce patch-dependent artifacts.Adversarial-loss models can score lower on PSNR while scoring highly on perceptual opinion measures; texture-loss results depend on patch size.
  • 3.3 LEARNING STRATEGIES: Batch normalization stabilizes and accelerates deep CNN training by reducing internal covariate shift, addressing vanishing gradients, and permitting higher learning rates.It normalizes every mini-batch and adds two parameters per channel to preserve representational ability.
  • 3.4 SR FRAMEWORKS: Pre-upsampling simplifies learning by enlarging LR images before feature extraction, but it can amplify blur and noise while increasing computational and memory requirements.Post-upsampling instead learns mappings in low-dimensional space to reduce memory requirements and improve computational efficiency.
  • 3.4 SR FRAMEWORKS: Iterative up-and-down sampling refines images through recursive back-propagation, while SRFBN and RBPN apply this strategy to image and video SR, respectively.The framework has shown significant improvement over other frameworks, although its back-propagation modules require further exploration.
  • 3.4 SR FRAMEWORKS: Progressive upsampling decomposes enlargement into smaller stages, supports multiple scaling factors, and can improve learning rates, but training stability and convergence remain concerns.LapSRN is cited as an example using cascaded CNN modules for gradual scaling.

4 UNSUPERVISED SUPER-RESOLUTION

Unsupervised super-resolution avoids requiring paired LR-HR images by learning real-world degradation or mappings from unpaired data. Reviewed approaches include cyclic weak supervision, independently trained degradation and reconstruction networks, and zero-shot learning, each with practical limitations.

  • 4 UNSUPERVISED SUPER-RESOLUTION: Unpaired LR and HR images enable unsupervised SR models to learn real-world degradation without predefined LR-HR correspondences.Supervised methods typically rely on LR images generated through known degradations, whereas unsupervised models learn from unpaired datasets.
  • 4.1 WEAKLY-SUPERVISED SUPER-RESOLUTION: Cycle-in-cycle SR recursively learns LR-to-HR and HR-to-LR mappings, using distribution matching and cyclic predictions between two unpaired datasets.The framework employs four generators and two discriminators for degraded LR-to-LR and LR-to-HR representations.
  • 4.1 WEAKLY-SUPERVISED SUPER-RESOLUTION: Cycle-in-cycle SR achieves comparable results on very noisy images, where classical supervised degradation functions cannot be used, but its computational cost remains a research concern.
  • 4.1.2 LEARNING THE DEGRADATION FUNCTION: Degradation-function learning separates HR-to-LR degradation modeling from LR-to-HR reconstruction, training the two networks independently in a two-stage GAN framework.
  • 4.2 ZERO-SHOT SUPER-RESOLUTION: Zero-shot SR trains a network for each test image using augmentation to learn its degradation, but this per-image training makes inference substantially slower.The learned degradation kernel generates scaled and augmented images before an SRCNN learns their representations.
  • 4.3 OTHER UNSUPERVISED METHODS: A randomly initialized CNN can serve as an image prior by optimizing an HR output whose downsampling resembles the input LR image.

5 DOMAIN-SPECIFIC APPLICATIONS OF SUPER-RESOLUTION

Domain-specific SR adapts reconstruction methods to application-specific structure, motion, sensing, and evaluation conditions. Reviewed applications span faces, depth maps, remote sensing, video, medical imaging, and other computer-vision tasks.

  • Face hallucination: Face hallucination uses facial priors, parsing maps, landmarks, correspondence fields, and GANs to constrain reconstructed images toward facial structure or attributes.
  • Real-world image SR: Real-world SR requires dedicated datasets because camera sensors and raw-to-RGB conversion introduce degradations before images are used as references.The survey notes SR-RAW and related efforts to develop real-world image datasets.
  • Depth-map SR: Depth-map SR increases low-resolution depth resolution, commonly combining RGB-derived features or energy minimization with CNN-based depth upsampling.Multiple cameras can provide HR scene information while separate networks downsample RGB images and upsample depth maps.
  • Remote sensing and satellite imaging: Remote sensing and satellite SR use multi-line cameras and multiple LR observations, while conventional evaluation may not remain valid under satellite-specific degradation and operating conditions.
  • Video SR: Video SR exploits interframe and intraframe dependencies through optical flow, spatial transformers, subpixel modules, and recurrent networks for motion compensation and reconstruction.
  • Medical imaging: Medical SR methods reconstruct higher-resolution MR images, including thin-sliced knee and diffusion-parameter images from lower-resolution acquisitions.
  • Other applications: Other applications include object detection and stereo image SR, underscoring the method’s use across medical science, computer vision, satellite imaging, and remote sensing.

6 CONCLUSION AND FUTURE DIRECTIONS

The survey concludes that deep-learning SR has advanced substantially, but important open problems remain in evaluation, learning objectives, architecture, upsampling, unsupervised learning, and practical deployment.

  • 6 CONCLUSION AND FUTURE DIRECTIONS: No standard loss-function selection criterion exists across SR frameworks, making alternative and combined losses an open research direction.The survey also highlights alternatives to batch normalization as a research direction.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: SR network design requires further exploration because architecture directly influences performance and current upsampling methods have distinct drawbacks.Reported issues include checkerboard artifacts, non-uniform receptive fields, stability problems, and lack of end-to-end learning.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: Reducing computational cost while preserving optimal SR performance remains a priority for practical network architectures.The survey notes that current deep networks improve performance at the expense of higher computational cost.
  • 6.3 EVALUATION METRICS: PSNR and SSIM do not completely capture perceptual image quality, while opinion scoring is impractical for large datasets.The survey identifies development of a universal SR quality metric as a future direction.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: Unsupervised SR remains open because matched-resolution scenes are difficult to collect and bicubic interpolation is used to create unpaired datasets.These methods learn the inverse mapping of the interpolation rather than directly learning the actual SR process.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: Domain-specific SR applications, including medical imaging and graphics or video rendering, are identified as promising research directions.The survey frames these applications as opportunities for generating high-resolution images in other domains.

I C , IL contract and Luminance of image I

This section lists notation, loss terms, image-quality quantities, and acronyms used throughout the SR survey.

  • I C , IL contract and Luminance of image I: The notation includes luminance, contrast, and structure comparison functions used in image-quality assessment.These components correspond to the comparison terms used for SSIM-related measurements.
  • I C , IL contract and Luminance of image I: The listed image-quality symbols include high-level layer representations, an image-classification network, maximum pixel value, pixel count, stability constants, and SSIM control parameters.The entries define these symbols as notation used in the survey's quality and feature formulations.
  • I C , IL contract and Luminance of image I: The loss-function entries cover cross-entropy and least-square adversarial losses, L1 and L2 pixel losses, Gram-matrix texture loss, and total-variation loss.These terms represent the principal GAN, pixel, texture, and regularization losses listed in the supplied notation.
  • I C , IL contract and Luminance of image I: The acronym list includes SR architectures and evaluation terms such as DRRN, EDSR, ESRGAN, FID, FSIM, and FSRCNN.The supplied acronym table is identified as the list of acronyms for the survey.
Loading 2102.09351v3…