PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-World Single Image Super-Resolution: A Brief Review

Honggang Chen, Xiaohai He, Linbo Qing, Yuanyuan Wu, Chao Ren, Ce Zhu

arXiv:2103.02368v1eess.IVcs.CV

TL;DR

Real-world SISR is difficult because synthetic degradation does not represent the complexity of realistic LR observations. This review organizes RSISR datasets, metrics, and four method categories, compares representative algorithms, and identifies unresolved research challenges.

  • Problem

    Synthetic-data results can overestimate SISR performance because the domain gap between synthetic and realistic degradation reduces real-world reconstruction accuracy.

  • Method

    The review synthesizes RSISR datasets, assessment metrics, four method categories, benchmark comparisons, and current challenges and future directions.

  • Results

    Representative RSISR methods are compared on synthetic and realistic benchmark datasets in terms of reconstruction quality and computational efficiency.

  • Takeaways & Limitations

    RSISR still needs more suitable realistic datasets, specialized real-world SR models, and evaluation criteria.

Abstract

from arXiv · show

Single image super-resolution (SISR), which aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) observation, has been an active research topic in the area of image processing in recent decades. Particularly, deep learning-based super-resolution (SR) approaches have drawn much attention and have greatly improved the reconstruction performance on synthetic data. Recent studies show that simulation results on synthetic data usually overestimate the capacity to super-resolve real-world images. In this context, more and more researchers devote themselves to develop SR approaches for realistic images. This article aims to make a comprehensive review on real-world single image super-resolution (RSISR). More specifically, this review covers the critical publically available datasets and assessment metrics for RSISR, and four major categories of RSISR methods, namely the degradation modeling-based RSISR, image pairs-based RSISR, domain translation-based RSISR, and self-learning-based RSISR. Comparisons are also made among representative RSISR methods on benchmark datasets, in terms of both reconstruction quality and computational efficiency. Besides, we discuss challenges and promising research topics on RSISR.

I. INTRODUCTION

Super-resolution reconstructs HR images from LR observations, offering a flexible alternative to hardware upgrades. This review focuses on RSISR because synthetic-data results overestimate performance on real-world images.

  • I. INTRODUCTION: Super-resolution enhances existing LR images through signal processing, unlike costly and inflexible hardware upgrades that capture only new HR images.Its outputs can support segmentation, detection, and recognition.
  • I. INTRODUCTION: SISR reconstructs an HR image from one LR observation, whereas MISR fuses complementary information from multiple correlated images.ISR also includes multi-frame image super-resolution, while VSR targets video spatial resolution or frame rate.
  • I. INTRODUCTION: Synthetic-data evaluations can overestimate SISR ability because the domain gap between synthetic and realistic data causes performance to degrade on real-world images.This degradation hinders practical SISR applications.
  • I. INTRODUCTION: The review is the first overview devoted to RSISR and covers datasets, assessment metrics, method taxonomy, benchmark comparisons, challenges, and future directions.It compares representative algorithms in reconstruction accuracy and computational efficiency.
  • I. INTRODUCTION: Simulated degradation commonly combines blurring, downsampling, and noise injection, or directly uses bicubic downscaling to create LR training and testing images.These methods assume a degradation process that may not match real-world image formation.

III. DATASETS AND ASSESSMENT METRICS

RSISR evaluation depends on datasets and metrics suited to realistic degradation. Conventional HR-only datasets require simulated LR generation and therefore do not fully match real-world conditions.

  • III. DATASETS AND ASSESSMENT METRICS: Training and testing datasets plus assessment metrics are described as cornerstones of SISR.The review introduces both resources for RSISR models.
  • III. DATASETS AND ASSESSMENT METRICS: Most widely used SISR datasets contain only HR images, so LR counterparts must be generated using an assumed degradation model such as bicubic downsampling.Their discrepancy from real-world degradation makes them unsuitable for RSISR study.
  • III. DATASETS AND ASSESSMENT METRICS: The review summarizes RSISR datasets including DIV2KRK, RealSR, DRealSR, City100, SR-RAW, TextZoom, SupER, and ImagePairs.These datasets are collected or constructed in different ways to represent realistic settings.
  • III. DATASETS AND ASSESSMENT METRICS: DIV2KRK is a synthetic blind-SR test dataset derived from DIV2K by blurring and downsampling 100 validation HR images with random kernels.Its random kernels are 11×11 anisotropic Gaussians.
  • III. DATASETS AND ASSESSMENT METRICS: RealSR contains 595 LR-HR pairs captured with Nikon D810 and Canon 5D3 cameras at four focal lengths for ×2, ×3, and ×4 super-resolution.Progressive registration addresses lens distortion and exposure differences before producing aligned pairs.

3) DRealSR [81]:

The reviewed datasets capture diverse real-world imaging conditions, including camera-dependent degradation, raw sensor inputs, and specialized text scenes. Their construction emphasizes aligned or annotated data for RSISR research.

  • 3) DRealSR [81]:: DRealSR uses five DSLR cameras to capture indoor and outdoor scenes at four resolutions, aligning images with SIFT.It contains 884, 783, and 840 LR-HR pairs for ×2, ×3, and ×4 SR, respectively.
  • 3) DRealSR [81]:: City100 characterizes resolution and field-of-view degradation for DSLR and smartphone cameras by changing focal length or shooting distance.Its NikonD5500 images use 55mm for HR and 18mm for LR capture, followed by alignment and intensity/color rectification.
  • 3) DRealSR [81]:: SR-RAW pairs RAW images captured at seven optical-zoom settings and uses raw sensor data to reconstruct HR RGB images.The reconstruction includes demosaicing, denoising, and super-resolution.
  • 3) DRealSR [81]:: TextZoom is the first real-scene text SR dataset, derived from RealSR and SR-RAW with annotated content, direction, focal length, and difficulty levels.Its easy, medium, and hard subsets support text SR and text recognition.
  • 3) DRealSR [81]:: SupER obtains more than 80,000 images from 14 lab scenes at four imaging resolutions and five compression levels using hardware binning.Changing the binning factor naturally guarantees LR-HR alignment.

7) SupER [79]:

The reviewed material highlights the need for realistic-image evaluation and contrasts subjective assessment with objective metrics for super-resolved images.

  • 7) SupER [79]:: ImagePairs contains 11,421 LR-HR image pairs captured simultaneously with 5MP and 20.1MP cameras.A beam-splitter cube enables simultaneous capture, while differing focal lengths produce different perspectives requiring pixel-wise alignment.
  • 7) SupER [79]:: ImagePairs also provides raw images that may support ISP and other tasks beyond super-resolution.
  • Assessment Metrics: Super-resolved image quality is assessed through subjective human perception or objective quality metrics.Subjective evaluation is direct and practically aligned, whereas objective evaluation is more convenient but may disagree with subjective judgments and other metrics.
  • Assessment Metrics: Subjective evaluation is affected by personal preferences and is costly and difficult to automate.
  • Assessment Metrics: PSNR is widely used for full-reference image-restoration assessment, but its pixel-wise emphasis can reduce consistency with perceptual quality.PSNR uses the mean squared error between the super-resolved and ground-truth images; for 8-bit images, the maximum pixel value is 255.

2) SSIM [135]:

SSIM evaluates full-reference image quality through luminance, contrast, and structural comparisons, and is reported to reflect visual quality better than PSNR.

  • 2) SSIM [135]:: SSIM is a full-reference metric that measures structural similarity between a ground-truth image and a super-resolved image.
  • 2) SSIM [135]:: SSIM jointly compares luminance, contrast, and structural characteristics.Its formulation uses means, standard deviations, covariance, weighting parameters, and stabilizing constants.
  • 2) SSIM [135]:: SSIM is reported to reflect visual quality better than PSNR.
  • 2) SSIM [135]:: PSNR and SSIM are generally combined when the corresponding ground-truth image is available.
  • 2) SSIM [135]:: LPIPS assesses reference-based image quality using l2 distances between reference and test images in deep feature space.The passage reports that LPIPS agrees well with human judgments.

5) NIQE [138]:

Real-world SISR research addresses the gap between synthetic degradation assumptions and complex real-world image degradation. The review organizes RSISR methods and highlights degradation modeling as a feasible solution.

  • Taxonomy: RSISR methods are grouped into degradation modeling-based, image pairs-based, domain translation-based, and self-learning-based approaches.
  • Degradation Modeling-based Methods: Degradation modeling methods estimate unknown blur and noise parameters before or during super-resolution reconstruction.Approaches may estimate degradation first or jointly optimize degradation parameters and the super-resolved image.
  • Degradation Modeling-based Methods: Realistic degradation modeling can use diverse blur kernels and noise distributions; one such framework won the NTIRE 2020 Challenge on Real-World Image Super-Resolution.
  • Degradation Modeling-based Methods: The reported performance of degradation modeling-based methods supports degradation modeling as a meaningful and feasible solution for real-world image super-resolution.

B. Image Pairs-based Methods

Image pairs-based RSISR methods construct realistic LR-HR training data by collecting different-resolution images of the same scene. They address pair misalignment and image-specific reconstruction challenges with specialized losses and architectures.

  • Pair Construction: Researchers collect same-scene images at different resolutions because explicit realistic degradation models are difficult to derive.These images provide realistic LR-HR pairs for RSISR model training.
  • Pair Alignment: Real-world image pairs can suffer unavoidable misalignment, which may introduce blurring artifacts when pixel-to-pixel losses are used.
  • Pair Alignment: CoBi combines pixel information, spatial coordinates, RGB patches, and pretrained perceptual features to improve robustness to mild pair misalignment.
  • Image-Specific Reconstruction: CDC separately predicts flat, edge, and corner components and aggregates them using learned component-attentive maps and a gradient-weighted loss.Results on RealSR and DRealSR support its reconstruction performance and generalization capability.
  • Image-Specific Reconstruction: TSRN uses bidirectional LSTM mechanisms for real scene text images, and its super-resolution increases recognition accuracy.

C. Domain Translation-based Methods

Domain translation-based RSISR addresses the lack of aligned real LR-HR pairs by translating between real-world LR and HR image domains. Two-stage methods use synthetic LR as an intermediate domain, whereas one-stage methods translate directly and generally reduce testing complexity.

  • Overview: Domain translation is used when only unpaired real LR and HR images are available, treating real LR, synthetic LR, and HR images as separate domains.
  • Two-Stage Methods: Two-stage approaches first translate real-world LR images toward a synthetic LR domain and then apply super-resolution.Their key distinction from one-stage methods is using synthetic LR as a relay domain.
  • Two-Stage Methods: CinCGAN uses unpaired training data to map realistic LR inputs to bicubic-like synthetic LR images before the subsequent super-resolution network.The translation can suppress artifacts such as noise in real-world LR inputs.
  • One-Stage Methods: One-stage frameworks produce super-resolved images directly from real-world LR inputs, making translation from the real-world LR domain to the HR domain the central challenge.
  • Comparison: Two-stage methods integrate synthetic-data-trained SR models more elegantly, while one-stage methods generally have lower testing-phase complexity.

D. Self-Learning-based Methods

Self-learning-based RSISR methods exploit internal information from the LR input to reduce dependence on mismatched external training data. ZSSR exemplifies this approach, while later meta-learning methods address its data and testing-time costs.

  • Self-learning methods use the LR input's internal information to learn an image-specific SR model, reducing the impact of training–testing discrepancy.
  • ZSSR trains an eight-layer CNN during testing on example pairs extracted from the LR image and its degraded version.The approach relies on cross-scale internal recurrence in natural images.
  • Self-supervised methods neglect external information and are usually time-consuming because they perform online training during testing.
  • MZSR introduces meta-transfer learning to find a generic initialization that adapts to new image conditions within a few gradient updates.Its pipeline comprises large-scale training, meta-transfer learning, and meta-test.

V. COMPARISONS AMONG STATE-OF-THE-ARTS

The review compares representative RSISR methods on DIV2KRK and RealSR at ×2 and ×4, examining visual quality, objective quality, and computational complexity. Results show clear practical trade-offs, while fair ranking remains difficult because evaluation settings are not standardized.

  • The comparison evaluates ZSSR, KernelGAN, MZSR, DBPI, DAN, IKC, and SRResCGAN on DIV2KRK and RealSR using ×2 or ×4 upsampling.RealSR testing uses 50 Canon images.
  • The reported comparisons are not intended to identify an accuracy or efficiency winner because test settings and hyperparameters differ and no universally accepted protocol exists.
  • SR produces more effective HR images than conventional interpolation, but reconstructed images still differ visibly from ground truth, especially in textures and edges.Some outputs show over-smoothing or over-sharpening artifacts.
  • Self-supervised SR methods generally use fewer parameters than models trained on large-scale datasets but take longer to produce upsampled images.
  • Subjective and objective assessments can differ, and scores from different objective metrics are not necessarily consistent.

VI. CURRENT CHALLENGES AND FUTURE DIRECTIONS

Despite progress in RSISR, the field still faces unresolved problems that motivate further research. The review frames these challenges as opportunities for developing more capable methods and evaluations.

  • The review identifies remaining RSISR problems and discusses corresponding challenges and promising future directions.

A. Image Datasets

RSISR progress depends strongly on realistic training data and evaluation criteria suited to real-world conditions. The review highlights shortages in datasets, alignment, no-reference assessment, and adaptation to varied degradations.

  • A. Image Datasets: Realistic datasets can substantially improve reconstruction performance, yet RSISR still lacks larger, more representative, and targeted datasets.
  • A. Image Datasets: More accurate alignment is needed for images of the same scene captured at different resolutions.
  • B. SR Algorithms: Real-world LR images can exhibit distinctly different degradations, requiring RSISR models to adapt to changing image conditions.
  • B. SR Algorithms: Paired training data may be difficult to obtain, motivating models that learn from unpaired data or the LR input alone.
  • C. Evaluation Criteria: PSNR and SSIM do not accurately measure visual quality and require HR references unavailable in practical applications.
  • C. Evaluation Criteria: RSISR needs task-specific and no-reference criteria that better assess region-specific goals such as texture detail, edge sharpness, and flat-area smoothness.
Loading 2103.02368v1…