PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Single Image Super-Resolution: A Brief Review

Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, Jing-Hao Xue

arXiv:1808.03344v3cs.CV

TL;DR

SISR seeks to recover high-resolution images from single low-resolution inputs despite severe ill-posedness. This survey organizes deep learning methods by architecture and optimization objectives, reviews representative solutions and comparisons, and identifies challenges including vague application-specific assessment criteria.

  • Problem

    SISR is ill-posed because one low-resolution input can correspond to many high-resolution images, while the LR-to-HR mapping and high-dimensional computation remain difficult.

  • Method

    The survey categorizes DL-based SISR research into efficient neural architectures and effective optimization objectives, establishing benchmarks and analyzing representative works.

  • Results

    Deep architectures and optimization-objective developments have produced promising DL-based SISR results, with performance generally improving as model depth and parameters grow before leveling off.

  • Takeaways & Limitations

    The review highlights lightweight and sparse architectures as practically meaningful because they can retain relatively good performance with less storage and computation.

Abstract

from arXiv · show

Single image super-resolution (SISR) is a notoriously challenging ill-posed problem, which aims to obtain a high-resolution (HR) output from one of its low-resolution (LR) versions. To solve the SISR problem, recently powerful deep learning algorithms have been employed and achieved the state-of-the-art performance. In this survey, we review representative deep learning-based SISR methods, and group them into two categories according to their major contributions to two essential aspects of SISR: the exploration of efficient neural network architectures for SISR, and the development of effective optimization objectives for deep SISR learning. For each category, a baseline is firstly established and several critical limitations of the baseline are summarized. Then representative works on overcoming these limitations are presented based on their original contents as well as our critical understandings and analyses, and relevant comparisons are conducted from a variety of perspectives. Finally we conclude this review with some vital current challenges and future trends in SISR leveraging deep learning algorithms.

I. INTRODUCTION

SISR reconstructs a high-resolution image from a single low-resolution observation, an ill-posed task with ambiguous solutions and difficult LR–HR mappings. This survey reviews deep learning approaches through neural architectures and optimization objectives, alongside background and representative comparisons.

  • Problem: SISR is ill-posed because one low-resolution input can correspond to many possible high-resolution images.The desired high-resolution natural-image space is usually intractable.
  • Survey scope: The survey organizes recent DL-based SISR research around efficient neural architectures and effective optimization objectives.It establishes benchmarks, discusses representative contributions and experiments, and reviews challenges and trends.
  • Prior methods: Traditional interpolation is fast but inaccurate, while reconstruction methods can produce sharp details yet become costly and degrade at larger scale factors.These trade-offs motivate learning-based methods.
  • Learning-based methods: Learning-based SISR models exploit statistical relationships between low- and high-resolution examples for fast computation and strong reconstruction performance.Earlier approaches include Markov random fields, neighbor embedding, sparse coding, and random forests.
  • Deep learning background: Deep learning automatically learns hierarchical representations rather than relying on handcrafted features selected with expert knowledge.Its progress has been supported by efficient computing hardware and sophisticated algorithms.

III. DEEP ARCHITECTURES FOR SISR

The survey uses SRCNN as a benchmark for examining efficient SISR architectures and identifies input, complexity, and SISR-prior limitations. Subsequent designs address direct LR processing and more efficient resolution increases through deconvolution and subpixel convolution.

  • SRCNN benchmark: SRCNN is a three-layer CNN whose transformations perform patch extraction, nonlinear mapping, and reconstruction.Its filters have sizes 64 × 1 × 9 × 9, 32 × 64 × 5 × 5, and 1 × 32 × 5 × 5.
  • SRCNN benchmark: SRCNN approximates the complex LR-to-HR mapping end to end and reportedly outperformed concurrent traditional methods.The survey attributes this success to CNN representation learning from large datasets.
  • SRCNN limitations: Using bicubic LR input can smooth details, add computation, and become unreasonable when the downsampling kernel is unknown.These limitations motivate architectures that directly accept LR inputs.
  • Architecture challenges: The survey asks whether deeper, wider, or differently connected CNNs can improve results and whether SISR properties can be integrated into network design.The latter question includes large-scale-factor and unknown-downsampling settings.
  • Learning effective upsampling: FSRCNN uses a deconvolution layer to reconstruct HR images from LR feature maps, reducing computation by increasing resolution near the network output.The layer combines interpolation with a stride-1 convolution.
  • Learning effective upsampling: ESPCN replaces redundant nearest-neighbor feature repetition with zero-padded subpixel interpolation, yielding a more efficient upsampling operation.The figure contrasts the zero-interpolation view with the original ESPCN process.

2) The Deeper, The Better:

Deep SISR architectures improve reconstruction through depth, residual and dense connections, multiscale sharing, progressive generation, reconstruction priors, and internal examples.

  • The Deeper, The Better:: VDSR uses a 20-layer VGG-net, residual learning, and joint training across scale factors to improve convergence and exploit relationships among scales.It learns the residual between bicubic input and HR output rather than directly mapping bicubic input to HR.
  • The Deeper, The Better:: Residual and dense skip-connected architectures enable deeper SISR models, with SRDenseNet concatenating features across blocks before deconvolution.ResNet supports feature re-usage, whereas DenseNet supports new feature exploration.
  • The Deeper, The Better:: EDSR removes batch normalization, increases feature width with residual scaling, and uses cross-scale pretraining; MDSR shares nonlinear-mapping kernels across scales.Only scale-specific front feature extraction and final subpixel upsampling convolutions differ in MDSR.
  • Combining Properties of the SISR Process with the Design of the CNN Frame:: Other architectures incorporate SISR-specific procedures, including DBPN-style iterative backprojection, ZSSR internal examples, progressive models, and adaptive model ensembling.IDBP followed by ZSSR combines reconstruction-based initialization with internal-example training and has achieved robust performance in reported cases.

C. Comparisons among Different Models and Discussion

Model comparisons evaluate reconstruction quality, storage, and computation while showing that larger models generally perform better, with diminishing returns and strong sensitivity to degradation mismatch.

  • Comparisons among Different Models and Discussion: The comparison uses PSNR/SSIM for reconstruction quality, parameter count for storage efficiency, and Mult&Adds for computational efficiency.Mult&Adds is computed for a desired 720p SR output.
  • Comparisons among Different Models and Discussion: Training data substantially influences final performance, with common training sources including the 291 dataset, ImageNet-derived images, and DIV2K.The review notes that more abundant training data usually leads to better results.
  • Comparisons among Different Models and Discussion: As depth and parameter count grow, performance generally improves, but the improvement rate levels off; lightweight and sparse models therefore offer practical efficiency benefits.The review identifies reduced storage and computation as meaningful in practice.
  • Comparisons among Different Models and Discussion: Models trained for specific degradation can lose performance drastically when the true degradation differs from the training assumption.Under Gaussian-kernel degradation, EDSR trained with bicubic degradation showed obvious blur.

A. Benchmark of Optimization Objectives for DL-based SISR

The benchmark frames MSE as a Gaussian-noise regression objective nested within maximum likelihood and KLD, motivating objectives that relax these assumptions for SISR.

  • A. Benchmark of Optimization Objectives for DL-based SISR: MSE is selected as the benchmark because it favors high PSNR and can be interpreted as regression producing a point estimate.The benchmark follows SRCNN’s MSE loss.
  • A. Benchmark of Optimization Objectives for DL-based SISR: Under independent Gaussian white noise, maximum likelihood estimation yields the MSE objective for a CNN mapping training inputs to outputs.The conditional output distribution is Gaussian with mean F(x; θ) and diagonal covariance σ2I.
  • A. Benchmark of Optimization Objectives for DL-based SISR: The forward KLD between conditional empirical and model distributions is equivalent to maximum likelihood when estimated from training samples.The distributions represent HR conditioned on LR and SR conditioned on LR, respectively.
  • A. Benchmark of Optimization Objectives for DL-based SISR: The review establishes the hierarchy MSE as a special case of MLE and MLE as a special case of KLD.This relationship motivates examining alternatives when the underlying assumptions may not hold.
  • A. Benchmark of Optimization Objectives for DL-based SISR: The benchmark raises whether Gaussian noise and parametric-distribution assumptions are violated or misspecified in complicated SISR scenes.It also asks how other probability-measure distances and objective properties should guide objective selection.

B. Objective Functions Based on non-Gaussian Additive Noises

This section reviews replacing Gaussian additive-noise assumptions with other distributions and their corresponding robust losses for SISR. MAE is presented as more robust than MSE, while these distributions may not precisely represent unknown noise.

  • 1) Denote Additive Noise with Other Probability Distributions:: Directly optimizing MSE produces poor perceptual quality, motivating alternative noise distributions or transformed spaces.The survey identifies Gaussian additive noise in HR space as insufficient for perceptual quality.
  • 1) Denote Additive Noise with Other Probability Distributions:: MAE models Laplacian additive noise and is considered more robust against outliers than MSE.The conditional distribution is p(y|x) = Laplace(y; F(x; θ), bI).
  • 1) Denote Additive Noise with Other Probability Distributions:: MAE optimization is reported to converge faster and produce better results than MSE optimization in neural networks.The authors suggest that MAE may guide the network toward a better local minimum.
  • 1) Denote Additive Noise with Other Probability Distributions:: Robust statistical losses are widely used in DL-based SISR despite their distributions often failing to represent unknown additive noise precisely.Their appeal is attributed to conciseness and advantages over MSE.

2) Using MSE in a Transformed Space:

This section transforms the HR space into a feature space where MSE can better emphasize perceptually relevant information. Perceptual-loss methods improve visual quality, and end-to-end VGG-16 training accelerates inference without affecting restoration quality, although the approach lacks theoretical analysis.

  • 2) Using MSE in a Transformed Space:: Perceptual loss applies MSE after mapping HR and LR-related representations into a transformed feature space.The mappings Φ and Ψ connect the original spaces to transformed ones, with Ψ represented by deep architectures.
  • 2) Using MSE in a Transformed Space:: Johnson et al. directly optimize an SISR network with VGG-16 feature-space MSE instead of using nonlinear inference with Φ.The change reportedly preserves restoration quality while accelerating the overall process.
  • 2) Using MSE in a Transformed Space:: Perceptual loss mitigates blurring and produces more visually pleasing results than HR-space MSE.The survey states that the reason this approach works has not been theoretically analyzed.
  • 2) Using MSE in a Transformed Space:: Feature-space MSE emphasizes human-relevant information while omitting small pixel variations and trivial content.Deep supervised features are described as perceptually stable and discriminative.

C. Optimizing Forward KLD with Nonparametric Estimation

This section connects contextual loss and IMLE to nonparametric estimation and forward KLD, using KDE to avoid assuming a parametric data distribution. The resulting objectives can simplify toward contextual loss, but KDE remains computationally expensive.

  • C. Optimizing Forward KLD with Nonparametric Estimation: Contextual loss and IMLE use nonparametric estimation to model distributions without specifying a parametric form.Nonparametric methods are presented as robust when the real distributional form is unknown.
  • C. Optimizing Forward KLD with Nonparametric Estimation: As bandwidth h approaches 0, the affinity becomes approximately an indicator, allowing the ℓ1 objective to approximate the ℓ∞ norm and degenerate into contextual loss.The affinity is near 1 for matching samples and near 0 otherwise.
  • C. Optimizing Forward KLD with Nonparametric Estimation: Minimizing IMLE is shown to equal minimizing an upper bound of forward KLD with KDE.The derivation uses a Gaussian kernel and rewrites the objective through successive bounds and simplifications.
  • C. Optimizing Forward KLD with Nonparametric Estimation: Contextual loss and IMLE report visually pleasing results, but KDE is generally very time-consuming.The survey notes that approximations and acceleration algorithms are therefore applied.

D. Other Distances between Probability Measures Used in SISR

This section explains how different probability distances behave under inadequate distributional models. Forward KLD tends toward regression to the mean, backward KLD concentrates on a dominant mode, and JSD or GAN-based optimization is introduced to address practical optimization difficulties.

  • D. Other Distances between Probability Measures Used in SISR: When a two-mode GMM is modeled by one Gaussian, forward KLD places the solution between modes while backward KLD approaches the most prominent mode.Both divergences reach zero when model and data distributions match, but differ under inadequate solutions.
  • D. Other Distances between Probability Measures Used in SISR: Under inadequate solutions, forward KLD causes regression to the mean, whereas backward KLD concentrates on the main modality.The survey associates regression to the mean with blurring and says backward KLD may improve visual quality while collapsing to patterns.
  • D. Other Distances between Probability Measures Used in SISR: Backward KLD is impractical for optimizing deep architectures in low-level vision because Pdata is empirical and Pmodel is intractable.The survey replaces asymmetric KLD with symmetric Jensen-Shannon divergence to relieve optimization difficulties.
  • D. Other Distances between Probability Measures Used in SISR: GANs implicitly optimize distributional objectives through adversarial training, avoiding approximate inference and partition-function gradient approximation.The generator and discriminator update alternately until the discriminator can no longer provide useful information.

E. Characters of Different objective functions

The survey distinguishes distortion-aimed losses from perception-aimed losses and emphasizes their inherent perception–distortion trade-off. Objective functions should therefore be selected according to the application context rather than treated as universally suitable.

  • Distortion-aimed losses measure dissimilarity between training pairs, whereas perception-aimed losses measure similarity between source and target distributions.
  • The perception–distortion curve indicates that improving one objective must generally come at the expense of the other.
  • Quantitative comparisons combine a nonreference quality metric with RMSE to evaluate perception and distortion across representative methods.
  • Improving perceptual quality can reduce PSNR and SSIM, even when visual quality improves.
  • There is no one-fits-all objective function, so the loss should match the application context.

V. TRENDS AND CHALLENGES

The survey identifies deployment efficiency, theoretical understanding, and application-specific assessment criteria as major unresolved challenges for deep-learning SISR. It also highlights large-scale and unknown-corruption settings as demanding problems requiring more effective solutions.

  • Challenges: Massive parameter counts and computation make advanced SISR models difficult to deploy in real-world scenarios.The survey calls for lighter or slimmed models with little or no performance degradation.
  • Challenges: Large-scale SISR and SISR with unknown corruption still lack very effective remedies despite broad gains on traditional tasks.
  • Challenges: Deep SISR architectures remain black boxes because their learned representations are not yet well understood.The survey argues that further theoretical exploration should address why and how these models work.
  • Challenges: Application-specific objectives remain vague because requirements often lack explicit, precise assessment definitions.The survey recommends clearer criteria for designing targeted objectives and comparing algorithms more rationally.
  • Conclusion: The review organizes future challenges around accelerating deep models, comprehending them extensively, and establishing criteria for objective-function design and evaluation.
Loading 1808.03344v3…