PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Journey into Super-resolution: A survey

Saeed Anwar, Salman Khan, Nick Barnes

arXiv:1904.07523v3cs.CV

TL;DR

Single-image super-resolution remains challenging because it is ill-posed, becomes harder at larger scaling factors, and standard metrics only loosely reflect human perception. This survey reviews, taxonomizes, and systematically evaluates deep CNN methods across six datasets, finding rapid accuracy growth alongside greater model complexity and identifying open problems for practical deployment.

  • Problem

    Super-resolution is an ill-posed inverse problem with multiple possible solutions, increasing difficulty at larger scaling factors and imperfect correspondence between quantitative metrics and human perception.

  • Method

    The survey reviews deep neural networks for single natural-image super-resolution, introduces a structural taxonomy, and compares methods across architectures, complexity, learning details, and six public datasets.

  • Results

    Extensive comparisons show rapid accuracy growth in recent years, while pioneering benchmark methods have been significantly outperformed by current contenders.

  • Takeaways & Limitations

    GAN-based methods generally produce visually pleasing outputs, whereas reconstruction-error methods more accurately preserve spatial details.

  • Takeaways & Limitations

    Existing methods remain limited by insufficient degradation-specific priors, perceptual shortcomings of pixel-level objectives, and poor generalization from synthetic bicubic degradation to real low-resolution images.

Abstract

from arXiv · show

Deep convolutional networks based super-resolution is a fast-growing field with numerous practical applications. In this exposition, we extensively compare 30+ state-of-the-art super-resolution Convolutional Neural Networks (CNNs) over three classical and three recently introduced challenging datasets to benchmark single image super-resolution. We introduce a taxonomy for deep-learning based super-resolution networks that groups existing methods into nine categories including linear, residual, multi-branch, recursive, progressive, attention-based and adversarial designs. We also provide comparisons between the models in terms of network complexity, memory footprint, model input and output, learning details, the type of network losses and important architectural differences (e.g., depth, skip-connections, filters). The extensive evaluation performed, shows the consistent and rapid growth in the accuracy in the past few years along with a corresponding boost in model complexity and the availability of large-scale datasets. It is also observed that the pioneering methods identified as the benchmark have been significantly outperformed by the current contenders. Despite the progress in recent years, we identify several shortcomings of existing techniques and provide future research directions towards the solution of these open problems.

1 INTRODUCTION

Single-image super-resolution is an important but challenging computer-vision problem because it must recover uncertain details from limited observations. This survey reviews deep neural approaches, introduces a structural taxonomy, and evaluates their development and challenges.

  • Super-resolution supports improved scene and object details for displays, mobile devices, detection, recognition, medical imaging, remote sensing, astronomy, and forensics.
  • SR is ill-posed because one low-resolution image can correspond to multiple high-resolution solutions, making reliable prior information necessary.
  • Higher up-scaling factors make missing-detail recovery more complex and can produce incorrect information.
  • PSNR and SSIM only loosely correlate with human perception, complicating output-quality assessment.
  • The survey reviews deep neural networks for single natural-image super-resolution, proposes a structural taxonomy, and systematically evaluates methods on six public datasets.

2 BACKGROUND

The background formulates super-resolution as inversion of an unknown degradation process that maps high-resolution images to low-resolution observations. Common models represent blur, downsampling, and noise, while deep learning methods learn image priors from data.

  • The degradation formulation maps a high-resolution image x to a low-resolution image y through an unknown function Φ with degradation parameters θη.
  • In real-world settings, only y is available, so super-resolution estimates an approximation x̂ of the ground-truth image x using an inverse function.
  • Degradation can involve sensor or speckle noise, compression, defocus or motion blur, and other artifacts.
  • The commonly preferred observation model applies blur and downsampling to x, then adds additive white Gaussian noise.
  • The objective minimizes a data-fidelity term while balancing it against an image prior Ψ(·), and the survey focuses on deep neural networks that learn this prior.

3 SINGLE IMAGE SUPER-RESOLUTION

Deep single-image super-resolution networks are organized by distinctive architectural features, including linear, residual, and multi-stage designs. Early upsampling is straightforward but computationally expensive, whereas late upsampling improves efficiency by operating on low-resolution inputs or features.

  • Existing methods are categorized into nine groups according to distinctive model-design features, with linear networks representing the earliest simple designs.
  • Linear Networks: Linear networks use a single sequential path without skip connections or branches, differing mainly in whether upsampling occurs early or late.
  • Early Upsampling Designs: Early upsampling first enlarges the low-resolution input to the target size before hierarchical feature learning, as in SRCNN.
  • Model-Based Designs: IRCNN combines CNN denoisers with model-based optimization, but related denoising approaches depend heavily on accurate noise estimation and can be computationally expensive.
  • Late Upsampling Designs: Late upsampling learns from low-resolution inputs or features and postpones enlargement, reducing computational cost and memory footprint.
  • Late Upsampling Designs: FSRCNN uses feature extraction, shrinking, nonlinear mapping, and expansion layers, with expansion increasing performance by 0.3dB.
  • Late Upsampling Designs: ESPCN uses sub-pixel convolution and ℓ1 loss for efficient real-time image and video super-resolution.

3.2 Residual Networks

Residual networks use skip connections to support very deep super-resolution models by learning high-frequency residuals. They range from single-stage networks to multi-stage designs that progressively refine predictions.

  • Residual-network designs: Residual learning uses skip connections to mitigate vanishing gradients and make very deep super-resolution networks feasible.These models learn the high-frequency residue between the input and ground-truth images.
  • Single-stage residual networks: Single-stage residual networks use one network, with EDSR removing batch normalization and external ReLU activations while extending the design to multiple scales.MDSR shares most parameters across scales and applies scale-specific layers near the input and output.
  • Training and evaluation: EDSR and related models are trained with ℓ1 loss, whereas earlier examples in the section use MSE-based training and varied patch-generation procedures.The reported training setups differ in datasets, patch sizes, optimization schedules, and network depth.
  • Multi-stage residual networks: Multi-stage residual designs train successive subnets so early stages predict coarse features and later stages improve the predictions.The section also includes encoder-decoder designs with distinct downsampling and upsampling stages.
  • Multi-stage residual networks: FormResNet separates formatting and structured-region restoration, improving on DnCNN by a small margin, while BTSRN combines low- and high-resolution stages.BTSRN upsamples low-resolution-stage features using deconvolution followed by nearest-neighbor upsampling.

3.3 Recursive networks

Recursive and progressive networks reduce the difficulty or scale of super-resolution through repeated refinement. Parameter sharing can limit complexity, while progressive designs address large upscaling factors in stages.

  • Recursive networks: Recursive networks repeatedly apply convolutional layers or linked units to decompose the difficult super-resolution problem into simpler steps.The survey presents recursive architectures as multi-path designs with repeated processing.
  • Recursive networks: DRCN uses embedding, inference, and reconstruction subnetworks, with recursive convolution and ReLU expanding the receptive field at each recursion.Its shared layer is applied multiple times while the inference network produces high-resolution feature maps.
  • Recursive networks: DRRN reaches 52 convolutional layers while reducing complexity by factors of 14, 6, and 2 relative to REDNet, DRCN, and VDSR, respectively.Residual learning, local identity connections, and shared recursive blocks provide the stated complexity reductions.
  • Recursive networks: SRFBN recursively refines low-resolution inputs through feedback blocks that alternate high-resolution projection with low-resolution convolution.The model is trained with curriculum learning for multiple degradations and uses four recursive iterations with an ℓ1 objective.
  • Progressive networks: Progressive methods predict large-scale outputs in multiple steps, while LapSRN produces 2×, 4×, and 8× residuals through a pyramidal framework.Multi-scale LapSRN jointly handles multiple scales and outperforms three separate scale-specific models.

3.5 Densely Connected Networks

Densely connected super-resolution networks combine information across depth, residual pathways, and feedback mechanisms to build richer representations and improve reconstruction. The section covers dense, residual-dense, back-projection, multi-branch, and information-distillation designs.

  • Dense connections: Dense connections expose each layer to preceding outputs, supporting hierarchical feature reuse, compact models, and faster training.SRDenseNet uses deconvolution layers near the rear of the network for upscaling.
  • Residual dense networks: RDN combines dense connections with local and global residual skips so hierarchical features are fully used to learn local patterns.RDN uses ℓ1 loss rather than the ℓ2 loss used in SRDenseNet.
  • Back-projection networks: Dense deep back-projection networks alternate upsampling and downsampling while explicitly feeding residual signals into upsampled feature maps.This feedback mechanism directs the network toward fine details.
  • Multi-branch networks: Multi-branch networks process features at multiple context scales and fuse complementary information through multi-path signal flow.CNF fuses SRCNN subnetworks with different depths using sum-pooling, after training on 20 million luminance patches.
  • Multi-branch networks: CMSC uses cascaded merge-and-run blocks with parallel branches, but its reported performance lags behind EDSR and MDSR.Its intermediate and final outputs are combined with an ℓ1 loss and balancing term.
  • Frequency-aware branches: EBRN routes more complex high-frequency signals through deeper modules and fuses ten parallel branches recursively rather than by simple summation.Its training first uses ℓ1 loss and then fine-tunes with an ℓ2 objective.

3.7 Attention-based Networks

Attention-based networks assign different importance to spatial features or channels instead of treating all representations uniformly. The surveyed designs use selection, channel attention, Laplacian attention, and residual attention mechanisms.

  • Attention motivation: Attention-based models selectively emphasize features because spatial locations and channels do not contribute equally to super-resolution.The survey presents attention as a flexibility mechanism layered onto deep networks.
  • Selection attention: SelNet inserts gated selection units after convolutional layers, combining identity mapping with ReLU, 1×1 convolution, and sigmoid operations.The network contains 22 convolutional layers and selects values from feature maps before subsequent processing.
  • Channel attention: RCAN combines recursive residual pathways with channel attention that collapses h×w×c activations to 1×1×c channel descriptors.These mechanisms provide multiple information pathways and focus processing on more important feature maps.
  • Channel attention: RCAN outperforms IRCNN, VDSR, and RDN, but its computational complexity is approximately 15 times higher than a stated comparison baseline.The passage attributes improved convergence to its recursive residual architecture and identifies high computational complexity as a shortcoming.
  • Laplacian attention: DRLN combines modular hierarchical structure, dense residual units, cascading connections, and Laplacian attention with long-, medium-, and local-skip pathways.Its reported improvement is attributed to the Laplacian-attention and cascading module.
  • Residual attention: SRRAM evaluates multiple attention mechanisms within common SR architectures and uses residual attention modules for feature extraction, upscaling, and reconstruction.The final model uses 64 RAM blocks with 3×3 filters and 64-channel feature maps.

3.8 Multiple-degradation handling networks

Multiple-degradation handling networks address the unrealistic assumption of bicubic-only degradation by incorporating degradation information or adapting to each test image.

  • Real-world super-resolution may involve multiple simultaneous degradations, making bicubic degradation an infeasible assumption.
  • Zero-Shot Super-Resolution trains a simple network on a downsampled version of the test image using internal image statistics.
  • SRMD concatenates the low-resolution image with degradation maps to process inputs affected by multiple degradations.
  • SRMD uses 12 convolutional layers with 128 feature maps per layer, ℓ1 loss, 40×40 patches, and 5,944 training images.

3.9 GAN Models

GAN-based super-resolution targets perceptual realism alongside pixel accuracy, using adversarial and perceptual objectives to generate sharper, more natural textures. These methods can improve visual quality but may reduce PSNR or introduce artifacts.

  • 3.9.1 SRGAN: SRGAN combines MSE, perceptual, and adversarial losses to favor perceptually similar high-resolution outputs.
  • 3.9.1 SRGAN: SRGAN introduced Mean Opinion Score to measure perceptual quality through manual human ratings, outperforming competitors on that metric.
  • 3.9.2 EnhanceNet: EnhanceNet adds perceptual and adversarial losses to pixel-level MSE, producing more realistic and perceptually better outputs despite lower best PSNR.
  • 3.9.2 EnhanceNet: EnhanceNet’s adversarial training can create visible artifacts when super-resolving highly textured regions.
  • 3.9.4 ESRGAN: ESRGAN removes batch normalization, incorporates dense blocks with residual connections, and uses a Relativistic GAN discriminator.
  • 3.9.4 ESRGAN: ESRGAN produces comparatively better visual results than RCAN but lags behind RCAN on quantitative measures.

4 EXPERIMENTAL EVALUATION

The evaluation benchmarks SR algorithms across six public datasets and multiple scaling factors using PSNR and SSIM, while comparing architectural complexity and training characteristics. DRLN performs best at 2× and 3×, ESRGAN at 4×, but comparisons depend on factors including complexity, depth, training data, and feature-map counts; higher magnification exposes texture-reconstruction weaknesses.

  • Datasets: Six public datasets—Set5, Set14, BSD100, Urban100, DIV2K, and Manga109—form the benchmark suite for single-image super-resolution.DIV2K results use validation images because its test set is not publicly available.
  • Quantitative Measures: PSNR and SSIM evaluate the methods at 2×, 3×, 4×, and 8× super-resolution.Tables 2 and 3 report results through 4×, while Table 4 reports PSNR and SSIM for 8×.
  • Quantitative Measures: DRLN is better for 2× and 3×, whereas ESRGAN is better for 4×.The paper cautions that a clear overall winner is difficult to declare because complexity, depth, training data, patch size, and feature-map counts vary.
  • Qualitative Comparison: GAN-based outputs are generally more crisp, but their PSNR values are relatively lower than methods optimizing pixel-level losses.Figure 6 compares PSNR-oriented methods, while Figure 7 compares perceptually driven GAN algorithms.
  • Architectural and Training Analysis: Recent SR networks increasingly use ℓ1 loss, deeper architectures, and combinations of global, local, recursive, and dense skip connections.The survey reports a shift from early ℓ2 training toward ℓ1, increasing depth from VDSR through RCAN, and diverse skip-connection designs.

5 SUPER-RESOLUTION COMPETITIONS

Competitions have accelerated single-image super-resolution research by introducing datasets, advancing the state of the art, comparing diverse algorithms, and testing challenging settings. NTIRE emphasizes known and unknown downsampling on DIV2K, while PIRM jointly evaluates perceptual quality and PSNR.

  • Overview: Competitions organized by companies and conferences are identified as a primary source of the rapid pace of single-image super-resolution research.The survey introduces NTIRE and PIRM as two important challenges.
  • NTIRE: NTIRE introduced the DIV2K dataset in 2017 to benchmark single-image super-resolution.Its stated challenge goals include introducing a dataset, advancing the state of the art, comparing diverse algorithms, and applying challenging settings.
  • NTIRE: NTIRE Track-1 uses classical bicubic degradation, whereas Track-2 uses an unknown downsampling operator learned only from training low-resolution and high-resolution pairs.Track-2 employs blur and decimation without added noise.
  • PIRM: PIRM focuses on perceptual image quality while jointly quantifying PSNR accuracy, allowing perceptual- and PSNR-driven algorithms to advance together.The challenge evaluates 4× super-resolution using full-reference and no-reference measures, including PSNR, SSIM, LPIPS, NIQE, and BRISQUE.
  • PIRM: PIRM evaluates submissions on 100 images from two diverse-content sets through an online portal without providing participants the ground-truth high-resolution images.Participants choose the datasets used for model training.

6 FUTURE DIRECTIONS/OPEN PROBLEMS

Future work targets three persistent weaknesses in deep-learning super-resolution: insufficient priors for scarce degradation data, perceptual objective limitations, and poor generalization from synthetic to real degradations.

  • Incorporation of Priors: Data-driven end-to-end SR can be sub-optimal for degradation classes lacking abundant training data, motivating priors based on sensor, scene, and acquisition information.Proposed priors include deep-network and sparse-coding approaches.
  • Objective Functions and Metrics: Pixel-level losses and existing perceptual metrics do not universally produce perceptually sound images or quantify image quality across all conditions.High PSNR and SSIM can coincide with overly smooth images, while learned metrics such as LPIPS and PieAPP also have failure cases.
  • Real vs Artificial Degradation: Networks trained with bicubic-generated degradations do not generalize well to real low-resolution images because their degradation distributions differ.Recent efforts model real-world degradation with GANs or exchange multi-scale information while preserving spatial resolution.

7 CONCLUSIONS

The survey finds that deep-learning super-resolution methods have advanced rapidly, but their strengths and limitations vary with perceptual goals, magnification, and computational cost. Residual learning contributes to performance improvement, while high-magnification reconstruction remains difficult.

  • 7 CONCLUSIONS: GAN-based methods generally produce visually pleasing outputs, whereas reconstruction-error methods more accurately preserve spatial details.The comparison distinguishes perceptual appearance from spatial-detail fidelity.
  • 7 CONCLUSIONS: At magnification rates of 8× or above, existing super-resolution models generally deliver sub-optimal results.High magnification makes recovery of missing scene details more difficult.
  • 7 CONCLUSIONS: Top-performing methods generally have higher computational complexity and greater depth than their counterparts.Accuracy improvements are accompanied by increased model complexity.
  • 7 CONCLUSIONS: Residual learning has been a major contributor to performance improvement by decomposing signals and making the learning task easier.The survey identifies residual learning as an important architectural factor in progress.
Loading 1904.07523v3…