PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Journey into Super-resolution: A survey

Saeed Anwar, Salman Khan, Nick Barnes

arXiv:1904.07523v3cs.CV

TL;DR

Single-image super-resolution remains challenging despite its real-life importance. This survey reviews and classifies deep-learning methods and compares them across six datasets, reporting rapid growth in the field and diverse architectural and learning approaches.

  • Problem

    Single-image super-resolution remains a challenging research problem despite its important real-life applications.

    Source: §7 · p. 18

  • Method

    The survey reviews deep-learning super-resolution methods, introduces a structural taxonomy, and systematically compares algorithms on six publicly available SISR datasets.

    Source: §1 · pp. 1–2

  • Results

    Deep-learning super-resolution has grown rapidly, with diverse architectural and learning approaches.

    Source: §7 · p. 18

  • Takeaways & Limitations

    GAN-based approaches generally produce visually pleasing outputs, while reconstruction-error methods more accurately preserve image spatial details.

    Source: §7 · p. 18

    Networks trained on bicubic-generated degradations do not generalize well to real-world low-resolution images.

    Source: §6 · p. 18

Abstract

from arXiv · show

Deep convolutional networks based super-resolution is a fast-growing field with numerous practical applications. In this exposition, we extensively compare 30+ state-of-the-art super-resolution Convolutional Neural Networks (CNNs) over three classical and three recently introduced challenging datasets to benchmark single image super-resolution. We introduce a taxonomy for deep-learning based super-resolution networks that groups existing methods into nine categories including linear, residual, multi-branch, recursive, progressive, attention-based and adversarial designs. We also provide comparisons between the models in terms of network complexity, memory footprint, model input and output, learning details, the type of network losses and important architectural differences (e.g., depth, skip-connections, filters). The extensive evaluation performed, shows the consistent and rapid growth in the accuracy in the past few years along with a corresponding boost in model complexity and the availability of large-scale datasets. It is also observed that the pioneering methods identified as the benchmark have been significantly outperformed by the current contenders. Despite the progress in recent years, we identify several shortcomings of existing techniques and provide future research directions towards the solution of these open problems.

1 INTRODUCTION

This survey focuses on deep-learning methods for single natural-image super-resolution, reviewing architectures, evaluating algorithms on six public datasets, and discussing challenges and future directions.

Introduction

  • Super-resolution supports improved scene and object detail for displays and handheld devices, with applications including object detection, small-object detection, face recognition, and medical imaging.Other cited application areas include remote sensing, astronomy, and forensics.Source: §1 · p. 1
  • SR is ill-posed, admits multiple solutions for one low-resolution image, becomes harder at higher scaling factors, and is difficult to assess because PSNR and SSIM loosely track perception.Reliable prior information is typically needed to constrain possible solutions; at higher scaling factors, detail recovery can reproduce incorrect information.Source: §1 · p. 1
  • Because deep-learning methods outperform classical algorithms, the survey focuses on data-driven SR reconstruction and provides only a brief background on traditional approaches.The survey focuses on deep neural networks for single natural-image super-resolution.Source: §1 · p. 1

Our Contributions

  • The survey reviews recent SR techniques, proposes a structural taxonomy, analyzes models and training, evaluates SISR algorithms on six public datasets, and discusses challenges and future directions.Its analysis covers parameters, algorithm settings, training details, and architectural innovations.Source: §1 · pp. 1–2

2 BACKGROUND

Super-resolution reconstructs an approximation of a high-resolution image from a low-resolution image when the degradation process is unknown. The section describes a common degradation model and places deep-learning methods among broader super-resolution approaches.

2 BACKGROUND

  • Super-resolution seeks to recover an approximation of the ground-truth high-resolution image from the available low-resolution image despite unknown degradation parameters.The degradation may be complex and include noise, compression, blur, and other artifacts.Source: §2 · p. 2
  • A common degradation model combines blur, downsampling, and additive white Gaussian noise, and super-resolution aims to minimize its data-fidelity term.The model identifies k as the blurring kernel, s as the downsampling scale, and σ as the noise standard deviation.Source: §2 · p. 2
  • Image-prior approaches include prediction, edge-based, statistical, patch-based, and deep-learning methods; this article focuses on deep neural networks learning the prior.The categorization is attributed to Yang et al., with cited examples for each category.Source: §2 · p. 2

3 SINGLE IMAGE SUPER-RESOLUTION · 3.1 Linear networks

The survey places single-image super-resolution methods into nine design categories and describes linear networks as sequential convolutional models distinguished by early or late upsampling. It traces early designs from SRCNN to residual-learning approaches and late designs that process low-resolution features for efficiency.

3 SINGLE IMAGE SUPER-RESOLUTION

  • The survey groups single-image super-resolution methods into nine categories based on distinctive model-design features.Figure 1 presents the taxonomy used to organize the literature.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Linear networks

  • Linear networks use a single sequential signal path without skip connections or multiple branches, and differ in whether upsampling occurs early or late.Some also learn the residual image—the difference between LR and HR images—while remaining linear networks.Source: §3.1 · p. 2

3.1.1 Early Upsampling Designs

  • SRCNN, pioneered convolution-only super-resolution with three stacked convolutions for feature extraction, nonlinear mapping, and HR-image reconstruction.It first upsamples LR inputs with bicubic interpolation; training uses nonoverlapping 32×32 HR patches and MSE loss.Source: §3.1.1 · p. 2
  • VDSR uses a 20-weight-layer VGG-style CNN with residual learning and gradient clipping, while DnCNN directly predicts high-frequency residuals.VDSR’s strategies support high learning rates, and its reported results support deeper networks for contextualization and multi-scale representations.Source: §3.1.1 · pp. 2–3
  • VDSR- and DnCNN-like approaches can depend heavily on noise-estimation accuracy and incur computational cost from batch normalization; IRCNN combines CNN denoisers with HQS optimization.IRCNN uses seven dilated convolution layers, and its set of 25 denoisers trained across noise levels [0,50] performs image denoising, deblurring, and super-resolution.Source: §3.1.1 · p. 3

3.1.2 Late Upsampling Designs

  • FSRCNN processes original LR patches before deconvolutional upsampling, targeting 24 fps versus SRCNN’s 1.3 fps; its expansion layer improves performance by 0.3dB.Its architecture includes feature extraction, shrinking, nonlinear mapping, expansion, and a final deconvolution layer.Source: §3.1.2 · pp. 3–4
  • ESPCN extracts features in LR space and uses a final sub-pixel convolution for HR reconstruction, reducing memory and computation while enabling real-time 1080p video processing on one GPU.The network is trained with ℓ1 loss.Source: §3.1.2 · p. 4

3.2 Residual Networks

Residual networks use skip connections to make very deep designs feasible and learn high-frequency residuals; the survey groups them into single-stage and multi-stage approaches.

3.2 Residual Networks

  • Residual learning uses skip connections to mitigate vanishing gradients and learn the high-frequency difference between input and ground truth.The survey classifies residual approaches by stage count as single-stage or multi-stage networks.Source: §3.2 · p. 4

3.2.1 Single-stage Residual Nets

  • EDSR and MDSR outperform older SRCNN, VDSR, and related ResNet designs, while CARN cascades intermediate features through local and global modules.EDSR removes batch normalization and ReLU in specified locations; MDSR shares most parameters across scales, and both use ℓ1 loss. CARN’s cascading modules converge features through a 1×1 convolution.Source: §3.2.1 · p. 4

3.2.2 Multi-Stage Residual Nets

  • Multi-stage residual designs use successive subnets to refine coarse predictions, including FormResNet, balanced two-stage BTSRN, and encoder-decoder REDNet.FormResNet separates removal of high-frequency corruption from learning structured regions and slightly improves on DnCNN; BTSRN processes low- then high-resolution features, while REDNet uses symmetric convolutional and deconvolutional layers with skip connections.Source: §3.2.2 · pp. 4–5

3.3 Recursive networks · 3.4 Progressive reconstruction designs

Recursive networks decompose super-resolution into simpler subproblems, while progressive designs predict large-scale outputs in stages.

3.3 Recursive networks

  • Recursive networks,, use recursively connected convolutional layers or linked units to break difficult super-resolution problems into simpler ones.Source: §3.3 · p. 5

3.3.1 DRCN

  • DRCN applies the same convolution layers repeatedly, keeping parameter count constant across recursions, and comprises embedding, inference, and reconstruction networks.Its inference network repeatedly applies a convolution-and-ReLU layer, increasing the receptive field with each recursion.Source: §3.3.1 · p. 5

3.3.2 DRRN

  • DRRN reaches 52 convolutional layers while reducing network complexity by factors of 14 versus REDNet, 6 versus DRCN, and 2 versus VDSR.Shared parameters across replicated skip-connection blocks reduce memory and computational cost; local identity connections support stable training of deeper architectures.Source: §3.3.2 · p. 5

3.3.3 MemNet

  • MemNet has six memory blocks with six recursions (80 layers), combining two-convolution pre-activation recursive units with dense connections to gate units.It uses MSE loss and is also applied to image denoising and JPEG deblocking.Source: §3.3.3 · p. 6

3.3.4 SRFBN

  • SRFBN recursively refines low-resolution inputs through feedback blocks with dense cross-scale connections, using four iterations and an ℓ1 objective.For multiple degradations, it uses curriculum learning with progressively more complex HR ground truths; evaluations include Gaussian blur as well as bicubic downsampling.Source: §3.3.4 · p. 6

3.4 Progressive reconstruction designs

  • Progressive reconstruction methods [56], predict outputs in multiple steps to address large scaling factors that may be infeasible to handle in one step.Source: §3.4 · p. 6

3.4.1 SCN

  • SCN combines sparse coding with a LISTA-style network of recurrent stages that learn thresholded sparse codes, then reconstructs HR patches using a dictionary.The design aims for a compact model and improved performance.Source: §3.4.1 · p. 6

3.4.2 LapSRN

  • One multi-scale LapSRN model outperforms three separate models for 2×, 4×, and 8× super-resolution, leveraging common inter-scale traits.The passage presents shared inter-scale traits as one explanation for the improvement.Source: §3.4.2 · p. 6

3.5 Densely Connected Networks · 3.6 Multi-branch designs

Densely connected designs combine features across network depth, while multi-branch designs fuse features from different contexts or processing paths. The section describes architectures that use these strategies to improve or tailor super-resolution reconstruction.

3.5.1 SRDenseNet

  • SRDenseNet uses dense connections to combine hierarchical features, and its variant combining feature maps from multiple stages performs best among its variants.Dense connections pass each layer the outputs of all preceding layers; the models use ℓ2 loss and improve over models without dense connections.Source: §3.5.1 · p. 8, §3.5 · p. 8

3.5.2 RDN

  • RDN combines dense connections with local and global residual learning, fusing features to focus blocks on residual patterns and improve over SRDenseNet.Local feature fusion uses 1×1 convolutions to reduce dimensions; RDN uses ℓ1 loss and trains on 32×32 patches.Source: §3.5.2 · p. 8

3.5.3 D-DBPN

  • D-DBPN densely connects up- and down-sampling stages and adds residual feedback to upsampled features, with a 4× model of about 10 million parameters.A lower-complexity version of the final model incurs a slight performance drop.Source: §3.5.3 · p. 8

3.6.1 CNF

  • Multi-branch designs fuse complementary features across context scales; CNF fuses SRCNNs with different depths using sum-pooling.CNF trains each SRCNN separately before training the fused network, a progressive strategy that moves from individual subnets to joint optimization.Source: §3.6.1 · pp. 8–9, §3.6 · p. 8

3.6.2 CMSC

  • CMSC uses parallel merge-and-run branches with 3×3, 5×5, and 7×7 receptive fields, but lags behind EDSR and MDSR.Its ℓ1 loss combines intermediate outputs with the final output using a balancing term.Source: §3.6.2 · p. 9

3.6.3 IDN

  • IDN stacks information-distillation blocks that slice and recombine features, use 1×1 convolutions for compression, and reconstruct with a deconvolution layer.The enhancement units use six convolutional layers and leaky ReLU.Source: §3.6.3 · p. 9

3.6.4 EBRN

  • EBRN reports impressive state-of-the-art results for 2×, 4×, and 8× super-resolution using ten parallel branches and recursive fusion.It trains first with ℓ1 loss and then fine-tunes with ℓ2 loss.Source: §3.6.4 · p. 9

3.7 Attention-based Networks

Attention-based SR networks selectively emphasize features instead of treating all spatial locations and channels as equally important; coupled with deep networks, they have shown significant improvements,.

3.7.1 SelNet

  • SelNet uses selection units between convolutional layers to gate feature-map values, with 22 convolutional layers and a selection unit after each.The unit combines identity mapping with ReLU, a 1×1 convolution, and sigmoid; SelNet also uses residual learning and gradient switching for faster learning.Source: §3.7.1 · p. 9

3.7.2 RCAN

  • RCAN combines recursive residual connections with channel attention that selects important feature maps and models their relationships.Its recursive residual style improves convergence in very deep networks, and it performs better than IRCNN, VDSR, and RDN, but has about 15 million parameters for 4× SR.Source: §3.7.2 · pp. 9–10

3.7.3 DRLN

  • DRLN currently provides the best results across all datasets using a modular architecture with dense residual units, cascading connections, and Laplacian attention.Its improvement is attributed to the Laplacian-attention module and cascading structure; compared with RCAN, it has fewer convolutional layers but more parameters and is computationally less expensive because it concatenates channels rather than adding them.Source: §3.7.3 · p. 10

3.7.4 SRRAM

  • SRRAM uses residual attention modules in its upscaling stage, with spatial and channel attention learning intra- and inter-channel dependencies.Its overall structure comprises feature extraction, feature upscaling, and feature reconstruction; the upscaling stage consists of residual blocks followed by spatial and channel attention.Source: §3.7.4 · p. 10

3.8 Multiple-degradation handling networks

Earlier super-resolution networks assume bicubic degradation, but real-world images may involve multiple simultaneous degradations, motivating methods that handle these scenarios.

3.8.1 ZSSR

  • ZSSR trains an image-specific network on a downsampled test image and reuses it during inference, requiring no training images for a particular degradation.Its eight convolutional layers have 64 channels and learn the residue image using ℓ1 norm.Source: §3.8.1 · p. 10

3.8.2 SRMD

  • SRMD concatenates the low-resolution image with degradation maps and directly learns high-resolution images, while SRMDNF learns from noise-free degradations.SRMD applies cascaded 3×3 convolutions, then Conv, ReLU, and Batch normalization layers; extracted HR sub-images are transformed into a single HR output.Source: §3.8.2 · p. 10
  • Both SRMD and its variant cannot break the PSNR records of earlier SR networks, including EDSR, MDSR, and CMSC, but uniquely jointly tackle multiple degradations.The models use 12 convolutional layers, each with 128 feature maps, and ℓ1 loss.Source: §3.8.2 · p. 10

3.9 GAN Models

GAN-based super-resolution pits a generator against a discriminator to produce perceptually better HR images, often with degraded PSNR, [80].

3.9.1 SRGAN

  • SRGAN combines pixel-wise MSE, deep-feature perceptual, and adversarial losses, and significantly outperformed competitors on human-rated perceptual quality measured by its MOS.MOS is assigned manually by human raters to rate each super-resolved image.Source: §3.9.1 · p. 11

3.9.2 EnhanceNet

  • EnhanceNet adds perceptual and texture-matching losses to MSE in adversarial training, yielding more realistic outputs but risking visible artifacts in highly textured regions.The perceptual loss uses ℓ1 distance on pretrained-network features, while the texture loss compares Gram matrices; pixel-level loss alone achieves the best PSNR.Source: §3.9.2 · p. 11

3.9.3 SRFeat

  • SRFeat [85] uses an additional discriminator to distinguish synthetic from real image features, encouraging high-frequency structural detail rather than noisy artifacts.Source: §3.9.3 · p. 11

3.9.4 ESRGAN

  • ESRGAN removes batch normalization, adds dense blocks with residual connections and a Relativistic GAN discriminator, and produces better visual results than RCAN despite weaker quantitative measures.It is trained first with ℓ1 loss and then perceptual loss on 3,450 DIV2K and Flicker2K images; the network is 23 blocks deep.Source: §3.9.4 · p. 12

4 EXPERIMENTAL EVALUATION

The evaluation compares super-resolution methods across six benchmark datasets and finds that the strongest reported method varies with scaling factor. It also relates performance and visual quality to computational cost, loss choice, network depth, and skip connections.

4.1 Datasets

  • The evaluation uses Set5, Set14, BSD100, Urban100, DIV2K, and Manga109; because the DIV2K test set is unavailable, results are reported on its validation images.DIV2K contains 800 training images and 100 each for testing and validation; Manga109 comprises 109 test images.Source: §4.1 · p. 12

4.2 Quantitative Measures

  • GAN-based outputs are generally crisper but have relatively lower PSNR than methods optimized for pixel-level losses.Figure 6 compares methods aiming to improve PSNR, while Figure 7 shows outputs from perceptually driven GAN methods.Source: §4.2 · p. 13

4.4 Number of parameters

  • Methods that upsample late in the network have considerably lower computational cost than methods that upsample earlier.Direct reconstruction upsamples once from LR to HR, whereas progressive reconstruction predicts HR images through multiple upsampling steps.Source: §4.4 · pp. 13, 16, Table 1 · p. 13

4.2 Quantitative Measures

  • DRLN performs better at 2× and 3×, while ESRGAN performs better at 4×; the paper does not identify one clear overall winner.The comparison notes that complexity, network depth, training data, patch size, and feature-map count also affect results, so a fair comparison requires consistent settings.Source: §4.2 · p. 13, Table 2 · p. 14, Table 3 · p. 15

4.3 8× Super-resolution

  • At 8× magnification, most state-of-the-art methods struggle to reproduce image textures, and artifacts become more visible.The paper provides 8× comparisons in Table 4 and Figure 6.Source: §4.3 · p. 13, Figure 6 · p. 15, Table 4 · p. 16

4.5 Choice of network loss

  • Recent CNN super-resolution methods have shifted from ℓ2 toward ℓ1 loss, which the paper describes as more robust because it balances the error distribution.GANs also use perceptual (adversarial) loss alongside pixel-level losses such as MSE; ℓ2 emphasizes more erroneous predictions.Source: §4.5 · p. 16

4.6 Network depth

  • After SRCNN disputed the value of depth, VDSR and EDSR showed that deeper networks can improve PSNR and image quality, a trend continued by RCAN and newer CNNs,.EDSR increased the number of convolutional layers to nearly four times that of VDSR, and RCAN used more than 400 convolutional layers.Source: §4.6 · p. 16

4.7 Skip Connections

  • Skip connections support improved super-resolution results through global, local, recursive, and dense designs used across methods including VDSR, DRRN, DRCN, EDSR, RCAN, RDN, and ESRGAN.EDSR and RCAN combine local residual learning with global residual learning; RDN and ESRGAN use dense and global connections.Source: §4.7 · p. 16

5 SUPER-RESOLUTION COMPETITIONS

Company- and conference-organized competitions have driven rapid research in single-image super-resolution. NTIRE and PIRM benchmark different aspects of super-resolution, from diverse algorithms and challenging settings to perceptual quality.

5.1 NTIRE

  • Introduced in 2017, NTIRE uses DIV2K to test 2×, 3×, and 4× super-resolution with bicubic Track 1 and unknown-downsampling Track 2.The challenge aims to introduce DIV2K, advance the state of the art, compare diverse algorithms, and apply challenging settings.Source: §5.1 · p. 16

5.2 PIRM

  • PIRM assesses 4× bicubic-degraded images for perceptual quality alongside PSNR, using PSNR, SSIM, RMSE, FC, LPIPS, Ma et al., NIQE, and BRISQUE.Its perceptual index is computed from Ma et al. and NIQE.Source: §5.2 · p. 17
  • PIRM evaluates 100 images from two diverse-content sets without releasing ground-truth high-resolution images to participants, who submit outputs online and choose their training datasets.Source: §5.2 · p. 17

6 FUTURE DIRECTIONS/OPEN PROBLEMS

The survey identifies open problems in super-resolution spanning priors, perceptual objectives, unified and unsupervised models, extreme or arbitrary upsampling, and the gap between synthetic and real degradations.

Incorporation of Priors

  • Known sensor, scene, and acquisition information could guide priors when end-to-end data-driven SR lacks sufficient training data for a degradation, such as in medical imaging.Proposed directions include deep-network [100] and sparse-coding priors.Source: §6 · pp. 17–18

Objective Functions and Metrics

  • No universal perceptual metric captures image quality across conditions, making new objective functions an open problem.Pixel-level ℓ1 and ℓ2 losses can produce perceptually unsound results; even high-PSNR and high-SSIM images may be overly smooth, while fixed and learned perceptual metrics each have failure cases.Source: §6 · p. 18

Need for Unified Solutions

  • Unified models must jointly address simultaneous degradations, such as low resolution and noise, despite task-specific architectures, losses, and training details.Designing models that perform well across several low-level vision tasks remains challenging.Source: §6 · p. 18

Unsupervised Image SR

  • Unsupervised SR could learn from unpaired LR-HR sets [107], avoiding matched pairs that are difficult to obtain in real-world settings.Zero-shot SR learns from a further-downsampled input but cannot work when the input is already of poor resolution.Source: §6 · p. 18

Higher SR rates

  • Very few works target SR rates above 8×, such as 16× or 32×, where preserving accurate local detail and perceptual quality remains challenging.Source: §6 · p. 18

Arbitrary SR rates

  • Unknown downsampling factors complicate training a single model to capture multiple detail levels, motivating degradation characterization before choosing an SR model.Source: §6 · p. 18

Real vs Artificial Degradation

  • Bicubic-trained SR networks generalize poorly to real LR images because their degradation distribution differs from actual images.Proposed directions model real-world degradation with a GAN or enrich features through preserved spatial resolution and multi-scale information exchange; CVPR’19 challenges promoted research,.Source: §6 · p. 18

7 CONCLUSIONS

The survey finds that deep-learning super-resolution has improved alongside growing network complexity, but current methods still face limitations that constrain real-world application. Its comparisons identify trade-offs in output quality, weaker results at high magnification, and residual learning as an important contributor to performance gains.

7 CONCLUSIONS

  • GAN-based approaches generally produce visually pleasing outputs, while reconstruction-error methods more accurately preserve spatial image details.These are different strengths rather than a single overall winner.Source: §7 · p. 18
  • At magnification rates of 8× or above, existing models generally deliver sub-optimal results.Source: §7 · p. 18
  • Top-performing methods are generally deeper and computationally more complex than their counterparts.Source: §7 · p. 18
  • Residual learning has substantially contributed to performance improvements by decomposing the signal and making the learning task easier.Source: §7 · p. 18
  • Despite recent performance gains, current methods remain limited by inadequate metrics, high model complexity, and inability to handle real-life degradations.These shortcomings restrict application to key real-world scenarios.Source: §7 · p. 18

Related papers

Loading 1904.07523v3…