PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Image Super-resolution: A Survey

Zhihao Wang, Jian Chen, Steven C. H. Hoi

arXiv:1902.06068v2cs.CV

TL;DR

Image super-resolution is challenging because low-resolution inputs can correspond to multiple high-resolution images and standard quality measures do not fully capture perception. This survey systematically reviews deep-learning SR across supervised, unsupervised, and domain-specific settings, together with datasets, metrics, components, and future directions. It concludes that substantial progress has been made, while many problems remain unsolved.

  • Problem

    SR is an inherently ill-posed recovery problem, and accurately measuring SR quality remains difficult because distortion and perceptual quality can be at odds.

  • Method

    The paper provides a comprehensive, hierarchical survey of deep-learning SR methods, settings, datasets, metrics, applications, components, and open issues.

  • Results

    The survey covers advances in supervised and unsupervised SR, domain-specific applications, and modular design elements including residual learning, losses, and multi-supervision.

  • Takeaways & Limitations

    The survey identifies challenges and future directions intended to improve understanding and facilitate future research and application development in image SR.

  • Takeaways & Limitations

    Perceptual quality assessment remains constrained because PSNR is pixel-based, while MOS has nonlinear scales, rating biases, and variance.

Abstract

from arXiv · show

Image Super-Resolution (SR) is an important class of image processing techniques to enhance the resolution of images and videos in computer vision. Recent years have witnessed remarkable progress of image super-resolution using deep learning techniques. This article aims to provide a comprehensive survey on recent advances of image super-resolution using deep learning approaches. In general, we can roughly group the existing studies of SR techniques into three major categories: supervised SR, unsupervised SR, and domain-specific SR. In addition, we also cover some other important issues, such as publicly available benchmark datasets and performance evaluation metrics. Finally, we conclude this survey by highlighting several future directions and open issues which should be further addressed by the community in the future.

1 INTRODUCTION

Image super-resolution recovers high-resolution images from low-resolution inputs, but the task is inherently ill-posed. This survey organizes deep-learning SR advances systematically, covering methods, evaluation resources, applications, challenges, and future directions.

  • Introduction: Super-resolution recovers high-resolution images from low-resolution inputs and supports applications including medical imaging, surveillance, and security.A single low-resolution image can correspond to multiple high-resolution images, making SR challenging and ill-posed.
  • Introduction: Deep-learning SR methods range from convolutional neural networks such as SRCNN to generative adversarial networks such as SRGAN.These models have often achieved state-of-the-art performance on SR benchmarks.
  • Introduction: The survey focuses on deep-learning SR techniques rather than primarily traditional SR algorithms or metric-centered evaluations.It provides a systematic and comprehensive deep-learning perspective on recent advances.
  • Introduction: The survey reviews problem settings, datasets, metrics, deep-learning SR methods, and domain-specific applications, while summarizing component advantages and limitations.It also discusses challenges, open issues, trends, and future directions.
  • Introduction: Its coverage is structured hierarchically across problem definitions, supervised SR, unsupervised SR, domain-specific applications, and future directions.The survey also includes benchmark datasets and evaluation metrics.

2 PROBLEM SETTING AND TERMINOLOGY

The SR problem models a low-resolution image as degraded from a corresponding high-resolution image, while practical systems must address unknown degradations and evaluation resources. The survey also describes common dataset construction and degradation patterns.

  • Problem definitions: Super-resolution seeks to recover a corresponding high-resolution image from a low-resolution image generated by a degradation process.The degradation mapping and its parameters may be unknown, requiring blind SR to estimate an HR approximation.
  • Datasets: Most generic SR datasets are built by applying predefined bicubic downsampling with anti-aliasing to high-resolution images.Other datasets provide matched LR-HR pairs, while some provide only HR images from which LR inputs are generated.
  • Degradation models: The degradation may include downsampling, blur-kernel convolution, and additive white Gaussian noise, with bicubic interpolation commonly used in generic SR datasets.The combined degradation pattern is closer to real-world cases and has been reported as more beneficial for SR.
  • Datasets: Benchmark datasets differ in image amount, quality, resolution, diversity, format, and category, and may be combined for training.Examples include combining T91 with BSDS300 or DIV2K with Flickr2K.

2.3 Image Quality Assessment

Image quality assessment for super-resolution combines subjective human judgments with objective computational metrics, but these measures can disagree and may trade off distortion against perceptual quality.

  • IQA methods: Objective IQA methods are mainstream because subjective human assessment is time-consuming and expensive, despite imperfect agreement with visual perception.Objective methods include full-reference, reduced-reference, and no-reference assessments.
  • Objective metrics: PSNR measures pixel-level reconstruction error through MSE, but it can poorly represent visual quality in real scenes.PSNR remains widely used because SR results must be compared with prior literature and no completely accurate perceptual metric exists.
  • Objective metrics: SSIM evaluates luminance, contrast, and structural similarity, making it more aligned with human visual-system-based perceptual assessment than purely pixel-level comparisons.Its formulation combines luminance, contrast, and structure comparisons with adjustable relative-importance parameters.
  • Subjective metrics: MOS averages human ratings, typically from 1 to 5, and is especially useful when perceptual quality conflicts with common metrics such as PSNR.However, MOS suffers from nonlinear scales, rating bias, and variance in rating criteria.
  • Learned perceptual metrics: Learned no-reference metrics predict perceptual quality without ground-truth images, but the desired notion of quality—realism or identity consistency—remains unresolved.Consequently, objective metrics such as PSNR and SSIM remain mainstream.
  • Evaluation challenge: Distortion and perceptual quality can be inherently opposed: mathematically, reducing distortion may worsen perceptual quality.This makes accurate SR-quality measurement an unresolved problem.

2.4 Operating Channels

Super-resolution studies use different image channels and benchmark challenges to evaluate reconstruction under ideal, realistic, and perceptual-quality-oriented conditions.

  • Operating channels: YCbCr represents images through luminance Y and blue- and red-difference chroma channels, while SR models differ in whether they operate on Y or RGB channels.The survey notes that no accepted best practice currently determines the preferred color space for SR.
  • NTIRE Challenge: NTIRE evaluates SR on DIV2K through bicubic-downscaling and blind tracks with unknown realistic degradation.Its tracks vary in degradation and scaling factors to cover ideal and adverse real-world conditions.
  • PIRM Challenge: PIRM focuses on the perception-distortion tradeoff, selecting winners by perceptual quality within RMSE-defined regions.It also includes a separate smartphone SR sub-challenge.

3 SUPERVISED SUPER-RESOLUTION

Supervised SR trains deep-learning models on paired low-resolution and high-resolution images, combining modular components into integrated reconstruction systems.

  • Supervised SR: Supervised SR models are trained with both LR images and their corresponding HR images.Although architectures differ substantially, they can be understood as combinations of recurring design components.
  • Supervised SR: These components include model frameworks, upsampling methods, network design, and learning strategies.Researchers combine them to construct integrated SR models for specific reconstruction settings.

3.1 Super-resolution Frameworks

Deep-learning SR models can be organized by where and how they perform upsampling. The survey compares four frameworks, emphasizing computational efficiency, scaling flexibility, reconstruction quality, and design limitations.

  • Pre-upsampling Super-resolution: Pre-upsampling frameworks first enlarge LR images with predefined interpolation, then use CNNs to refine the resulting coarse HR images.This reduces learning difficulty and supports arbitrary input sizes and scaling factors, but performs most computation in high-dimensional space.
  • Pre-upsampling Super-resolution: Pre-upsampling models are popular, yet predefined upsampling can amplify noise or blur images while increasing time and space costs.Their main differences lie in posterior network design and learning strategies.
  • Post-upsampling Super-resolution: Post-upsampling frameworks extract features in low-dimensional space and apply learnable upsampling layers only at the network end.This substantially reduces computation and spatial complexity, making the framework mainstream.
  • Progressive Upsampling Super-resolution: Progressive upsampling decomposes large-scale reconstruction into simpler stages, supporting large factors and multi-scale SR with limited additional cost.However, multi-stage design and training stability remain challenges.
  • Iterative Up-and-down Sampling Super-resolution: Iterative up-and-down sampling frameworks repeatedly compute reconstruction errors and feed them back to refine HR image intensities.They can better model LR-HR relationships, but back-projection module design criteria remain unclear.

3.2 Upsampling Methods

The survey contrasts fixed interpolation with learnable upsampling layers. Interpolation is simple and interpretable but can introduce artifacts and inefficiency, motivating end-to-end learned alternatives.

  • Interpolation-based Upsampling: Traditional interpolation methods resize images using neighboring pixel values and include nearest-neighbor, bilinear, bicubic, Sinc, and Lanczos methods.They remain useful because they are interpretable and easy to implement.
  • Interpolation-based Upsampling: Nearest-neighbor interpolation is fastest but usually produces blocky, low-quality results, whereas bilinear interpolation uses a 2 × 2 receptive field for better speed-quality balance.Bilinear interpolation performs linear interpolation along both image axes.
  • Interpolation-based Upsampling: Bicubic interpolation considers 4 × 4 pixels, producing smoother results with fewer artifacts than bilinear interpolation but at lower speed.With anti-aliasing, it is mainstream for constructing SR datasets and pre-upsampling frameworks.
  • Learning-based Upsampling: Interpolation-based upsampling relies only on existing image signals and may cause computational complexity, noise amplification, and blurring.These drawbacks motivate replacing fixed interpolation with learnable upsampling layers.
  • Learning-based Upsampling: Transposed convolution enlarges images by inserting zeros and convolving, while sub-pixel layers generate s^2 channels and reshape them into spatially enlarged outputs.Sub-pixel layers provide larger contextual receptive fields but may create boundary artifacts and unsmooth outputs.
  • Learning-based Upsampling: Meta upscale modules predict convolution weights for target positions, enabling SR at arbitrary scaling factors instead of requiring separate modules for each factor.Learning-based layers are widely used at the final upsampling stage of post-upsampling frameworks.

3.3 Network Design

The survey decomposes SR network design into reusable strategies layered onto framework choices. These strategies target representation quality, receptive field, feature fusion, attention, and computational efficiency, while introducing distinct trade-offs.

  • Residual Learning: Residual learning replaces complete image-to-image mapping with prediction of residual high-frequency details, exploiting strong input-target correlation.Global residual learning connects input and output images, while local residual learning uses internal shortcuts.
  • Recursive Learning: Recursive learning reuses modules multiple times to obtain larger receptive fields or stronger representations without introducing excessive parameters.DRCN reaches a 41 × 41 receptive field versus SRCNN’s 13×13, while DRRN outperforms a 17-ResBlock baseline after 25 recursions.
  • Recursive Learning: Recursive learning still incurs high computational costs and can cause vanishing or exploding gradients, so residual learning and multi-supervision are often combined with it.These techniques are used to mitigate recursive optimization difficulties.
  • Multi-path Learning: Multi-path learning sends features through different operations and fuses them to improve modeling, including global, local, and scale-specific variants.Scale-specific paths allow one network to handle multiple SR scales while sharing principal feature-extraction components.
  • Dense Connections: Dense connections expose each layer or block to preceding feature maps, supporting low- and high-level feature fusion and richer reconstruction details.They produce l · (l −1)/2 connections in an l-layer dense block and are used in several SR architectures.
  • Attention: Attention mechanisms model channel interdependence or long-range pixel dependencies to adaptively improve feature representations.Channel attention uses global average pooling and dense layers, while non-local attention rescales trunk features with learned masks.

3.4 Learning Strategies

Learning strategies shape the reconstruction objective, balancing pixel fidelity, perceptual quality, texture realism, stability, and optimization difficulty. The survey describes pixel, content, texture, adversarial, cycle-consistency, total-variation, curriculum-learning, and multi-supervision strategies.

  • Loss Functions: Pixel losses measure image differences directly, with L1 often improving performance and convergence over L2, but may produce oversmooth textures and lack high-frequency details.L2 penalizes larger errors more strongly and often produces smoother results; pixel loss is closely correlated with PSNR.
  • Loss Functions: Content loss compares high-level representations from a pretrained classification network, encouraging perceptual similarity rather than exact pixel matching.VGG and ResNet are commonly used feature extractors for this purpose.
  • Loss Functions: Texture loss matches feature correlations to improve texture realism, but empirical patch-size selection can cause artifacts when patches are too small or too large.EnhanceNet is reported to produce more realistic textures and visually more satisfactory results with texture loss.
  • Loss Functions: Adversarial and content losses improve perceptual quality despite lower PSNR than pixel-loss training, while GAN optimization remains difficult and unstable.The discriminator encourages generated images to conform to learned patterns of real high-resolution images.
  • Loss Functions: Cycle-consistency loss requires a reconstructed HR image to downsample back into an LR image matching the input, enforcing pixel-level consistency.The cycle-in-cycle approach super-resolves an LR image and then regenerates it through another CNN.
  • Loss Functions: Total-variation loss penalizes neighboring-pixel differences to measure and suppress noise while imposing spatial smoothness.It is introduced to reduce noise in generated images.
  • Learning Strategies: Curriculum learning decomposes training difficulty and shortens total training time, especially for large scaling factors, while multi-supervision improves gradient propagation.Multi-supervision feeds intermediate recursive outputs into reconstruction modules to address vanishing and exploding gradients.

3.5 Other Improvements

Beyond core architectures and learning strategies, SR performance can be improved through model fusion, data augmentation, auxiliary tasks, model interpolation, and self-ensemble inference.

  • Other Improvements: Context-wise network fusion combines predictions from separately trained SR networks through convolutional layers and summation.The framework can fuse multiple lightweight SRCNN models with different architectures.
  • Other Improvements: Data augmentation uses cropping, flipping, scaling, rotation, color jittering, and RGB-channel shuffling to boost performance and reduce color bias.RGB-channel shuffling addresses color imbalance in the dataset while augmenting the data.
  • Other Improvements: Multi-task learning supplies domain-specific information from related tasks, while SFT-GAN uses semantic maps to generate more realistic textures in rich semantic regions.DNSR separately trains denoising and SR networks before concatenating and jointly fine-tuning them to address noisy inputs.
  • Other Improvements: Network interpolation combines PSNR-oriented and GAN-based models to better balance distortion and perception.The models are trained separately before their corresponding parameters are interpolated.
  • Other Improvements: Self-ensemble applies rotations and horizontal flipping, runs the transformed images through the SR model, reverses the transformations, and aggregates the outputs.The final prediction is formed from the transformed reconstructions, commonly using their mean or median.

3.6 State-of-the-art Super-resolution Models

The survey reviews state-of-the-art deep-learning SR models by decomposing them into frameworks, upsampling methods, network designs, and learning strategies. It also emphasizes that efficiency matters alongside SR accuracy and summarizes representative methodologies in a comparison table.

  • State-of-the-art Models: State-of-the-art SR models are analyzed hierarchically through model frameworks, upsampling methods, network design, and learning strategies.The survey identifies advantages and limitations of these components across recent deep-learning SR models.
  • State-of-the-art Models: Table 2 compares representative models across frameworks, upsampling, recursive learning, residual learning, dense connections, and attention mechanisms.The table abbreviates these dimensions as Fw., Up., Rec., Res., Dense., and Att.
  • State-of-the-art Models: Efficiency is treated as an important complement to accuracy, with the survey examining computational impacts from strategies such as post-upsampling, recursion, dense connections, and xUnit.The survey also benchmarks representative approaches for efficiency.

4 UNSUPERVISED SUPER-RESOLUTION

Unsupervised super-resolution addresses the mismatch between predefined degradations and real-world low-resolution images by learning from unpaired data or image-specific internal statistics. The surveyed approaches improve robustness under nonideal conditions, but may incur longer inference or unstable training.

  • Motivation: Unsupervised SR uses unpaired LR-HR images to avoid relying on predefined degradation processes that may not reflect real-world imaging.Supervised datasets commonly construct LR images by degrading HR images with fixed procedures, so models may learn only the inverse of that degradation.
  • Zero-shot Super-resolution: ZSSR trains an image-specific network at test time using cross-scale internal recurrence within each image.It estimates a degradation kernel and generates a small training set by applying that degradation at multiple scales.
  • Zero-shot Super-resolution: 1 dB for estimated kernels and 2 dB for known kernels: ZSSR outperforms previous approaches on nonideal images while remaining competitive under ideal bicubic degradation.Nonideal conditions include blurring, noise, compression artifacts, and non-bicubic degradation.
  • Zero-shot Super-resolution: ZSSR has much longer inference time because it trains a different network for each image during testing.The image-specific optimization improves adaptation but increases test-time computation relative to other methods.
  • Weakly-supervised Super-resolution: Learned-degradation methods first estimate HR-to-LR degradation from unpaired images, then use generated LR-HR pairs to train an LR-to-HR SR model.The degradation generator is trained to match both downscaled HR images and the distribution of real LR images.
  • Weakly-supervised Super-resolution: Cycle-in-cycle SR avoids predefined degradation and can achieve performance comparable to supervised methods under harsh conditions, but its complicated architecture makes training difficult and unstable.CinCGAN learns mappings across noisy LR, clean LR, and clean HR domains using two coupled CycleGAN structures.

5 DOMAIN-SPECIFIC APPLICATIONS

Domain-specific super-resolution adapts reconstruction to specialized inputs, priors, and temporal or cross-modal structure. The survey covers depth, face, hyperspectral, raw-image, video, and feature-space applications.

  • Depth Map Super-resolution: Depth-map SR commonly uses a high-resolution RGB image of the same scene to guide reconstruction of a low-resolution depth map.Methods exploit depth statistics and local correlations between depth and RGB images, or jointly process depth and RGB features.
  • Face Image Super-resolution: Face hallucination incorporates facial priors such as landmarks, parsing maps, identities, and component structure to reconstruct plausible high-resolution faces.Explicit approaches constrain facial attributes or landmarks, while implicit approaches address alignment and component-specific reconstruction.
  • Face Image Super-resolution: Face SR also uses attention and adversarial learning to enhance local patches, exploit global facial dependencies, or generate hallucinated outputs.Attention-FH sequentially discovers attended patches, while other methods use adversarial architectures or multi-class GANs.
  • Hyperspectral Image Super-resolution: Hyperspectral SR combines high-resolution panchromatic images with low-resolution hyperspectral images to predict high-resolution hyperspectral outputs.HSIs provide abundant spectral features but are harder to collect at high quality and resolution than RGB-like panchromatic images.
  • Real-world Image Super-resolution: Real-world camera SR must account for the gap between processed 8-bit RGB images and original 12-bit or 14-bit RAW captures.Demosaicing, denoising, compression, and other camera-ISP operations discard or alter signals before RGB image formation; datasets such as City100 and SR-RAW target this setting.
  • Video Super-resolution: Video SR exploits inter-frame temporal dependency alongside intra-frame spatial dependency through motion compensation, recurrent modeling, or adaptive temporal aggregation.Methods use optical flow, learned alignment, recurrent networks, bidirectional models, and dynamic filters to fuse information across frames.
  • Other Applications: Feature-space SR applies super-resolution to representations of small objects or images to improve their usefulness for downstream detection and related vision tasks.Perceptual GAN and FSR-GAN target feature representations rather than directly reconstructing pixel-space images.

6 CONCLUSION AND FUTURE DIRECTIONS

The survey synthesizes deep-learning advances in supervised, unsupervised, and domain-specific SR while identifying unresolved challenges in efficiency, upsampling, learning strategies, evaluation, and real-world deployment. It proposes architectural, metric, and application directions for future work.

  • Conclusion: The survey reviews supervised and unsupervised SR, domain-specific applications, and future problems intended to guide research and application development.Its coverage includes recent advances and explicitly discusses unresolved issues and promising trends.
  • Network Design: Combining local and global information is proposed to provide contextual cues at different scales and support more realistic SR results.Large receptive fields supply broader context while local information preserves finer structure.
  • Network Design: Combining low-level details with high-level semantics may help reconstruct HR images by integrating shallow-layer features with deeper representations.Shallow layers capture colors and edges, whereas deeper layers learn higher-level representations such as object identities.
  • Network Design: Context-specific attention is proposed to emphasize features that matter differently across image regions, such as textures in grass and semantic details on animals.The direction targets more realistic detail generation by adapting attention to image context.
  • Network Design: 20s per image for 4× SR and 35s for 8× SR illustrate the efficiency challenge posed by large models and slow inference.The survey argues that practical SR requires reducing model size and prediction time while maintaining performance.
  • Network Design: Upsampling remains unresolved because interpolation is costly, transposed convolution can cause checkerboard artifacts, and other modules have receptive-field, stability, or efficiency drawbacks.The survey identifies effective and efficient upsampling at high scaling factors as an open problem.
  • Learning Strategies: Loss functions, normalization, and neural architecture search remain active directions for improving SR learning and model design.The best loss combination is unclear, batch normalization is suboptimal for SR, and NAS may improve performance or efficiency with less manual intervention.
  • Evaluation: PSNR and SSIM do not fully capture perceptual quality, while MOS is costly and non-reproducible, motivating more accurate evaluation metrics.The survey also calls for blind IQA methods because common reference-based evaluation assumes paired LR-HR images and often uses manual degradation.
Loading 1902.06068v2…