Source-linked AI summary
A Fully Progressive Approach to Single-Image Super-Resolution
Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga Sorkine-Hornung, Christopher Schroers
TL;DR
Large upsampling factors remain difficult for SISR because existing direct approaches face computational costs or artifacts. ProSR and ProGanSR address this with progressive multi-scale architectures and curriculum training, achieving strong quality across scales and competitive NTIRE2018 rankings while running faster than the top-ranking team.
Problem
High-quality SISR remains challenging at large upsampling factors, where the required upscaling function becomes more complex.
Method
ProSR progressively upsamples images by 2× through a pyramidal network, while ProGanSR applies the same multi-scale principle to a GAN and curriculum training.
Results
ProSR improves reconstruction quality across scales, with curriculum learning outperforming simultaneous training by an average of 0.04dB.
Takeaways & Limitations
The progressive multi-scale design supports high upsampling ratios while improving reconstruction quality and efficiency.
Takeaways & Limitations
Without task-specific adaptation, performance on unknown-degradation challenge tracks remained in the mid-range, with rankings of 7th/9th/7th.
Abstract
from arXiv · showhide
Recent deep learning approaches to single image super-resolution have achieved impressive results in terms of traditional error measures and perceptual quality. However, in each case it remains challenging to achieve high quality results for large upsampling factors. To this end, we propose a method (ProSR) that is progressive both in architecture and training: the network upsamples an image in intermediate steps, while the learning process is organized from easy to hard, as is done in curriculum learning. To obtain more photorealistic results, we design a generative adversarial network (GAN), named ProGanSR, that follows the same progressive multi-scale design principle. This not only allows to scale well to high upsampling factors (e.g., 8x) but constitutes a principled multi-scale approach that increases the reconstruction quality for all upsampling factors simultaneously. In particular ProSR ranks 2nd in terms of SSIM and 4th in terms of PSNR in the NTIRE2018 SISR challenge [34]. Compared to the top-ranking team, our model is marginally lower, but runs 5 times faster.
1. Introduction
SISR methods achieve strong results but remain challenged by large upsampling factors. The paper proposes progressive architecture and training, including an asymmetric pyramid and GAN design, to improve quality and efficiency across scales.
- Motivation: Large upsampling factors remain difficult because direct SISR approaches either require high-resolution processing or risk checkerboard artifacts.Early upsampling increases memory and computation, while late upsampling is more prone to artifacts.
- Progressive architecture: ProSR progressively reconstructs high-resolution images through intermediate 2× upsampling steps using dense compression units.Each pyramid level builds on the reconstruction from the previous level.
- Progressive architecture: The asymmetric pyramid allocates more layers to lower levels, enabling high upsampling ratios while remaining efficient.The architecture is designed to improve reconstruction accuracy and reduce memory consumption.
- Progressive GAN: ProGanSR uses a progressive generator and discriminator to produce more photorealistic results across multiple scales.The discriminator operates on residual outputs from each scale.
- Progressive training: Curriculum learning trains from small to large upsampling factors, improving results at all scales while shortening training and stabilizing GAN optimization.The learning process proceeds from easy settings to harder ones.
- Evaluation: Experiments report improved PSNR and perceptual quality against state-of-the-art methods, particularly at larger upsampling ratios.The evaluation covers a variety of datasets and both traditional error measures and perceptual quality.
2. Related Work
Prior SISR research spans statistical priors, exemplar-based methods, and direct or progressive deep architectures. The paper situates its approach against limitations in computational cost, artifacts, and perceptual detail reconstruction.
- Statistical and exemplar-based methods: Earlier SISR methods addressed the ill-posed problem using image priors such as gradient statistics, recurrence, self-examples, and total variation.These approaches formed a major strand of pre-deep-learning SISR research.
- Statistical and exemplar-based methods: Exemplar-based methods exploit redundancy in large image datasets through nearest-neighbor search and sparse dictionary learning.Deep convolutional networks later surpassed sparse coding techniques in reported comparisons.
- Direct versus progressive reconstruction: Direct reconstruction methods upscale images in one step, either before CNN processing or at the end using learned upsampling layers.The latter includes sub-pixel and transposed convolution layers.
- Direct versus progressive reconstruction: Progressive methods such as LapSRN predict residuals at successive pyramid levels and apply intermediate supervision at each scale.Later recursive and wider architectures with multi-scale training improved accuracy but did not eliminate the remaining gap.
- Direct versus progressive reconstruction: Higher pyramid levels can exhibit quadratic computational growth, creating a runtime bottleneck for expanding network capability.The paper also motivates progressive discriminator and training designs alongside the generator.
- Perceptual loss functions: Perceptual loss methods combine content similarity with adversarial guidance because reconstruction-error objectives cannot hallucinate perceptually plausible high-frequency details.This line of work targets perceptual quality beyond small reconstruction error.
3. Progressive Multi-scale Super-resolution
The paper proposes progressive multi-scale super-resolution through a pyramidal network that performs repeated 2× upsampling, paired with progressive GAN training and curriculum learning. This design targets large upsampling ratios while improving efficiency, reconstruction quality, and GAN training stability.
- Large upsampling ratios require increasingly complex parameterizations of the LR-to-HR upscaling function.
- 3.1. Pyramidal Decomposition: ProSR decomposes upscaling into pyramid levels, each refining features and performing its own 2× upsampling with dense compression units and sub-pixel convolution.The asymmetric pyramid assigns more DCUs to lower levels.
- 3.1. Pyramidal Decomposition: The network predicts a residual relative to a fixed bicubic upsampling, then adds that residual to obtain the estimated HR image.This residual formulation is expressed as ŷ = Rs(x) + ϕs(x).
- 3.1. Pyramidal Decomposition: Unlike Laplacian-pyramid methods, intermediate outputs are neither supervised nor reused as base images, simplifying the backward pass and avoiding subsampling artefacts.
- 3.2. Dense Compression Units: Dense compression units combine modified dense connectivity with 1×1 convolutional compression, while local and pyramid-wise residual links support very deep models.
- 3.3. Progressive GAN: ProGanSR uses a progressive, reverse-pyramid discriminator that accommodates generator outputs at multiple scales and operates on residual images.The discriminator progressively reduces spatial dimensions with average pooling and produces patch-based convolutional outputs.
- 3.4. Curriculum Learning: Curriculum training proceeds from 2× to larger scales by gradually blending each new pyramid level with the previous output.Compared with simple multi-scale training, this shortens total training time, improves all included scales, and alleviates GAN instability.
4. Evaluation
The evaluation studies component ablations, curriculum learning, progressive alternatives, and comparisons with state-of-the-art methods across quantitative and qualitative settings.
- Ablation study: The ablation study evaluates DCUs, block division, asymmetric pyramids, and curriculum learning using 4× Set14 results.These components are introduced incrementally to measure their effects on reconstruction quality.
- Ablation study: DCUs drastically reduce parameter count without harming reconstruction accuracy and can provide a slight performance gain.Compression layers help the network reassemble features more efficiently.
- Ablation study: The asymmetric pyramid improves reconstruction accuracy over direct upsampling by utilizing high-dimensional features directly.The comparison keeps the total number of dense compression units constant.
- Ablation study: Curriculum learning reaches comparable quality faster because the 2× subnet requires less computation per update.The training-time comparison contrasts curriculum learning with simultaneous multiscale learning.
- Comparison with state-of-the-art approaches: Across datasets, ProSRs achieves the lowest error in most cases, while ProSRℓ has a consistent advantage at higher upsampling ratios and approaches EDSR at 2×.The models are compared against VDSR, DRRN, LapSRN, MsLapSRN, and EDSR.
- Comparison with state-of-the-art approaches: The GAN model produces hallucinated details that align with fine ground-truth structures, extends to 8× upsampling, and is compared with SRGAN and EnhanceNet.The qualitative comparison reports results at 4× and 8×.
5. Runtime.
The asymmetric pyramid supports faster runtime while maintaining similar reconstruction accuracy, and the GAN results show fewer artefacts with good alignment to the original image.
- Runtime: The asymmetric pyramid contributes to faster runtime than approaches with similar reconstruction accuracy.ProSRℓ takes 0.8s, 2.1s, and 4.4s for 2×, 4×, and 8× upsampling of a 520 × 520 image.
- GAN results: In 4× GAN comparisons, the approach is less prone to artefacts and aligns well with the original image.The figure is intended for detailed visual inspection when zoomed in.
- GAN results: At 8×, the adversarial-loss model produces hallucinated details.The 8× result is presented as a qualitative GAN example.
- Runtime: ProSRℓ is reported as 5 times faster than the top-ranking NTIRE challenge team, including the geometric ensemble runtime.The ensemble requires 8 forward passes for each transformed input version.
6. NTIRE Challenge
The NTIRE 2018 challenge evaluated SISR under distinct 8× bicubic and 4× unknown-degradation settings. ProSR ranked highly in the 8× track but only mid-range in the unmatched 4× tracks.
- The NTIRE 2018 SISR challenge benchmarked methods in challenging scenarios, including an 8× track with known bicubic downsampling.
- 2nd in SSIM and 4th in PSNR, ProSR performed strongly in the 8× bicubic challenge track.
- 0.002 SSIM and 0.04dB PSNR below the top-ranking team, ProSR nevertheless ran 5 times as fast at test time.
- 7th/9th/7th in the 4× unknown-degradation tracks, the model ranked in the mid-range without scenario-specific adaptation.
7. Conclusion
The paper concludes that progressive architecture and curriculum learning support efficient, high-quality SISR across scales, including large upsampling ratios. The same progressive principle extends to GAN-based perceptual reconstruction through ProGanSR.
- ProSR incrementally upsamples inputs by 2× in a pyramidal architecture, using asymmetric computation allocation and Dense Compression Units.
- Curriculum learning improves reconstruction quality at large upsampling ratios while simultaneously increasing performance across scales.
- The progressive design improves memory efficiency and supports deeper architectures through dense connectivity in the proposed building blocks.
- ProGanSR applies the progressive principle to GAN-based super-resolution to optimize perceptual quality.