PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks

Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang

arXiv:1710.01992v3cs.CV

TL;DR

Single-image super-resolution needs accurate reconstruction without the large models and runtime costs common in CNN-based methods. LapSRN addresses this with LR-space feature extraction, progressive residual reconstruction, robust loss training, and parameter sharing, achieving favorable runtime, image quality, and parameter efficiency while retaining limits for arbitrary scales and unrecoverable fine details.

  • Problem

    CNN-based super-resolution methods can require many parameters and heavy runtime computation, while existing designs also face oversmoothing and difficulty with large scaling factors.

  • Method

    LapSRN progressively predicts high-frequency residuals across Laplacian-pyramid levels from LR-space features, using Charbonnier loss, deep supervision, and recursive parameter sharing.

  • Results

    LapSRN performs favorably against state-of-the-art methods in runtime and image quality, while MS-LapSRN uses 73% fewer parameters than the preliminary LapSRN.

  • Takeaways & Limitations

    Progressive reconstruction provides intermediate super-resolution predictions, allowing one network to support multiple scales and resource-aware use.

  • Takeaways & Limitations

    Training supports only 2^n× upsampling scales, and parametric methods generally cannot recover fine details absent from severely downscaled inputs.

Abstract

from arXiv · show

Convolutional neural networks have recently demonstrated high-quality reconstruction for single image super-resolution. However, existing methods often require a large number of network parameters and entail heavy computational loads at runtime for generating high-accuracy super-resolution results. In this paper, we propose the deep Laplacian Pyramid Super-Resolution Network for fast and accurate image super-resolution. The proposed network progressively reconstructs the sub-band residuals of high-resolution images at multiple pyramid levels. In contrast to existing methods that involve the bicubic interpolation for pre-processing (which results in large feature maps), the proposed method directly extracts features from the low-resolution input space and thereby entails low computational loads. We train the proposed network with deep supervision using the robust Charbonnier loss functions and achieve high-quality image reconstruction. Furthermore, we utilize the recursive layers to share parameters across as well as within pyramid levels, and thus drastically reduce the number of parameters. Extensive quantitative and qualitative evaluations on benchmark datasets show that the proposed algorithm performs favorably against the state-of-the-art methods in terms of run-time and image quality.

1 INTRODUCTION

Existing CNN-based super-resolution methods face computational, modeling, and scaling challenges. LapSRN addresses these through low-resolution feature extraction, progressive reconstruction, robust loss training, and parameter sharing.

  • Existing CNN-based methods often upscale LR images before deep processing, creating large feature maps and increasing computational cost.
  • Limited-capacity LR-space methods struggle with complicated mappings, while L2 training can produce oversmoothed images and one-step upsampling complicates large scaling factors.
  • LapSRN progressively reconstructs HR images coarse-to-fine by extracting features from LR inputs and predicting sub-band residuals at multiple pyramid levels.
  • The network jointly optimizes convolutional and upsampling filters with Charbonnier loss, while deep supervision supports high-quality reconstruction.
  • Parameter sharing across pyramid levels and recursive feature extraction reduces model size while preserving reconstruction accuracy.The paper further reports local skip connections and multi-scale training as design extensions.

2 RELATED WORK

Related work spans example-based, CNN-based, Laplacian-pyramid, and GAN-based super-resolution methods. The paper positions LapSRN against their computational costs, reconstruction strategies, and modeling objectives.

  • SR based on internal databases: Internal-database methods exploit image self-similarity but can lack sufficient patch diversity and incur heavy patch-search costs.
  • CNN-based SR: CNN-based methods jointly learn feature extraction, nonlinear mapping, and reconstruction, but pre-upsampling increases memory and computation.
  • CNN-based SR: ESPCN and FSRCNN reduce cost by extracting LR-space features, but their limited capacity and one-step reconstruction constrain complex mappings and scale handling.
  • CNN-based SR: L2-trained CNN methods often yield oversmooth results, motivating robust Charbonnier loss; MDSR instead shares parameters across scale-specific branches.
  • Laplacian pyramid: Unlike LAPGAN, which synthesizes diverse images from noise and inputs, LapSRN predicts a particular HR image from an LR image and scale factor.
  • Laplacian pyramid: LAPGAN upsamples before convolution, whereas LapSRN extracts LR-space features and upsamples at each level, reducing computational cost and increasing receptive fields.

3 DEEP LAPLACIAN PYRAMID NETWORK FOR SR

The proposed method is designed within a Laplacian-pyramid framework, with architecture, parameter sharing, loss functions, multi-scale training, and implementation treated as connected design components.

  • The design methodology covers the LapSRN architecture, parameter sharing, loss functions, multi-scale training, implementation, and network training.

3.1 Network architecture

LapSRN takes an LR image and progressively predicts residual images across log2 S pyramid levels. Its feature extraction and image reconstruction branches combine learned upsampling with residual prediction.

  • LapSRN uses log2 S pyramid levels for an upsampling scale factor S, such as three levels for 8× super-resolution.
  • The feature extraction branch embeds features, upsamples them by 2× with transposed convolution, and predicts sub-band residual images.
  • At each level, the upsampled image is added to the predicted residual, and the resulting HR image feeds the next reconstruction level.

3.2 Feature embedding sub-network

The feature embedding sub-network reduces parameters through sharing across pyramid levels and recursive sharing within each level, while local skip connections support deeper-model training.

  • Repeated level structure: At each pyramid level, feature embedding, transposed convolution, and residual prediction form the repeated network structure whose weights can be shared.The architecture extracts nonlinear features, upsamples feature maps, and predicts sub-band residuals.
  • Parameter sharing across pyramid levels: Parameters are shared across pyramid levels because each level has the same structure and predicts residual images at 2× resolution.The shared components include feature embedding, upsampling, and residual prediction layers; parameter count becomes independent of upsampling scale.
  • Parameter sharing within pyramid levels: Recursive layers increase feature-embedding depth without increasing parameters by sharing D convolutional-layer weights across R recursive blocks.Each recursive block contains D distinct convolutional layers, while those weights are reused among recursive blocks.
  • Network depth: The network depth is depth = (D × R + 1) × L + 2, where L = log2 S is the number of pyramid levels.The formula counts recursive convolutional layers, transposed convolutions, and the first and last convolutional layers.
  • Local residual learning: Three local residual-learning designs are examined to stabilize training of deep feature-embedding sub-networks: no skip, distinct-source, and shared-source connections.The study targets gradient vanishing and explosion when training deep models.

3.3 Loss function

LapSRN uses deep multi-scale supervision with a robust Charbonnier loss instead of mean squared error to train coarse-to-fine reconstruction and handle outliers.

  • Residual reconstruction: The target at level l is modeled as ˆy_l = x_l + ˆr_l, where the network predicts a residual added to the upscaled LR image.Here, x_l is the upscaled LR image and ˆr_l is the predicted residual image.
  • Robust loss: The overall objective replaces mean squared error with a robust Charbonnier penalty to handle outliers.The penalty is a differentiable variant of the L1 norm, with ϵ empirically set to 1e-3.
  • Deep supervision: Each pyramid level has its own loss against the corresponding ground-truth HR image, providing deep supervision for coarse-to-fine reconstruction.The multi-loss structure guides reconstruction at multiple scales and is intended to reduce spatial aliasing artifacts.

3.4 Multi-scale training

LapSRN trains multi-scale models using samples from multiple upsampling scales, exploiting inter-scale correlations while restricting training scales to powers of two.

  • Multi-scale training: Multi-scale training jointly uses samples from multiple upsampling scales because super-resolution tasks have inter-scale correlations.The approach trains one model across selected scales and evaluates the corresponding pyramid levels.
  • Scale limitation: Unlike pre-upsampling methods, LapSRN limits training to upsampling scales of 2^n×, where n is an integer.Pre-upsampling methods can use scale augmentation for arbitrary upsampling scales.

3.5 Implementation and training details

The implementation uses fixed convolutional settings and a 291-image training set, while ablations assess convergence and the contributions of the pyramid structure, losses, and global residual learning.

  • Implementation settings: All convolutional layers use 64 filters except designated input, residual-prediction, and image-upsampling layers; convolution and transposed-convolution kernels are 3×3 and 4×4.Zero padding preserves feature-map sizes at each level.
  • Training data: The training set contains 291 images, with 128×128 HR patches, batch size 64, and augmentation by scaling, rotation, and other transformations.The images come from Yang et al. and the Berkeley Segmentation Dataset.
  • Convergence analysis: The full model converges faster and performs better when pyramid structures, loss functions, and global residual learning are retained.The convergence analysis replaces each component with one used in existing methods.
  • Ablation evaluation: The ablation study compares the full LapSRN with variants on SET5 and SET14 for 4× super-resolution.The caption reports that the full model performs favorably against several LapSRN variants.

4 DISCUSSIONS AND ANALYSIS

The analysis evaluates LapSRN’s architectural components, parameter-sharing strategies, depth, and multi-scale training. The results show that pyramid reconstruction, multi-scale supervision, recursive sharing, and suitable skip connections improve efficiency or reconstruction quality.

  • 4.1.1 Pyramid structure: The pyramid structure improves 4× SR performance by 0.7 dB on SET5 and 0.4 dB on SET14.The comparison removes the pyramid structure while retaining global residual learning and matches network depth.
  • 4.1.2 Global residual learning: Global residual learning accelerates convergence, while the full LapSRN outperforms SRCNN within 10 epochs.The non-residual network converges slowly and fluctuates significantly during training.
  • 4.1.4 Multi-scale supervision: Multi-scale supervision progressively reconstructs clear, sharp lines and reduces spatial aliasing artifacts.Without supervision beyond the finest scale, spatial aliasing artifacts remain more pronounced.
  • 4.2 Parameter sharing: Parameter sharing reduces the preliminary LapSRN’s parameters from 812k to 407k, while further compression to 222k or 112k causes performance drops on URBAN100.Weights are shared across pyramid levels, and recursive blocks share layers within the feature-embedding sub-network.
  • 4.4 Multi-scale training: Training with 2×, 4×, and 8× samples produces one model that handles multiple scales, generalizes to unseen 3× SR, and performs favorably against single-scale training.All multi-scale models use the same parameter count because of parameter sharing.

5 EXPERIMENT RESULTS

Experiments across benchmark datasets compare LapSRN with state-of-the-art methods in quality, runtime, parameter count, visual fidelity, and adversarial training. LapSRN performs favorably, particularly at 4× and 8× SR, while retaining a small model footprint and fast execution.

  • 5.1 Comparisons with state-of-the-arts: LapSRN performs favorably against existing methods, especially for 4× and 8× SR, and achieves higher IFC values.The method also produces comparable 3× SR results without using 3× training samples.
  • 5.1 Comparisons with state-of-the-arts: Progressive reconstruction and robust loss suppress spatial aliasing artifacts while recovering straight lines, grid patterns, and text in visual comparisons.At 8× SR, MS-LapSRN reconstructs high-quality images at a relatively fast speed compared with the evaluated methods.
  • 5.2 Execution time: MS-LapSRN-D5R2 is faster than every evaluated method except FSRCNN, while MS-LapSRN-D5R8 outperforms DRRN and is an order of magnitude faster.Runtime is measured on the same CPU and GPU system using the compared methods’ source code or rebuilt implementations.
  • 5.3 Model parameters: MS-LapSRN uses 73% fewer parameters than LapSRN, 66% fewer than VDSR, 87% fewer than DRCN, and 25% fewer than DRRN while achieving state-of-the-art performance.On URBAN100 for 4× SR, MS-LapSRN-D5R8 improves over SRCNN and FSRCNN by about 0.9 to 1 dB.
  • 5.6 Adversarial training: Adversarial training generates more plausible irregular details but is less faithful to ground truth and less accurate than Charbonnier-loss training.The comparison concerns 4× SR visual results involving regions such as grass and feathers.
  • 5.7 Limitations: At 8× SR, LapSRN does not hallucinate fine details lost in the input, and all evaluated algorithms except SelfExSR fail to recover the building’s fine structure.The paper identifies this as a limitation shared by parametric super-resolution methods.

6 CONCLUSIONS

The paper concludes that LapSRN combines coarse-to-fine residual reconstruction, deep supervision, parameter sharing, local skip connections, and multi-scale training for fast, accurate super-resolution. The authors report reduced parameters and a single model capable of multiple upsampling scales.

  • 6 CONCLUSIONS: LapSRN progressively predicts high-frequency residuals in a Laplacian pyramid with deep supervision from robust Charbonnier losses.The network is designed for fast and accurate image super-resolution.
  • 6 CONCLUSIONS: Parameter sharing across and within pyramid levels uses 73% fewer parameters than the preliminary method while improving performance.Local skip connections support training deeper models, and multi-scale training supports multiple upsampling scales.
  • 6 CONCLUSIONS: Multi-scale training enables a single model to handle multiple upsampling scales.The conclusion presents this as part of LapSRN’s demonstrated image-super-resolution capability.

7 APPENDIX

The appendix evaluates multi-scale training and human perceptual preferences for MS-LapSRN. Multi-scale training improves performance across evaluated scales, while paired user comparisons favor MS-LapSRN on BSDS100 and URBAN100.

  • Multi-scale training: Training with combined 2×, 4×, and 8× samples achieves better performance across all evaluated upsampling scales and generalizes to unseen 3× examples.The evaluation covers SET14, BSDS100, and URBAN100 at 2×, 3×, 4×, and 8× super-resolution.
  • Human subject study: The human subject study uses paired comparisons through an interface that lets participants switch between two results to judge perceptual differences.Results from 71 participants were collected after sanity-check filtering.
  • Human subject study: Bradley-Terry scores indicate that MS-LapSRN performs favorably against the other compared approaches on BSDS100 and Urban100.The paired-comparison results were fitted to the Bradley-Terry model and normalized to zero means.
  • Human subject study: MS-LapSRN is preferred by 75% of users on BSDS100 and 84% on URBAN100 in comparisons with other super-resolution methods.The study compares MS-LapSRN with FSRCNN, VDSR, DRRN, and LapSRN.
  • DIV2K evaluation: MS-LapSRN performs favorably against existing approaches on 2×, 4×, and 8× super-resolution in the DIV2K evaluation.Table 10 marks the best and second-best performances using red and blue, respectively.
Loading 1710.01992v3…