PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution

Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, Shuicheng Yan

arXiv:1604.08671v2cs.CV

TL;DR

Image SR must recover perceptually important high-frequency details from inherently ill-posed LR observations. DEGREE instead recurrently learns LR-to-HR residuals while using LR edge maps to guide HR edge recovery, with bypass connections supporting deeper training. Experiments on three benchmark datasets show superiority over established baselines and new state-of-the-arts.

  • Problem

    Image SR seeks to recover HR images from LR observations, but the ill-posed problem makes sharp high-frequency details difficult to recover.

  • Method

    DEGREE recurrently predicts LR-to-HR residuals and HR edges, progressively recovering frequency sub-bands with edge guidance and bypass connections.

  • Results

    DEGREE outperforms well-established baselines on three benchmark datasets and provides new state-of-the-arts.

  • Takeaways & Limitations

    Recurrent residual learning with bypass structures provides an end-to-end framework that exploits edge priors for progressive HR image reconstruction.

  • Takeaways & Limitations

    The hand-crafted sub-band paradigm depends heavily on frequency-filter choice, where a bad choice can limit cross-band modeling and HR recovery.

Abstract

from arXiv · show

In this work, we consider the image super-resolution (SR) problem. The main challenge of image SR is to recover high-frequency details of a low-resolution (LR) image that are important for human perception. To address this essentially ill-posed problem, we introduce a Deep Edge Guided REcurrent rEsidual~(DEGREE) network to progressively recover the high-frequency details. Different from most of existing methods that aim at predicting high-resolution (HR) images directly, DEGREE investigates an alternative route to recover the difference between a pair of LR and HR images by recurrent residual learning. DEGREE further augments the SR process with edge-preserving capability, namely the LR image and its edge map can jointly infer the sharp edge details of the HR image during the recurrent recovery process. To speed up its training convergence rate, by-pass connections across multiple layers of DEGREE are constructed. In addition, we offer an understanding on DEGREE from the view-point of sub-band frequency decomposition on image signal and experimentally demonstrate how DEGREE can recover different frequency bands separately. Extensive experiments on three benchmark datasets clearly demonstrate the superiority of DEGREE over well-established baselines and DEGREE also provides new state-of-the-arts on these datasets.

1 Introduction

Image super-resolution is ill-posed because recovering sharp high-frequency details from LR observations is difficult. DEGREE addresses this by recurrently recovering residual and edge information, achieving strong benchmark performance.

  • 1 Introduction: SR estimates an HR image from LR observations, but the inverse signal estimation problem is essentially ill-posed.Recent methods use natural-image priors to regularize recovery.
  • 1 Introduction: Minimizing MSE alone often fails to recover sharp high-frequency details such as textures and edges.
  • 1 Introduction: Edges are easier to detect in LR images than textures and therefore provide informative priors for recovering HR details.Separating and modeling edge information can benefit image SR.
  • 1 Introduction: DEGREE predicts LR-to-HR residuals and HR edges instead of directly predicting HR images.Combining these predictions produces the recovered HR image.
  • 1 Introduction: DEGREE progressively recovers different frequency sub-bands through recurrent residual learning and uses bypass connections to fuse and propagate intermediate results.Edge features guide the recovery of sharp details.
  • 1 Introduction: DEGREE outperforms well-established baselines on three benchmark datasets and provides new state-of-the-arts.The network is described as end-to-end trainable and effective in exploiting edge priors.

2 Related Work

Prior image-SR work uses deep networks, sub-band decomposition, and edge-preserving regularization to recover details. These approaches motivate architectures that better model high-frequency information and image priors.

  • 2 Related Work: Deep SR methods use CNNs to model image contents, while some architectures embed sparsity priors into the network.Designing suitable prior-aware architectures can require substantial domain knowledge and effort.
  • 2 Related Work: Sub-band methods recover image information at different frequency bands separately to address high-frequency information loss from MSE minimization.Examples use DWT interpolation or constraints on each sub-band.
  • 2 Related Work: TV and BTV regularizations guide SR recovery by suppressing spurious details and preserving sharp edges.
  • 2 Related Work: Figure 2 presents sub-band reconstruction, its relaxed element-wise-summation version, and a two-convolution-layer deep network derived from that formulation.The relaxed framework uses only MSE loss to constrain HR recovery.

3 Deep Recurrent Residual Learning for Image SR

The section develops progressive sub-band reconstruction for image super-resolution, then reformulates it as an end-to-end recurrent residual network. The approach models dependencies across frequency bands while addressing limitations of hand-crafted filter-based recovery.

  • 3.1 Sub-Band Reconstruction for Image SR: Sub-band reconstruction separates image signals into frequency components so low-energy bands receive sufficient attention and sharper details can be preserved.The method models each sub-band individually during super-resolution.
  • 3.1 Sub-Band Reconstruction for Image SR: Progressive recovery models dependencies across sub-bands by using lower-frequency reconstructions to recover higher-frequency components.This replaces independent sub-band recovery with an iterative process.
  • 3.1 Sub-Band Reconstruction for Image SR: An auxiliary signal accumulates recovered sub-bands, while functions F_i and G_i progressively estimate sub-band components and the reconstructed HR image.The process starts from s_0 = y and eventually reproduces the HR image x.
  • 3.2 Learning Sub-Band Decomposition by Recurrent Residual Net: The original sub-band paradigm depends on hand-crafted frequency filters and does not provide an end-to-end trainable framework.Poor filter choices can limit modeling of correlations between sub-bands and recovery of the HR image.
  • 3.2 Learning Sub-Band Decomposition by Recurrent Residual Net: Using summation for G_i yields the residual update s_i = s_{i−1} + F_i(s_{i−1}), eliminating explicit intermediate sub-band estimation and enabling end-to-end training.The reformulation uses an MSE loss at the top HR-prediction layer.
  • 3.2 Learning Sub-Band Decomposition by Recurrent Residual Net: Stacked recurrent units perform progressive sub-band recovery, with nonlinear transformations and data-adaptive functions learned by minimizing MSE.The network follows the intuition of gradual sub-band recovery.

4 DEGREE Network for Edge Preserving SR

DEGREE embeds LR edge information into a recurrent residual network that progressively recovers HR details across frequency sub-bands. It jointly predicts HR images and edges, combining edge and reconstruction losses during end-to-end training.

  • Recurrent Residual Learning: The recurrent residual architecture progressively recovers high-frequency details at multiple recurrence stages and fuses outputs through bypass connections.
  • Edge Guidance: DEGREE uses LR edge features alongside the raw LR image to guide HR edge and image recovery.
  • Edge-Guided Reconstruction: Features from the penultimate layer separately support HR image reconstruction and HR edge prediction, with edge-guided reconstruction features formed from both outputs.
  • Sub-band Recovery: DEGREE models an HR image as low- and high-frequency components, using the LR image as an estimate of the low-frequency component.
  • Training Objective: Training minimizes a joint MSE for HR image and HR edge estimation, with λ balancing data fidelity and edge-prior terms.The paper sets λ to 1 because performance remains similar across a large range of values.

5 Experiments

The experiments compare DEGREE with interpolation, sparse-coding, neighborhood-regression, CNN, exemplar-based, and sub-band recovery methods on three standard SR datasets and three scale factors.

  • Evaluation Setup: Experiments evaluate DEGREE against Bicubic interpolation and six established SR methods on Set5, Set14, and BSD100.The datasets contain 5, 14, and 100 images respectively.

Implementation Details

Implementation evaluates two DEGREE variants with controlled network settings, augmentation, luminance-channel processing, and PSNR/SSIM metrics. DEGREE-1 performs best, while DEGREE-2 shows gains over the strongest baseline across scale factors.

  • Network Configuration: The study evaluates 10-layer and 20-layer networks with 64 channels, 3 × 3 filters, and bypass connections every two convolutional layers.
  • Training Data: Training uses flipping and rotations to generate 16 augmented images per training image, producing around 240,000 sub-images.
  • Models and Metrics: DEGREE-1 and DEGREE-2 are measured with PSNR and SSIM, with DEGREE-1 defined as the 10-layer model and DEGREE-2 as the 20-layer model.
  • Results: DEGREE-1 performs best, while DEGREE-2 improves over CSCN by up to 0.52 PSNR and 0.0033 SSIM on Set5 at ×2.On Set5, DEGREE-2 also improves performance by 0.29 at ×3 and 0.17 at ×4.

5.2 Subjective Evaluation

DEGREE produces visually cleaner and sharper super-resolved images than several baselines, with fewer artifacts in challenging regions. Figures 4 and 5 compare effectiveness and model variants on Set5.

  • Visual comparisons: DEGREE generates cleaner details and sharper edges than A+, SRCNN, JSB-NE, and CSCN on the Butterfly image at 4× enlargement.The method avoids artifacts near the corners of the yellow and white plaques, where CSCN still leaves some artifacts.
  • Benchmark comparison: Figure 4 compares the method’s effectiveness and time complexity with state-of-the-art methods for 2× enlargement on Set5.
  • Model variants: Figure 5 compares three versions of the proposed method for 3× enlargement on Set5.

5.3 Running Time

The running-time evaluation compares SR methods by PSNR and execution time. On Set5, the GPU implementation of DEGREE processes all images in 1.81 seconds.

  • Runtime evaluation: 1.81 seconds is the GPU running time reported for DEGREE to perform SR on all Set5 images.Figure 4 plots running time in seconds against PSNR, using public implementations and a specified Intel X5675 machine with 24 GB memory.
  • Evaluation scope: The study investigates DEGREE’s effectiveness and time complexity alongside its broader model analysis.
  • Training behavior: Without bypass connections, DEGREE does not converge within 40,000 iterations across learning rates of 0.1, 0.01, and 0.001.The ablation attributes faster training convergence to adding bypass connections.

Ablation Analysis

The ablations examine visual quality, model size, and learned features. They show artifact avoidance in visual comparisons, size-dependent performance, and frequency-organized feature progression.

  • Subjective evaluation: DEGREE avoids artifacts near the corners of the Butterfly image’s white and yellow plaques at 4× enlargement.
  • Model Size: Models larger than (20, 32) × 10^5 and (8, 64) × 10^5 parameters are necessary for reasonably good performance.
  • Model Size: Increasing the model to (20, 16) × 10^4 parameters performs better than SCN128, whose model size is slightly smaller.
  • Visualization of learned sub-bands: Earlier recurrent stages capture edge features, middle stages contain textures, and the fourth recurrent stage fixes details.The last layer models the residual signal, matching a progression across frequency-related features.

Visualization of Learned Sub-Bands

DEGREE provides a general framework for embedding task-specific prior knowledge by configuring preliminary inputs and replacing edge-related feature maps.

  • General prior framework: DEGREE can embed different priors by setting g(y) in finput and replacing fedge with feature maps representing the chosen prior.For JPEG artifact reduction, the inputs include compressed-image edge maps and a DCT block map.

6 Conclusions

The paper concludes that DEGREE uses edge-guided recurrent residual learning to recover richer high-resolution details, while providing a general framework for embedding natural-image priors. Experiments and visualizations examine its parameterization, learned sub-bands, and reconstruction quality.

  • Conclusions: DEGREE separates edge information from the image signal to guide high-resolution image recovery.Low-resolution edge maps enter the input features, while high-resolution edge maps constrain feature learning for reconstruction.
  • Conclusions: Recurrent residual learning with by-pass connections enables training deeper networks progressively.The conclusion links this structure to the method's ability to produce high-resolution images with richer details.
  • Experiments: Figure 7 evaluates PSNR for 2× super-resolution on Set5 across parameter counts, comparing DEGREE with CSCN and CNN.The figure examines how model size relates to performance relative to the two comparison methods.
  • Experiments: The paper visualizes learned sub-bands during recovery on Butterfly to examine frequency-specific representations.This visualization complements the paper's frequency-decomposition perspective on recurrent recovery.
  • Experiments: On image 86000 at 3× super-resolution, DEGREE presents fewer artifacts around window boundaries than the compared algorithms.The comparison is reported as a visual reconstruction outcome.
  • Experiments: On image 223061 at 3× super-resolution, DEGREE produces more complete and sharper edges than the compared algorithms.The visual comparison directly reflects the network's edge-guided reconstruction objective.
  • Conclusions: The framework is presented as applicable to embedding various natural-image priors into image-processing tasks.This extends the paper's contribution beyond the specific edge prior used in DEGREE.
Loading 1604.08671v2…